Working with WebSPHINX Web Crawler
3 likes3,868 views
The document provides instructions for using the WebSPHINX web crawler. It describes the 4 main steps: 1) Running the Java program, 2) Specifying a starting URL, 3) Choosing an action (e.g. save, concatenate, extract, highlight pages), and 4) Selecting a visualization mode (graph, outline, statistics). It then demonstrates saving pages, concatenating results, extracting objects, highlighting text, and viewing the crawling process in the different visualization modes.
More Related Content
Similar to Working with WebSPHINX Web Crawler (20)
![From Curiosity to ROI — Cost-Benefit Analysis of Agentic Automation [3/6]](https://cdn.slidesharecdn.com/ss_thumbnails/session-3-250903141006-a3917e1a-thumbnail.jpg?width=560&fit=bounds)
Working with WebSPHINX Web Crawler
- 1. Working With WebSPHINX Web Crawler
- 2. Step 1: Run “Java -jar webshinx.jar” in the command prompt
- 3. This opens the WebSPHINX GUI
- 4. Step 2: Specifying Starting URL (http://www.google.com here)
- 5. Step 3: Specify the action to be taken There exist five options: 1) Save: We can save pages to a specified directory 2) Concatenate: We can concatenate the results into a single page, mainly used for printing the search results 3) Extract: We can extract a specific object type, but need to provide HTML code for the same. Eg- <a>(?{logo}<img>)<p>({caption})</a> allows us to extract images 4) Highlight: Results are highlighted in a specific colour (default: blue) 5) Script: (We couldn't try this out as no scripts were available to us)
- 6. Step 4: Select the visualization mode 1) Graph 2) Outline 3) Statistics
- 7. Action to be taken Save
- 8. Action: Save
- 9. As the Myresults folder was specified as the save location, history and index can be seen here
- 10. Files associated with the URL can be seen here
- 11. Action to be taken Concatenate
- 14. Action to be taken Extract
- 15. Action: Extract
- 17. Action to be taken Highlight
- 18. Action: Highlight Highlight colour: Blue (default)
- 20. Crawling Visualization Graph Mode
- 21. Starting URL or Root of the tree
- 22. Root of the tree Son the root Son of the previous son of the root
- 23. In this crawler, this process is continues till either all the links have been retrieved or no more hard disk space is available.
- 24. The number of links crawled keeps on increasing
- 25. We can even see the URL associated with the page
- 26. Crawling visualization OutlineMode
- 27. The outline mode displays the flow in the crawling process, i.e. it shows how the process is proceeding by displaying the nesting of URLs
- 28. We can clearly see increase in the number of pages crawled
- 29. This screenshot displays the increase in the number of home pages as well
- 30. Crawling visualization Statistics Mode
- 33. Now showing the process in command prompt