Web Crawling is extremely useful to automate certain tasks performed routinely on websites. You can write a crawler to interact with a website just as a human would do. In an earlier article, they covered the basics of writing a webcrawler using the python module, scrapy. The limitation of that approach is that the crawler does not support javascript. It will not work properly with those websites that make heavy use of javascript to manage the user interface. For such situations, you can write a crawler which uses Google Chrome and hence can handle javascript just like a normal user-driven Chrome browser. Automating Google Chrome involves use of a tool called Selenium. It is a software component which sits between your program and the Browser, and helps you drive the browser through your program. In this article, they take you through the complete process of automating Google Chrome. This will be added to Bot Research Subject Tracer™. This will be added to World Wide Web Reference Subject Tracer™. This will be added to Business Intelligence Resources Subject Tracer™. This will be added to Entrepreneurial Resources Subject Tracer™. This will be added to the tools section of Research Resources Subject Tracer™.
