Sometimes one needs to crawl certain information online as part of their project. However, websites do not like crawlers much because of obvious reasons. As a result, websites would implement a mechanism for blocking crawlers. In this post, I will explain how to crawl websites without exposing your information and, in case the crawler got blocked, it is capable of …
