https://python.plainenglish.io/harnessing-distributed-crawling-for-large-scale-web-scraping-tasks-00814ed37770