Web-Crawling

Downloading the originals behind 2.98 billion image URLs in OmniCorpus-CW, I left almost all operations to an AI agent: patrols, rate tuning, creating and deleting machines, incident handling and billing. I only made the decisions. In 79 hours it saved 99.6% of the images, 744 TB, for about $175 in machines.

在慕课网学习 Scrapy 时所作的笔记，后来不断发展，内容更丰富，不仅限于 Scrapy 的内容。