The Python, AI & Data series goes from collecting and cleaning data to machine learning, deep learning, Vietnamese language processing, LLMs, RAG, agents and running models in production, with runnable Python code ...
A crawler that finishes without errors does not mean the data is usable, and pandas is the first tool you reach for to fix it. The same product can show up three times under three different URLs; the...
The previous articles made the crawler fast: asyncio opens dozens of connections at once, and Playwright can read pages built with JavaScript. But fast is not the same as good. A crawler that only kno...
You write a crawler with requests and BeautifulSoup, run it, and get back an almost empty HTML page: no products, no prices, just a few script tags. The real content is built later by JavaScript in th...
The previous article in this series used Scrapy, a framework that ships with its own scheduler and concurrency limits. But many smaller jobs, such as calling a few hundred internal APIs or checking th...
The requests plus BeautifulSoup loop from the previous article on crawling with requests, BeautifulSoup and robots.txt is enough for a few dozen pages. Once you need thousands of pages, you end up wri...
Almost every data project eventually needs numbers that the source does not offer through an API: a competitor's price list, a public product catalog, an exchange-rate table. That is when crawling dat...
In a data or AI project, the thing that costs the most hours is usually not the algorithm but the question "why does it work on my machine?". One notebook uses one version of numpy, a teammate's machi...
