Automated Web Scraping Pipeline: From Cron Job to Clean Data in 30 Minutes
A complete automated web scraping pipeline: scheduling with cron, crawling with Crawlee, cleaning with an LLM, and storing in SQLite. Real stack, real numbers.
💡 What You Will Learn
A complete automated web scraping pipeline: scheduling with cron, crawling with Crawlee, cleaning with an LLM, and storing in SQLite. Real stack, real numbers.
📜 Table of Contents
Manual scraping is a hobby. Automated web scraping is a pipeline that runs while you sleep. Here is the stack we actually use for a daily price-tracking job, with the failure modes you will hit.
The 4-Stage Pipeline
Stage 1: a cron job fires at 06:00 daily; a Python script checks if the target site changed (ETag or status check) before crawling. Stage 2: Crawlee (25,204 stars) handles queueing and retries; 200-500 URLs take 5-8 minutes with 4 concurrent workers, and a page failing twice gets logged and skipped. Stage 3: raw HTML goes to an LLM call with a strict JSON schema, so field extraction survives markup changes. Stage 4: results land in SQLite with a UNIQUE constraint on (url, date).
Failure modes: anti-bot walls need rotated User-Agent and 2-5s random delays; layout changes degrade LLM extraction gracefully while regex dies instantly; duplicate runs are solved by the UNIQUE constraint; silent failures are caught by alerting via Telegram or email when a batch returns 0 new rows.
Comparison
| Stage | Tool | Notes |
|---|---|---|
| Schedule | cron | Daily 06:00 ETag check |
| Crawl | Crawlee | 25,204 stars, 4 workers |
| Clean | LLM + JSON schema | Survives markup changes |
| Store | SQLite | UNIQUE(url, date) |
FAQ
Q: Do I need a proxy?
A: For 200-500 pages/day on well-behaved sites, no. Add proxies only when you hit 403s or CAPTCHAs.
Q: How do I monitor it?
A: A simple health check: if today row count is zero and yesterday was not, alert. That one rule catches 90 percent of pipeline failures.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only โ no paid placements.
