How to scrape a website: the complete guide
A complete guide to scraping a website: choosing a tool, fetching HTML, handling JavaScript, beating anti-bot protections, storing data and best practices.
Read the guide
Guides
Web scraping guides
Everything you need to scrape a website cleanly: methods, anti-bot bypass, AI extraction and the legal framework.
All guides
Bypassing anti-bot protection: the complete guide
How anti-bot protections work (fingerprinting, CAPTCHA, rate limiting) and how to bypass them cleanly, while respecting robots.txt and GDPR.
Read the guideConvert a website to Markdown: the complete guide
Convert a website to Markdown: why this format suits LLMs, how to clean the HTML, which tools to use and how to automate the conversion at scale.
Read the guideWeb scraping and GDPR: the compliance guide
Is web scraping legal? A GDPR guide: legal bases, personal data, robots.txt and compliance measures to collect public data lawfully in the EU.
Read the guideScraping API vs scraping: which should you choose?
Official API or web scraping? How each works, cost, reliability, legality and how to choose — plus how to combine an API and scraping for complete data.
Read the guideWeb scraping for LLMs: preparing the web for your models
Web scraping for LLMs: fetch the real content, convert it to clean Markdown, estimate the token reduction and build a reliable RAG pipeline.
Read the guide