Firecrawl turns websites into clean data for AI. You give it a URL, it gives you markdown.

The annoying part of scraping isn’t fetching HTML — it’s the JavaScript rendering, the anti-bot blocks, the rotating proxies, and then still getting back a wall of navigation and ads with the thing you wanted buried inside. Firecrawl takes care of that pipeline. /scrape returns a page as markdown or JSON. /crawl walks a whole site without needing a sitemap. /search queries the web and returns full page content. There’s also an agent mode that can click, type, and scroll before extracting, which is useful for anything behind a login or a UI interaction.

PDFs and DOCX files work too.

Open source and self-hostable. Free tier covers 500 pages; paid plans go up from there.