Home › Directory › Scraping & Browsing

Scraping & Browsing

Crawlers, headless browsers and page readers that let agents use the web.

JSON: /api/v1/directory.json (filter on category=scraping-browsing)

Turns websites into clean markdown or structured data, with crawl, scrape and map endpoints.

Managed headless browsers in the cloud for agents, with session recording and stealth features.

Open-source browser automation library for Chromium, Firefox and WebKit in several languages.

Node library that drives Chrome or Firefox through DevTools protocols for scraping and testing.

Prefix a URL and get an LLM-friendly text version of the page back; a keyless tier exists with rate limits.

Platform with thousands of ready-made scrapers called Actors, plus infrastructure to run your own.

Open-source crawler that outputs markdown tuned for LLM pipelines, runnable locally or in Docker.

Scraping API that handles proxies, headless rendering and blocks behind a single request.

Large proxy network with web unlocker, scraper APIs and datasets for data collection at scale.

Run headless Chrome from a Worker or REST API to screenshot, scrape or automate pages.

Open-source library that lets an LLM control a real browser, with an optional hosted cloud.

Browser automation SDK that mixes plain-language actions with code, built on Playwright.

Mature Python framework for writing fast, scalable crawlers and extracting structured data.

Free, open repository of petabytes of web crawl data, useful for research and training datasets.