Home › Directory › Scraping Libraries
Scraping Libraries
Crawling and HTML-parsing libraries.
JSON: /api/v1/directory.json (filter on category=scraping-libraries)
Scrapy
Mature Python framework for writing fast, scalable crawlers and extracting structured data.
FreeNo authApache Nutch
Nutch is a highly extensible, highly scalable, matured, production-ready Web crawler which enables fine grained configuration and accomodates a wide variety of data acquisition tasks.
See siteSee docsCrawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
FreeNo authColly
Elegant Scraper and Crawler Framework for Golang.
FreeNo authGoutte
Goutte, a simple PHP Web Scraper.
FreeNo authScrapy-Selenium
Scrapy middleware to handle javascript pages using selenium
FreeNo authScrapy-Selector
Scrapy, a fast high-level web crawling & scraping framework for Python.
FreeNo authCheerio
The fast, flexible & elegant library for parsing and manipulating HTML and XML.
See siteSee docsJsoup
Open source Java HTML parser, with the best of HTML5 DOM methods and CSS selectors, for easy data extraction.
See siteSee docsMechanicalSoup (Python)
A Python library for automating interaction with websites.
FreeNo authRequests-HTML
Pythonic HTML Parsing for Humans™.
FreeNo authNode-fetch
A light-weight module that brings the Fetch API to Node.js
FreeNo authAxios
Documentation for the axios HTTP project
See siteSee docs