Logo
Deploy Now

Categories

Developer Tools Automation API Machine Learning Data Pipeline AI Tools Web Scraping

Stars

78,887

Forks

8,172

Watchers

402

Developer links

Last Updated: 08-21-2026 04:58 PM (UTC)

Crawl4AI

With over 77,000 GitHub stars, Crawl4AI is the most-starred open-source web crawler on GitHub and the go-to tool for converting the web into AI-ready data. Built on Python and Playwright, it transforms any website into clean Markdown with headings, tables, code blocks, and citation hints optimized for LLM ingestion, or extracts structured JSON via CSS selectors, XPath expressions, or direct LLM-based schema extraction through OpenAI, Anthropic, and Ollama providers. The self-hosted Docker server exposes a REST API on port 11235 with endpoints for crawling, streaming results, screenshots, PDF generation, JavaScript execution, and LLM-powered extraction. Version 0.9.x introduced secure-by-default operation with mandatory JWT authentication, strict request validation, declarative hooks replacing inline code, and bounded job queues. Adaptive crawling uses information foraging algorithms to determine when sufficient data has been gathered, while deep crawl mode traverses link graphs intelligently. The async browser pool manages concurrent sessions with stealth plugins, proxy rotation, custom headers, and session persistence for authenticated scraping. A built-in MCP server enables direct integration with Claude, ChatGPT, and Cursor for AI-driven web research workflows. Content filtering applies BM25 and TF-IDF relevance scoring to extract only pertinent sections from noisy pages. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

Crawl4AI
Crawl4AI
Crawl4AI
Crawl4AI
Crawl4AI

Benefits

  • LLM-Optimized Output Format
  • Generates clean Markdown with preserved headings, tables, code blocks, and citation metadata specifically structured for RAG pipeline ingestion and LLM context windows.
  • Secure Self-Hosted REST API
  • Docker server with JWT authentication, CORS deny-by-default, bounded job queues, and declarative hooks eliminates the per-request costs of commercial scraping APIs entirely.
  • Intelligent Adaptive Crawling
  • Information foraging algorithms automatically determine when sufficient data has been collected, reducing unnecessary requests and optimizing token budgets for AI workflows.
  • Native MCP Protocol Integration
  • Built-in Model Context Protocol server connects directly to Claude, ChatGPT, and Cursor, enabling AI agents to trigger web crawls without custom API integration code.

Features

  • Structured Data Extraction
  • Extract JSON from any page using CSS selectors, XPath expressions, LLM-based schemas through OpenAI, Anthropic, or Ollama, and regex pattern matching strategies.
  • Async Browser Pool
  • Playwright-based concurrent browser sessions with stealth plugins, proxy rotation, cookie persistence, and JavaScript execution for rendering complex single-page applications.
  • Real-Time Monitoring
  • Built-in dashboard tracks browser pool status, active crawl jobs, system health metrics, and request throughput with per-job correlation identifiers for debugging.
  • Deep Crawl Engine
  • Graph-based link traversal with configurable depth, domain filtering, and deduplication discovers and processes entire site structures automatically.
  • Content Relevance Filtering
  • BM25 and TF-IDF scoring algorithms filter extracted content by relevance to your query, eliminating boilerplate navigation, ads, and irrelevant page sections.