README.md
<a href="https://trendshift.io/repositories/11716" target="_blank"></a>
Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.
๐ Apply here for early access
Weโll be onboarding in phases and working closely with early users.
Limited slots.
</a>
<a href="https://www.linkedin.com/company/crawl4ai">
</a>
<a href="https://discord.gg/jP8KfhDhyN">
</a>
Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.
โจ Check out latest update v0.8.5
โจ New in v0.8.5: Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes! Automatic 3-tier anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, config defaults API, consent popup removal, and critical security patches. Release notes โ
โจ Recent v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with resume_state and on_state_change callbacks for long-running crawls. New prefetch=True mode for 5-10x faster URL discovery. Release notes โ
โจ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. Release notes โ
โจ Previous v0.7.7: Complete Self-Hosting Platform with Real-time Monitoring! Enterprise-grade monitoring dashboard, comprehensive REST API, WebSocket streaming, and smart browser pool management. Release notes โ
<details> <summary>๐ค <strong>My Personal Story</strong></summary>I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. Thatโs where I learned how much extraction matters.
In 2023, I needed web-to-Markdown. The โopen sourceโ option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now itโs the most-starred crawler on GitHub.
I made it open source for availability, anyone can use it without a gate. Now Iโm building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.
</details> <details> <summary>Why developers pick Crawl4AI</summary># Install the package
pip install -U crawl4ai
# For pre release versions
pip install crawl4ai --pre
# Run post-installation setup
crawl4ai-setup
# Verify your installation
crawl4ai-doctor
If you encounter any browser-related issues, you can install them manually:
python -m playwright install --with-deps chromium
import asyncio
from crawl4ai import *
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://www.nbcnews.com/business",
)
print(result.markdown)
if __name__ == "__main__":
asyncio.run(main())
# Basic crawl with markdown output
crwl https://www.nbcnews.com/business -o markdown
# Deep crawl with BFS strategy, max 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
# Use LLM extraction with a specific question
crwl https://www.example.com/products -q "Extract all product prices"
๐ Sponsorship Program Now Open! After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for startups and enterprises. Be among the first 50 Founding Sponsors for permanent recognition in our Hall of Fame.
Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community โ while giving you direct access to premium benefits.
<div align=""> </div>Why sponsor?
No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI.
srcset and picture.raw:) or local files (file://).โจ Visit our Documentation Website
Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.
<details> <summary>๐ <strong>Using pip</strong></summary>Choose the installation option that best fits your needs:
For basic web crawling and scraping tasks:
pip install crawl4ai
crawl4ai-setup # Setup the browser
By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.
๐ Note: When you install Crawl4AI, the crawl4ai-setup should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:
Through the command line:
playwright install
If the above doesn't work, try this more specific command:
python -m playwright install chromium
This second method has proven to be more reliable in some cases.
The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:
pip install crawl4ai[sync]
For contributors who plan to modify the source code:
git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e . # Basic installation in editable mode
Install optional features:
pip install -e ".[torch]" # With PyTorch features
pip install -e ".[transformer]" # With Transformer features
pip install -e ".[cosine]" # With cosine similarity features
pip install -e ".[sync]" # With synchronous crawling (Selenium)
pip install -e ".[all]" # Install all optional features
๐ Now Available! Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.
The new Docker implementation includes:
# Pull and run the latest release
docker pull unclecode/crawl4ai:latest
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest
# Visit the monitoring dashboard at http://localhost:11235/dashboard
# Or the playground at http://localhost:11235/playground
Run a quick test (works for both Docker options):
import requests
# Submit a crawl job
response = requests.post(
"http://localhost:11235/crawl",
json={"urls": ["https://example.com"], "priority": 10}
)
if response.status_code == 200:
print("Crawl job submitted successfully.")
if "results" in response.json():
results = response.json()["results"]
print("Crawl job completed. Results:")
for result in results:
print(result)
else:
task_id = response.json()["task_id"]
print(f"Crawl job submitted. Task ID:: {task_id}")
result = requests.get(f"http://localhost:11235/task/{task_id}")
For more examples, see our Docker Examples. For advanced configuration, monitoring features, and production deployment, see our Self-Hosting Guide.
</details>You can check the project structure in the directory docs/examples. Over there, you can find a variety of examples; here, some popular examples are shared.
<details> <summary>๐ <strong>Heuristic Markdown Generation with Clean and Fit Markdown</strong></summary>import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
async def main():
browser_config = BrowserConfig(
headless=True,
verbose=True,
)
run_config = CrawlerRunConfig(
cache_mode=CacheMode.ENABLED,
markdown_generator=DefaultMarkdownGenerator(
content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed", min_word_threshold=0)
),
# markdown_generator=DefaultMarkdownGenerator(
# content_filter=BM25ContentFilter(user_query="WHEN_WE_FOCUS_BASED_ON_A_USER_QUERY", bm25_threshold=1.0)
# ),
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(
url="https://docs.micronaut.io/4.9.9/guide/",
config=run_config
)
print(len(result.markdown.raw_markdown))
print(len(result.markdown.fit_markdown))
if __name__ == "__main__":
asyncio.run(main())
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy
import json
async def main():
schema = {
"name": "KidoCode Courses",
"baseSelector": "section.charge-methodology .w-tab-content > div",
"fields": [
{
"name": "section_title",
"selector": "h3.heading-50",
"type": "text",
},
{
"name": "section_description",
"selector": ".charge-content",
"type": "text",
},
{
"name": "course_name",
"selector": ".text-block-93",
"type": "text",
},
{
"name": "course_description",
"selector": ".course-content-text",
"type": "text",
},
{
"name": "course_icon",
"selector": ".image-92",
"type": "attribute",
"attribute": "src"
}
]
}
extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
browser_config = BrowserConfig(
headless=False,
verbose=True
)
run_config = CrawlerRunConfig(
extraction_strategy=extraction_strategy,
js_code=["""(async () => {const tabs = document.querySelectorAll("section.charge-methodology .tabs-menu-3 > div");for(let tab of tabs) {tab.scrollIntoView();tab.click();await new Promise(r => setTimeout(r, 500));}})();"""],
cache_mode=CacheMode.BYPASS
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(
url="https://www.kidocode.com/degrees/technology",
config=run_config
)
companies = json.loads(result.extracted_content)
print(f"Successfully extracted {len(companies)} companies")
print(json.dumps(companies[0], indent=2))
if __name__ == "__main__":
asyncio.run(main())
import os
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy
from pydantic import BaseModel, Field
class OpenAIModelFee(BaseModel):
model_name: str = Field(..., description="Name of the OpenAI model.")
input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
async def main():
browser_config = BrowserConfig(verbose=True)
run_config = CrawlerRunConfig(
word_count_threshold=1,
extraction_strategy=LLMExtractionStrategy(
# Here you can use any provider that Litellm library supports, for instance: ollama/qwen2
# provider="ollama/qwen2", api_token="no-token",
llm_config = LLMConfig(provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY')),
schema=OpenAIModelFee.schema(),
extraction_type="schema",
instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens.
Do not miss any models in the entire content. One extracted model JSON format should look like this:
{"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}."""
),
cache_mode=CacheMode.BYPASS,
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(
url='https://openai.com/api/pricing/',
config=run_config
)
print(result.extracted_content)
if __name__ == "__main__":
asyncio.run(main())
import os, sys
from pathlib import Path
import asyncio, time
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
async def test_news_crawl():
# Create a persistent user data directory
user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
os.makedirs(user_data_dir, exist_ok=True)
browser_config = BrowserConfig(
verbose=True,
headless=True,
user_data_dir=user_data_dir,
use_persistent_context=True,
)
run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS
)
async with AsyncWebCrawler(config=browser_config) as crawler:
url = "ADDRESS_OF_A_CHALLENGING_WEBSITE"
result = await crawler.arun(
url,
config=run_config,
magic=True,
)
print(f"Successfully crawled {url}")
print(f"Content length: {len(result.markdown)}")
๐ก Tip: Some websites may use CAPTCHA based verification mechanisms to prevent automated access. If your workflow encounters such challenges, you may optionally integrate a third-party CAPTCHA-handling service such as <strong>CapSolver</strong>. They support reCAPTCHA v2/v3, Cloudflare Turnstile, Challenge, AWS WAF, and more. Please ensure that your usage complies with the target websiteโs terms of service and applicable laws.
Our biggest release since v0.8.0. Anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes.
๐ก๏ธ Anti-Bot Detection & Proxy Escalation:
from crawl4ai import CrawlerRunConfig
from crawl4ai.async_configs import ProxyConfig
config = CrawlerRunConfig(
proxy_config=[ProxyConfig.DIRECT, ProxyConfig(server="http://my-proxy:8080")],
max_retries=2,
fallback_fetch_function=my_web_unlocker,
)
๐ Shadow DOM Flattening:
config = CrawlerRunConfig(flatten_shadow_dom=True)
๐ Deep Crawl Cancellation:
cancel() or should_cancel callbackโ๏ธ Config Defaults API:
set_defaults() / get_defaults() / reset_defaults() on BrowserConfig and CrawlerRunConfig๐ Critical Security Fixes:
/crawl endpoint โ removed eval(), added allowlist60+ Bug Fixes across browser management, proxy, deep crawling, extraction, CLI, and Docker
This release introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments.
๐ Deep Crawl Crash Recovery:
on_state_change callback fires after each URL for real-time state persistenceresume_state parameter to continue from a saved checkpointfrom crawl4ai.deep_crawling import BFSDeepCrawlStrategy
strategy = BFSDeepCrawlStrategy(
max_depth=3,
resume_state=saved_state, # Continue from checkpoint
on_state_change=save_to_redis, # Called after each URL
)
โก Prefetch Mode for Fast URL Discovery:
prefetch=True skips markdown, extraction, and media processingconfig = CrawlerRunConfig(prefetch=True)
result = await crawler.arun("https://example.com", config=config)
# Returns HTML and links only - no markdown generation
๐ Security Fixes (Docker API):
CRAWL4AI_HOOKS_ENABLED=false)file:// URLs blocked on API endpoints to prevent LFI__import__ removed from hook execution sandboxThis release focuses on stability with 11 bug fixes addressing issues reported by the community. No new features, but significant improvements to reliability.
๐ณ Docker API Fixes:
ContentRelevanceFilter deserialization in deep crawl requests (#1642)ProxyConfig JSON serialization in BrowserConfig.to_dict() (#1629).cache folder permissions in Docker image (#1638)๐ค LLM Extraction Improvements:
LLMConfig parameters (#1269):
from crawl4ai import LLMConfig
config = LLMConfig(
provider="openai/gpt-4o-mini",
backoff_base_delay=5, # Wait 5s on first retry
backoff_max_attempts=5, # Try up to 5 times
backoff_exponential_factor=3 # Multiply delay by 3 each attempt
)
LLMExtractionStrategy (#1178):
from crawl4ai import LLMExtractionStrategy
strategy = LLMExtractionStrategy(
llm_config=config,
instruction="Extract table data",
input_format="html" # Now supports: "html", "markdown", "fit_markdown"
)
"Raw HTML" instead of HTML blob (#1116)๐ URL Handling:
๐ฆ Dependency Updates:
๐ง AdaptiveCrawler:
๐ Real-time Monitoring Dashboard: Interactive web UI with live system metrics and browser pool visibility
# Access the monitoring dashboard
# Visit: http://localhost:11235/dashboard
# Real-time metrics include:
# - System health (CPU, memory, network, uptime)
# - Active and completed request tracking
# - Browser pool management (permanent/hot/cold)
# - Janitor cleanup events
# - Error monitoring with full context
๐ Comprehensive Monitor API: Complete REST API for programmatic access to all monitoring data
import httpx
async with httpx.AsyncClient() as client:
# System health
health = await client.get("http://localhost:11235/monitor/health")
# Request tracking
requests = await client.get("http://localhost:11235/monitor/requests")
# Browser pool status
browsers = await client.get("http://localhost:11235/monitor/browsers")
# Endpoint statistics
stats = await client.get("http://localhost:11235/monitor/endpoints/stats")
โก WebSocket Streaming: Real-time updates every 2 seconds for custom dashboards
๐ฅ Smart Browser Pool: 3-tier architecture (permanent/hot/cold) with automatic promotion and cleanup
๐งน Janitor System: Automatic resource management with event logging
๐ฎ Control Actions: Manual browser management (kill, restart, cleanup) via API
๐ Production Metrics: 6 critical metrics for operational excellence with Prometheus integration
๐ Critical Bug Fixes:
๐ง Docker Hooks System: Complete pipeline customization with user-provided Python functions at 8 key points
โจ Function-Based Hooks API (NEW): Write hooks as regular Python functions with full IDE support:
from crawl4ai import hooks_to_string
from crawl4ai.docker_client import Crawl4aiDockerClient
# Define hooks as regular Python functions
async def on_page_context_created(page, context, **kwargs):
"""Block images to speed up crawling"""
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
await page.set_viewport_size({"width": 1920, "height": 1080})
return page
async def before_goto(page, context, url, **kwargs):
"""Add custom headers"""
await page.set_extra_http_headers({'X-Crawl4AI': 'v0.7.5'})
return page
# Option 1: Use hooks_to_string() utility for REST API
hooks_code = hooks_to_string({
"on_page_context_created": on_page_context_created,
"before_goto": before_goto
})
# Option 2: Docker client with automatic conversion (Recommended)
client = Crawl4aiDockerClient(base_url="http://localhost:11235")
results = await client.crawl(
urls=["https://httpbin.org/html"],
hooks={
"on_page_context_created": on_page_context_created,
"before_goto": before_goto
}
)
# โ Full IDE support, type checking, and reusability!
๐ค Enhanced LLM Integration: Custom providers with temperature control and base_url configuration
๐ HTTPS Preservation: Secure internal link handling with preserve_https_for_internal_links=True
๐ Python 3.10+ Support: Modern language features and enhanced performance
๐ ๏ธ Bug Fixes: Resolved multiple community-reported issues including URL processing, JWT authentication, and proxy configuration
๐ LLMTableExtraction: Revolutionary table extraction with intelligent chunking for massive tables:
from crawl4ai import LLMTableExtraction, LLMConfig
# Configure intelligent table extraction
table_strategy = LLMTableExtraction(
llm_config=LLMConfig(provider="openai/gpt-4.1-mini"),
enable_chunking=True, # Handle massive tables
chunk_token_threshold=5000, # Smart chunking threshold
overlap_threshold=100, # Maintain context between chunks
extraction_type="structured" # Get structured data output
)
config = CrawlerRunConfig(table_extraction_strategy=table_strategy)
result = await crawler.arun("https://complex-tables-site.com", config=config)
# Tables are automatically chunked, processed, and merged
for table in result.tables:
print(f"Extracted table: {len(table['data'])} rows")
โก Dispatcher Bug Fix: Fixed sequential processing bottleneck in arun_many for fast-completing tasks
๐งน Memory Management Refactor: Consolidated memory utilities into main utils module for cleaner architecture
๐ง Browser Manager Fixes: Resolved race conditions in concurrent page creation with thread-safe locking
๐ Advanced URL Processing: Better handling of raw:// URLs and base tag link resolution
๐ก๏ธ Enhanced Proxy Support: Flexible proxy configuration supporting both dict and string formats
๐ต๏ธ Undetected Browser Support: Bypass sophisticated bot detection systems:
from crawl4ai import AsyncWebCrawler, BrowserConfig
browser_config = BrowserConfig(
browser_type="undetected", # Use undetected Chrome
headless=True, # Can run headless with stealth
extra_args=[
"--disable-blink-features=AutomationControlled",
"--disable-web-security"
]
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun("https://protected-site.com")
# Successfully bypass Cloudflare, Akamai, and custom bot detection
๐จ Multi-URL Configuration: Different strategies for different URL patterns in one batch:
from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode
configs = [
# Documentation sites - aggressive caching
CrawlerRunConfig(
url_matcher=["*docs*", "*documentation*"],
cache_mode=CacheMode.WRITE_ONLY,
markdown_generator_options={"include_links": True}
),
# News/blog sites - fresh content
CrawlerRunConfig(
url_matcher=lambda url: 'blog' in url or 'news' in url,
cache_mode=CacheMode.BYPASS
),
# Fallback for everything else
CrawlerRunConfig()
]
results = await crawler.arun_many(urls, config=configs)
# Each URL gets the perfect configuration automatically
๐ง Memory Monitoring: Track and optimize memory usage during crawling:
from crawl4ai.memory_utils import MemoryMonitor
monitor = MemoryMonitor()
monitor.start_monitoring()
results = await crawler.arun_many(large_url_list)
report = monitor.get_report()
print(f"Peak memory: {report['peak_mb']:.1f} MB")
print(f"Efficiency: {report['efficiency']:.1f}%")
# Get optimization recommendations
๐ Enhanced Table Extraction: Direct DataFrame conversion from web tables:
result = await crawler.arun("https://site-with-tables.com")
# New way - direct table access
if result.tables:
import pandas as pd
for table in result.tables:
df = pd.DataFrame(table['data'])
print(f"Table: {df.shape[0]} rows ร {df.shape[1]} columns")
๐ฐ GitHub Sponsors: 4-tier sponsorship system for project sustainability
๐ณ Docker LLM Flexibility: Configure providers via environment variables
๐ง Adaptive Crawling: Your crawler now learns and adapts to website patterns automatically:
config = AdaptiveConfig(
confidence_threshold=0.7, # Min confidence to stop crawling
max_depth=5, # Maximum crawl depth
max_pages=20, # Maximum number of pages to crawl
strategy="statistical"
)
async with AsyncWebCrawler() as crawler:
adaptive_crawler = AdaptiveCrawler(crawler, config)
state = await adaptive_crawler.digest(
start_url="https://news.example.com",
query="latest news content"
)
# Crawler learns patterns and improves extraction over time
๐ Virtual Scroll Support: Complete content extraction from infinite scroll pages:
scroll_config = VirtualScrollConfig(
container_selector="[data-testid='feed']",
scroll_count=20,
scroll_by="container_height",
wait_after_scroll=1.0
)
result = await crawler.arun(url, config=CrawlerRunConfig(
virtual_scroll_config=scroll_config
))
๐ Intelligent Link Analysis: 3-layer scoring system for smart link prioritization:
link_config = LinkPreviewConfig(
query="machine learning tutorials",
score_threshold=0.3,
concurrent_requests=10
)
result = await crawler.arun(url, config=CrawlerRunConfig(
link_preview_config=link_config,
score_links=True
))
# Links ranked by relevance and quality
๐ฃ Async URL Seeder: Discover thousands of URLs in seconds:
seeder = AsyncUrlSeeder(SeedingConfig(
source="sitemap+cc",
pattern="*/blog/*",
query="python tutorials",
score_threshold=0.4
))
urls = await seeder.discover("https://example.com")
โก Performance Boost: Up to 3x faster with optimized resource handling and memory efficiency
Read the full details in our 0.7.0 Release Notes or check the CHANGELOG.
</details>Crawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.
<details> <summary>๐ <strong>Version Numbers Explained</strong></summary>Our version numbers follow this pattern: MAJOR.MINOR.PATCH (e.g., 0.4.3)
We use different suffixes to indicate development stages:
dev (0.4.3dev1): Development versions, unstablea (0.4.3a1): Alpha releases, experimental featuresb (0.4.3b1): Beta releases, feature complete but needs testingrc (0.4.3): Release candidates, potential final versionRegular installation (stable version):
pip install -U crawl4ai
Install pre-release versions:
pip install crawl4ai --pre
Install specific version:
pip install crawl4ai==0.4.3b1
We use pre-releases to:
For production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the --pre flag.
๐จ Documentation Update Alert: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!
For current documentation, including installation instructions, advanced features, and API reference, visit our Documentation Website.
To check our development plans and upcoming features, visit our Roadmap.
<details> <summary>๐ <strong>Development TODOs</strong></summary>We welcome contributions from the open-source community. Check out our contribution guidelines for more information.
I'll help modify the license section with badges. For the halftone effect, here's a version with it:
Here's the updated license section:
This project is licensed under the Apache License 2.0, attribution is recommended via the badges below. See the Apache 2.0 License file for details.
When using Crawl4AI, you must include one of the following attribution methods:
<details> <summary>๐ <strong>1. Badge Attribution (Recommended)</strong></summary> Add one of these badges to your README, documentation, or website:| Theme | Badge |
|---|---|
| Disco Theme (Animated) | <a href="https://github.com/unclecode/crawl4ai"></a> |
| Night Theme (Dark with Neon) | <a href="https://github.com/unclecode/crawl4ai"></a> |
| Dark Theme (Classic) | <a href="https://github.com/unclecode/crawl4ai"></a> |
| Light Theme (Classic) | <a href="https://github.com/unclecode/crawl4ai"></a> |
HTML code for adding the badges:
<!-- Disco Theme (Animated) -->
<a href="https://github.com/unclecode/crawl4ai">
</a>
<!-- Night Theme (Dark with Neon) -->
<a href="https://github.com/unclecode/crawl4ai">
</a>
<!-- Dark Theme (Classic) -->
<a href="https://github.com/unclecode/crawl4ai">
</a>
<!-- Light Theme (Classic) -->
<a href="https://github.com/unclecode/crawl4ai">
</a>
<!-- Simple Shield Badge -->
<a href="https://github.com/unclecode/crawl4ai">
</a>
If you use Crawl4AI in your research or project, please cite:
@software{crawl4ai2024,
author = {UncleCode},
title = {Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper},
year = {2024},
publisher = {GitHub},
journal = {GitHub Repository},
howpublished = {\url{https://github.com/unclecode/crawl4ai}},
commit = {Please use the commit hash you're working with}
}
Text citation format:
UncleCode. (2024). Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper [Computer software].
GitHub. https://github.com/unclecode/crawl4ai
For questions, suggestions, or feedback, feel free to reach out:
Happy Crawling! ๐ธ๏ธ๐
Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.
We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.
<details> <summary>๐ <strong>Key Opportunities</strong></summary>For more details, see our full mission statement.
</details>Our enterprise sponsors and technology partners help scale Crawl4AI to power production-grade data pipelines.
| Company | About | Sponsorship Tier |
|---|---|---|
| <a href="https://www.thordata.com/?ls=github&lk=crawl4ai" target="_blank"></a> | Leveraging Thordata ensures seamless compatibility with any AI/ML workflows and data infrastructure, massively accessing web data with 99.9% uptime, backed by one-on-one customer support. | ๐ฅ Silver |
| <a href="https://app.nstproxy.com/register?i=ecOqW9" target="_blank"><picture><source width="250" media="(prefers-color-scheme: dark)" srcset="https://gist.github.com/aravindkarnam/62f82bd4818d3079d9dd3c31df432cf8/raw/nst-light.svg"><source width="250" media="(prefers-color-scheme: light)" srcset="https://www.nstproxy.com/logo.svg"></picture></a> | NstProxy is a trusted proxy provider with over 110M+ real residential IPs, city-level targeting, 99.99% uptime, and low pricing at $0.1/GB, it delivers unmatched stability, scale, and cost-efficiency. | ๐ฅ Silver |
| <a href="https://app.scrapeless.com/passport/register?utm_source=official&utm_term=crawl4ai" target="_blank"><picture><source width="250" media="(prefers-color-scheme: dark)" srcset="https://gist.githubusercontent.com/aravindkarnam/0d275b942705604263e5c32d2db27bc1/raw/Scrapeless-light-logo.svg"><source width="250" media="(prefers-color-scheme: light)" srcset="https://gist.githubusercontent.com/aravindkarnam/22d0525cc0f3021bf19ebf6e11a69ccd/raw/Scrapeless-dark-logo.svg"></picture></a> | Scrapeless provides production-grade infrastructure for Crawling, Automation, and AI Agents, offering Scraping Browser, 4 Proxy Types and Universal Scraping API. | ๐ฅ Silver |
| <a href="https://dashboard.capsolver.com/passport/register?inviteCode=ESVSECTX5Q23" target="_blank"><picture><source width="120" media="(prefers-color-scheme: dark)" srcset="https://docs.crawl4ai.com/uploads/sponsors/20251013045338_72a71fa4ee4d2f40.png"><source width="120" media="(prefers-color-scheme: light)" srcset="https://www.capsolver.com/assets/images/logo-text.png"></picture></a> | AI-powered Captcha solving service. Supports all major Captcha types, including reCAPTCHA, Cloudflare, and more | ๐ฅ Bronze |
| <a href="https://kipo.ai" target="_blank"></a> | Helps engineers and buyers find, compare, and source electronic & industrial parts in seconds, with specs, pricing, lead times & alternatives. | ๐ฅ Gold |
| <a href="https://www.kidocode.com/" target="_blank"><p align="center">KidoCode</p></a> | Kidocode is a hybrid technology and entrepreneurship school for kids aged 5โ18, offering both online and on-campus education. | ๐ฅ Gold |
| <a href="https://www.alephnull.sg/" target="_blank"></a> | Singapore-based Aleph Null is Asiaโs leading edtech hub, dedicated to student-centric, AI-driven educationโempowering learners with the tools to thrive in a fast-changing world. | ๐ฅ Gold |
A heartfelt thanks to our individual supporters! Every contribution helps us keep our opensource mission alive and thriving!
<p align="left"> <a href="https://github.com/hafezparast"></a> <a href="https://github.com/ntohidi"></a> <a href="https://github.com/Sjoeborg"></a> <a href="https://github.com/romek-rozen"></a> <a href="https://github.com/Kourosh-Kiyani"></a> <a href="https://github.com/Etherdrake"></a> <a href="https://github.com/shaman247"></a> <a href="https://github.com/work-flow-manager"></a> </p>Want to join them? Sponsor Crawl4AI โ