docs/blog/release-v0.7.7.md
November 14, 2025 โข 10 min read
Today I'm releasing Crawl4AI v0.7.7โthe Self-Hosting & Monitoring Update. This release transforms Crawl4AI Docker from a simple containerized crawler into a complete self-hosting platform with enterprise-grade real-time monitoring, full operational transparency, and production-ready observability.
The Problem: Running Crawl4AI in Docker was like flying blind. Users had no visibility into what was happening inside the containerโmemory usage, active requests, browser pools, or errors. Troubleshooting required checking logs, and there was no way to monitor performance or manually intervene when issues occurred.
My Solution: I built a complete real-time monitoring system with an interactive dashboard, comprehensive REST API, WebSocket streaming, and manual control actions. Now you have full transparency and control over your crawling infrastructure.
Before v0.7.7, Docker was just a containerized crawler. After v0.7.7, it's a complete self-hosting platform that gives you:
Access the dashboard at http://localhost:11235/dashboard to see:
The dashboard updates every 2 seconds via WebSocket, giving you live visibility into your crawling operations.
The Problem: Monitoring dashboards are great for humans, but automation and integration require programmatic access.
My Solution: A comprehensive REST API that exposes all monitoring data for integration with your existing infrastructure.
import httpx
import asyncio
async def monitor_system_health():
async with httpx.AsyncClient() as client:
response = await client.get("http://localhost:11235/monitor/health")
health = response.json()
print(f"Container Metrics:")
print(f" CPU: {health['container']['cpu_percent']:.1f}%")
print(f" Memory: {health['container']['memory_percent']:.1f}%")
print(f" Uptime: {health['container']['uptime_seconds']}s")
print(f"\nBrowser Pool:")
print(f" Permanent: {health['pool']['permanent']['active']} active")
print(f" Hot Pool: {health['pool']['hot']['count']} browsers")
print(f" Cold Pool: {health['pool']['cold']['count']} browsers")
print(f"\nStatistics:")
print(f" Total Requests: {health['stats']['total_requests']}")
print(f" Success Rate: {health['stats']['success_rate_percent']:.1f}%")
print(f" Avg Latency: {health['stats']['avg_latency_ms']:.0f}ms")
asyncio.run(monitor_system_health())
async def track_requests():
async with httpx.AsyncClient() as client:
response = await client.get("http://localhost:11235/monitor/requests")
requests_data = response.json()
print(f"Active Requests: {len(requests_data['active'])}")
print(f"Completed Requests: {len(requests_data['completed'])}")
# See details of recent requests
for req in requests_data['completed'][:5]:
status_icon = "โ
" if req['success'] else "โ"
print(f"{status_icon} {req['endpoint']} - {req['latency_ms']:.0f}ms")
async def monitor_browser_pool():
async with httpx.AsyncClient() as client:
response = await client.get("http://localhost:11235/monitor/browsers")
browsers = response.json()
print(f"Pool Summary:")
print(f" Total Browsers: {browsers['summary']['total_count']}")
print(f" Total Memory: {browsers['summary']['total_memory_mb']} MB")
print(f" Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")
# List all browsers
for browser in browsers['permanent']:
print(f"๐ฅ Permanent: {browser['browser_id'][:8]}... | "
f"Requests: {browser['request_count']} | "
f"Memory: {browser['memory_mb']:.0f} MB")
async def get_endpoint_stats():
async with httpx.AsyncClient() as client:
response = await client.get("http://localhost:11235/monitor/endpoints/stats")
stats = response.json()
print("Endpoint Analytics:")
for endpoint, data in stats.items():
print(f" {endpoint}:")
print(f" Requests: {data['count']}")
print(f" Avg Latency: {data['avg_latency_ms']:.0f}ms")
print(f" Success Rate: {data['success_rate_percent']:.1f}%")
The Monitor API includes these endpoints:
GET /monitor/health - System health with pool statisticsGET /monitor/requests - Active and completed request trackingGET /monitor/browsers - Browser pool details and efficiencyGET /monitor/endpoints/stats - Per-endpoint performance analyticsGET /monitor/timeline?minutes=5 - Time-series data for chartsGET /monitor/logs/janitor?limit=10 - Cleanup activity logsGET /monitor/logs/errors?limit=10 - Error logs with contextPOST /monitor/actions/cleanup - Force immediate cleanupPOST /monitor/actions/kill_browser - Kill specific browserPOST /monitor/actions/restart_browser - Restart browserPOST /monitor/stats/reset - Reset accumulated statisticsThe Problem: Polling the API every few seconds wastes resources and adds latency. Real-time dashboards need instant updates.
My Solution: WebSocket streaming with 2-second update intervals for building custom real-time dashboards.
import websockets
import json
import asyncio
async def monitor_realtime():
uri = "ws://localhost:11235/monitor/ws"
async with websockets.connect(uri) as websocket:
print("Connected to real-time monitoring stream")
while True:
# Receive update every 2 seconds
data = await websocket.recv()
update = json.loads(data)
# Access all monitoring data
print(f"\n--- Update at {update['timestamp']} ---")
print(f"Memory: {update['health']['container']['memory_percent']:.1f}%")
print(f"Active Requests: {len(update['requests']['active'])}")
print(f"Total Browsers: {update['browsers']['summary']['total_count']}")
if update['errors']:
print(f"โ ๏ธ Recent Errors: {len(update['errors'])}")
asyncio.run(monitor_realtime())
Expected Real-World Impact:
The Problem: Creating a new browser for every request is slow and memory-intensive. Traditional browser pools are static and inefficient.
My Solution: A smart 3-tier browser pool that automatically adapts to usage patterns.
import httpx
async def demonstrate_browser_pool():
async with httpx.AsyncClient() as client:
# Request 1-3: Default config โ Uses permanent browser
print("Phase 1: Using permanent browser")
for i in range(3):
await client.post(
"http://localhost:11235/crawl",
json={"urls": [f"https://httpbin.org/html?req={i}"]}
)
print(f" Request {i+1}: Reused permanent browser")
# Request 4-6: Custom viewport โ Cold pool (first use)
print("\nPhase 2: Custom config creates cold pool browser")
viewport_config = {"viewport": {"width": 1280, "height": 720}}
for i in range(4):
await client.post(
"http://localhost:11235/crawl",
json={
"urls": [f"https://httpbin.org/json?v={i}"],
"browser_config": viewport_config
}
)
if i < 2:
print(f" Request {i+1}: Cold pool browser")
else:
print(f" Request {i+1}: Promoted to hot pool! (after 3 uses)")
# Check pool status
response = await client.get("http://localhost:11235/monitor/browsers")
browsers = response.json()
print(f"\nPool Status:")
print(f" Permanent: {len(browsers['permanent'])} (always active)")
print(f" Hot: {len(browsers['hot'])} (frequently used configs)")
print(f" Cold: {len(browsers['cold'])} (on-demand)")
print(f" Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")
asyncio.run(demonstrate_browser_pool())
Pool Tiers:
Expected Real-World Impact:
The Problem: Long-running crawlers accumulate idle browsers and consume memory over time.
My Solution: An automatic janitor system that monitors and cleans up idle resources.
async def monitor_janitor_activity():
async with httpx.AsyncClient() as client:
response = await client.get("http://localhost:11235/monitor/logs/janitor?limit=5")
logs = response.json()
print("Recent Cleanup Activities:")
for log in logs:
print(f" {log['timestamp']}: {log['message']}")
# Example output:
# 2025-11-14 10:30:00: Cleaned up 2 cold pool browsers (idle > 5min)
# 2025-11-14 10:25:00: Browser reuse rate: 85.3%
# 2025-11-14 10:20:00: Hot pool browser promoted (10 requests)
The Problem: Sometimes you need to manually interveneโkill a stuck browser, force cleanup, or restart resources.
My Solution: Manual control actions via the API for operational troubleshooting.
async def force_cleanup():
async with httpx.AsyncClient() as client:
response = await client.post("http://localhost:11235/monitor/actions/cleanup")
result = response.json()
print(f"Cleanup completed:")
print(f" Browsers cleaned: {result.get('cleaned_count', 0)}")
print(f" Memory freed: {result.get('memory_freed_mb', 0):.1f} MB")
async def kill_stuck_browser(browser_id: str):
async with httpx.AsyncClient() as client:
response = await client.post(
"http://localhost:11235/monitor/actions/kill_browser",
json={"browser_id": browser_id}
)
if response.status_code == 200:
print(f"โ
Browser {browser_id} killed successfully")
async def reset_stats():
async with httpx.AsyncClient() as client:
response = await client.post("http://localhost:11235/monitor/stats/reset")
print("๐ Statistics reset for fresh monitoring")
# Export metrics for Prometheus scraping
async def export_prometheus_metrics():
async with httpx.AsyncClient() as client:
health = await client.get("http://localhost:11235/monitor/health")
data = health.json()
# Export in Prometheus format
metrics = f"""
# HELP crawl4ai_memory_usage_percent Memory usage percentage
# TYPE crawl4ai_memory_usage_percent gauge
crawl4ai_memory_usage_percent {data['container']['memory_percent']}
# HELP crawl4ai_request_success_rate Request success rate
# TYPE crawl4ai_request_success_rate gauge
crawl4ai_request_success_rate {data['stats']['success_rate_percent']}
# HELP crawl4ai_browser_pool_count Total browsers in pool
# TYPE crawl4ai_browser_pool_count gauge
crawl4ai_browser_pool_count {data['pool']['permanent']['active'] + data['pool']['hot']['count'] + data['pool']['cold']['count']}
"""
return metrics
async def check_alerts():
async with httpx.AsyncClient() as client:
health = await client.get("http://localhost:11235/monitor/health")
data = health.json()
# Memory alert
if data['container']['memory_percent'] > 80:
print("๐จ ALERT: Memory usage above 80%")
# Trigger cleanup
await client.post("http://localhost:11235/monitor/actions/cleanup")
# Success rate alert
if data['stats']['success_rate_percent'] < 90:
print("๐จ ALERT: Success rate below 90%")
# Check error logs
errors = await client.get("http://localhost:11235/monitor/logs/errors")
print(f"Recent errors: {len(errors.json())}")
# Latency alert
if data['stats']['avg_latency_ms'] > 5000:
print("๐จ ALERT: Average latency above 5s")
CRITICAL_METRICS = {
"memory_usage": {
"current": "container.memory_percent",
"target": "<80%",
"alert_threshold": ">80%",
"action": "Force cleanup or scale"
},
"success_rate": {
"current": "stats.success_rate_percent",
"target": ">95%",
"alert_threshold": "<90%",
"action": "Check error logs"
},
"avg_latency": {
"current": "stats.avg_latency_ms",
"target": "<2000ms",
"alert_threshold": ">5000ms",
"action": "Investigate slow requests"
},
"browser_reuse_rate": {
"current": "browsers.summary.reuse_rate_percent",
"target": ">80%",
"alert_threshold": "<60%",
"action": "Check pool configuration"
},
"total_browsers": {
"current": "browsers.summary.total_count",
"target": "<15",
"alert_threshold": ">20",
"action": "Check for browser leaks"
},
"error_frequency": {
"current": "len(errors)",
"target": "<5/hour",
"alert_threshold": ">10/hour",
"action": "Review error patterns"
}
}
This release includes significant bug fixes that improve stability and performance:
The Problem: LLM extraction was blocking async execution, causing URLs to be processed sequentially instead of in parallel (issue #1055).
The Fix: Resolved the blocking issue to enable true parallel processing for LLM extraction.
# Before v0.7.7: Sequential processing
# After v0.7.7: True parallel processing
async with AsyncWebCrawler() as crawler:
urls = ["url1", "url2", "url3", "url4"]
# Now processes truly in parallel with LLM extraction
results = await crawler.arun_many(
urls,
config=CrawlerRunConfig(
extraction_strategy=LLMExtractionStrategy(...)
)
)
# 4x faster for parallel LLM extraction!
Expected Impact: Major performance improvement for batch LLM extraction workflows.
The Problem: DFS (Depth-First Search) deep crawl strategy had implementation issues.
The Fix: Enhanced DFSDeepCrawlStrategy with proper seen URL tracking and improved documentation.
The Problem: Documentation didn't match the actual async_configs.py implementation.
The Fix: Updated all configuration documentation to accurately reflect the current implementation.
The Problem: Sitemap parsing and URL normalization issues in AsyncUrlSeeder (issue #1559).
The Fix: Added comprehensive tests and fixes for sitemap namespace parsing and URL normalization.
The Problem: The remove_overlay_elements functionality wasn't working (issue #1396).
The Fix: Fixed by properly calling the injected JavaScript function.
The Problem: Viewport configuration wasn't working in managed browsers (issue #1490).
The Fix: Added proper viewport size configuration support for browser launch.
The Problem: CDP (Chrome DevTools Protocol) endpoint verification had timing issues causing connection failures (issue #1445).
The Fix: Added exponential backoff for CDP endpoint verification to handle timing variations.
fit_html property serialization in /crawl and /crawl/stream endpointsNone! This release is fully backward compatible.
# Pull the latest version
docker pull unclecode/crawl4ai:0.7.7
# Or use the latest tag
docker pull unclecode/crawl4ai:latest
# Run with monitoring enabled (default)
docker run -d \
-p 11235:11235 \
--shm-size=1g \
--name crawl4ai \
unclecode/crawl4ai:0.7.7
# Access the monitoring dashboard
open http://localhost:11235/dashboard
# Upgrade to latest version
pip install --upgrade crawl4ai
# Or install specific version
pip install crawl4ai==0.7.7
Run the comprehensive demo that showcases all monitoring features:
python docs/releases_review/demo_v0.7.7.py
The demo includes:
docs/releases_review/demo_v0.7.7.py - Working examples/dashboard to get familiar with the monitoring systemThank you to our community for the feedback, bug reports, and feature requests that shaped this release. Special thanks to everyone who contributed to the issues that were fixed in this version.
The monitoring system was built based on real user needs for production deployments, and your input made it comprehensive and practical.
http://localhost:11235/dashboard (when running)Crawl4AI v0.7.7 delivers complete self-hosting with enterprise-grade monitoring. You now have full visibility and control over your web crawling infrastructure. The monitoring dashboard, comprehensive API, and WebSocket streaming give you everything needed for production deployments. Try the self-hosting platformโit's a game changer for operational excellence!
Happy crawling with full visibility! ๐ท๏ธ๐
- unclecode