docs/blog/release-v0.7.6.md
Release Date: October 22, 2025
I'm excited to announce Crawl4AI v0.7.6, featuring a complete webhook infrastructure for the Docker job queue API! This release eliminates polling and brings real-time notifications to both crawling and LLM extraction workflows.
The headline feature of v0.7.6 is comprehensive webhook support for asynchronous job processing. No more constant polling to check if your jobs are done - get instant notifications when they complete!
Key Capabilities:
/crawl/job and /llm/job endpoints now support webhooksconfig.yml for all jobscrawl and llm_extraction tasksInstead of constantly checking job status:
OLD WAY (Polling):
# Submit job
response = requests.post("http://localhost:11235/crawl/job", json=payload)
task_id = response.json()['task_id']
# Poll until complete
while True:
status = requests.get(f"http://localhost:11235/crawl/job/{task_id}")
if status.json()['status'] == 'completed':
break
time.sleep(5) # Wait and try again
NEW WAY (Webhooks):
# Submit job with webhook
payload = {
"urls": ["https://example.com"],
"webhook_config": {
"webhook_url": "https://myapp.com/webhook",
"webhook_data_in_payload": True
}
}
response = requests.post("http://localhost:11235/crawl/job", json=payload)
# Done! Webhook will notify you when complete
# Your webhook handler receives the results automatically
curl -X POST http://localhost:11235/crawl/job \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://example.com"],
"browser_config": {"headless": true},
"crawler_config": {"cache_mode": "bypass"},
"webhook_config": {
"webhook_url": "https://myapp.com/webhooks/crawl-complete",
"webhook_data_in_payload": false,
"webhook_headers": {
"X-Webhook-Secret": "your-secret-token"
}
}
}'
curl -X POST http://localhost:11235/llm/job \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/article",
"q": "Extract the article title, author, and publication date",
"schema": "{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}",
"provider": "openai/gpt-4o-mini",
"webhook_config": {
"webhook_url": "https://myapp.com/webhooks/llm-complete",
"webhook_data_in_payload": true
}
}'
Success (with data):
{
"task_id": "llm_1698765432",
"task_type": "llm_extraction",
"status": "completed",
"timestamp": "2025-10-22T10:30:00.000000+00:00",
"urls": ["https://example.com/article"],
"data": {
"extracted_content": {
"title": "Understanding Web Scraping",
"author": "John Doe",
"date": "2025-10-22"
}
}
}
Failure:
{
"task_id": "crawl_abc123",
"task_type": "crawl",
"status": "failed",
"timestamp": "2025-10-22T10:30:00.000000+00:00",
"urls": ["https://example.com"],
"error": "Connection timeout after 30s"
}
from flask import Flask, request, jsonify
app = Flask(__name__)
@app.route('/webhook', methods=['POST'])
def handle_webhook():
payload = request.json
task_id = payload['task_id']
task_type = payload['task_type']
status = payload['status']
if status == 'completed':
if 'data' in payload:
# Process data directly
data = payload['data']
else:
# Fetch from API
endpoint = 'crawl' if task_type == 'crawl' else 'llm'
response = requests.get(f'http://localhost:11235/{endpoint}/job/{task_id}')
data = response.json()
# Your business logic here
print(f"Job {task_id} completed!")
elif status == 'failed':
error = payload.get('error', 'Unknown error')
print(f"Job {task_id} failed: {error}")
return jsonify({"status": "received"}), 200
app.run(port=8080)
None! This release is fully backward compatible.
No migration needed! Webhooks are opt-in:
webhook_config to your job payload# Just add webhook_config to your existing payload
payload = {
# Your existing configuration
"urls": ["https://example.com"],
"browser_config": {...},
"crawler_config": {...},
# NEW: Add webhook configuration
"webhook_config": {
"webhook_url": "https://myapp.com/webhook",
"webhook_data_in_payload": True
}
}
webhooks:
enabled: true
default_url: "https://myapp.com/webhooks/default" # Optional
data_in_payload: false
retry:
max_attempts: 5
initial_delay_ms: 1000
max_delay_ms: 32000
timeout_ms: 30000
headers:
User-Agent: "Crawl4AI-Webhook/1.0"
# Pull the latest image
docker pull unclecode/crawl4ai:0.7.6
# Or use latest tag
docker pull unclecode/crawl4ai:latest
# Run with webhook support
docker run -d \
-p 11235:11235 \
--env-file .llm.env \
--name crawl4ai \
unclecode/crawl4ai:0.7.6
pip install --upgrade crawl4ai
Try the release demo:
python docs/releases_review/demo_v0.7.6.py
This comprehensive demo showcases:
Thank you to the community for the feedback that shaped this feature! Special thanks to everyone who requested webhook support for asynchronous job processing.
Happy crawling with webhooks! 🕷️🪝
- unclecode