1. BrowserConfig – Controlling the Browser

BrowserConfig focuses on how the browser is launched and behaves. This includes headless mode, proxies, user agents, and other environment tweaks.

from crawl4ai import AsyncWebCrawler, BrowserConfig

browser_cfg = BrowserConfig(
    browser_type="chromium",
    headless=True,
    viewport_width=1280,
    viewport_height=720,
    proxy_config="http://user:pass@proxy:8080",
    user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36",
)

1.1 Parameter Highlights

Tips:

• Set headless=False to visually debug how pages load or how interactions proceed.
• If you need authentication storage or repeated sessions, consider use_persistent_context=True and specify user_data_dir.
• For large pages, you might need a bigger viewport_width and viewport_height to handle dynamic content.

2. CrawlerRunConfig – Controlling Each Crawl

While BrowserConfig sets up the environment, CrawlerRunConfig details how each crawl operation should behave: caching, content filtering, link or domain blocking, timeouts, JavaScript code, etc.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

run_cfg = CrawlerRunConfig(
    wait_for="css:.main-content",
    word_count_threshold=15,
    excluded_tags=["nav", "footer"],
    exclude_external_links=True,
    stream=True,  # Enable streaming for arun_many()
)

2.1 Parameter Highlights

We group them by category.

A) Content Processing

B) Browser Location and Identity

C) Caching & Session

Use these for controlling whether you read or write from a local content cache. Handy for large batch crawls or repeated site visits.

D) Page Navigation & Timing

E) Page Interaction

If your page is a single-page app with repeated JS updates, set js_only=True in subsequent calls, plus a session_id for reusing the same tab.

F) Media Handling

G) Link/Domain Handling

Use these for link-level content filtering (often to keep crawls “internal” or to remove spammy domains).

H) Debug, Logging & Network Monitoring

I) Connection & HTTP Parameters

J) Virtual Scroll Configuration

When sites use virtual scrolling (content replaced as you scroll), use VirtualScrollConfig:

from crawl4ai import VirtualScrollConfig

virtual_config = VirtualScrollConfig(
    container_selector="#timeline",    # CSS selector for scrollable container
    scroll_count=30,                   # Number of times to scroll
    scroll_by="container_height",      # How much to scroll: "container_height", "page_height", or pixels (e.g. 500)
    wait_after_scroll=0.5             # Seconds to wait after each scroll for content to load
)

config = CrawlerRunConfig(
    virtual_scroll_config=virtual_config
)

VirtualScrollConfig Parameters:

When to use Virtual Scroll vs scan_full_page:

• Use virtual_scroll_config when content is replaced during scroll (Twitter, Instagram)
• Use scan_full_page when content is appended during scroll (traditional infinite scroll)

See Virtual Scroll documentation for detailed examples.

K) URL Matching Configuration

The url_matcher parameter enables URL-specific configurations when used with arun_many():

from crawl4ai import CrawlerRunConfig, MatchMode
from crawl4ai.processors.pdf import PDFContentScrapingStrategy
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

# Simple string pattern (glob-style)
pdf_config = CrawlerRunConfig(
    url_matcher="*.pdf",
    scraping_strategy=PDFContentScrapingStrategy()
)

# Multiple patterns with OR logic (default)
blog_config = CrawlerRunConfig(
    url_matcher=["*/blog/*", "*/article/*", "*/news/*"],
    match_mode=MatchMode.OR  # Any pattern matches
)

# Function matcher
api_config = CrawlerRunConfig(
    url_matcher=lambda url: 'api' in url or url.endswith('.json'),
    # Other settings like extraction_strategy
)

# Mixed: String + Function with AND logic
complex_config = CrawlerRunConfig(
    url_matcher=[
        lambda url: url.startswith('https://'),  # Must be HTTPS
        "*.org/*",                               # Must be .org domain
        lambda url: 'docs' in url                # Must contain 'docs'
    ],
    match_mode=MatchMode.AND  # ALL conditions must match
)

# Combined patterns and functions with AND logic
secure_docs = CrawlerRunConfig(
    url_matcher=["https://*", lambda url: '.doc' in url],
    match_mode=MatchMode.AND  # Must be HTTPS AND contain .doc
)

# Default config - matches ALL URLs
default_config = CrawlerRunConfig()  # No url_matcher = matches everything

UrlMatcher Types:

• None (default): When url_matcher is None or not set, the config matches ALL URLs
• String patterns: Glob-style patterns like "*.pdf", "*/api/*", "https://*.example.com/*"
• Functions: lambda url: bool - Custom logic for complex matching
• Lists: Mix strings and functions, combined with MatchMode.OR or MatchMode.AND

Important Behavior:

• When passing a list of configs to arun_many(), URLs are matched against each config's url_matcher in order. First match wins!
• If no config matches a URL and there's no default config (one without url_matcher), the URL will fail with "No matching configuration found"
• Always include a default config as the last item if you want to handle all URLs

L) Advanced Crawling Features

Deep Crawl Strategy enables automatic site exploration by following links according to defined rules. Useful for sitemap generation or comprehensive site archiving.

Link Preview Config allows efficient link discovery and scoring by fetching only the <head> section of linked pages, enabling smart crawl prioritization without the overhead of full page loads.

Experimental parameters are features in beta testing. They may change or be removed in future versions. Check documentation for currently available experimental features.

2.2 Helper Methods

Both BrowserConfig and CrawlerRunConfig provide a clone() method to create modified copies:

# Create a base configuration
base_config = CrawlerRunConfig(
    cache_mode=CacheMode.ENABLED,
    word_count_threshold=200
)

# Create variations using clone()
stream_config = base_config.clone(stream=True)
no_cache_config = base_config.clone(
    cache_mode=CacheMode.BYPASS,
    stream=True
)

The clone() method is particularly useful when you need slightly different configurations for different use cases, without modifying the original config.

Class-Level Defaults (set_defaults / get_defaults / reset_defaults)

Both config classes support class-level default overrides. When deploying in a server or cloud context, this eliminates the need to pass the same parameters at every call site.

Resolution order: explicit arg > class-level default > hardcoded default

from crawl4ai import BrowserConfig, CrawlerRunConfig

# Set once at application startup
BrowserConfig.set_defaults(
    cache_cdp_connection=True,
    cdp_close_delay=0,
    create_isolated_context=True,
)
CrawlerRunConfig.set_defaults(verbose=False)

# All new instances inherit the class defaults
cfg1 = BrowserConfig(cdp_url="ws://localhost:9222")
# → cache_cdp_connection=True, cdp_close_delay=0

cfg2 = BrowserConfig(cdp_url="ws://localhost:9222", cache_cdp_connection=False)
# → cache_cdp_connection=False (explicit value wins)

# Inspect current defaults
BrowserConfig.get_defaults()
# → {"cache_cdp_connection": True, "cdp_close_delay": 0, "create_isolated_context": True}

# Remove a single default
BrowserConfig.reset_defaults("cdp_close_delay")

# Remove all defaults
BrowserConfig.reset_defaults()

API Reference:

Notes:

• Defaults are independent per class — BrowserConfig.set_defaults() has no effect on CrawlerRunConfig.
• Mutable values (lists, dicts) are deep-copied on storage and on each instance creation, so instances do not share objects.
• clone(), dump()/load(), and from_kwargs() all work correctly with class defaults — serialized data is self-contained and independent of the current class defaults.
• Defaults are stored in memory for the lifetime of the process. They are not persisted to disk.

2.3 Example Usage

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    # Configure the browser
    browser_cfg = BrowserConfig(
        headless=False,
        viewport_width=1280,
        viewport_height=720,
        proxy_config="http://user:pass@myproxy:8080",
        text_mode=True
    )

    # Configure the run
    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        session_id="my_session",
        css_selector="main.article",
        excluded_tags=["script", "style"],
        exclude_external_links=True,
        wait_for="css:.article-loaded",
        screenshot=True,
        stream=True
    )

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(
            url="https://example.com/news",
            config=run_cfg
        )
        if result.success:
            print("Final cleaned_html length:", len(result.cleaned_html))
            if result.screenshot:
                print("Screenshot captured (base64, length):", len(result.screenshot))
        else:
            print("Crawl failed:", result.error_message)

if __name__ == "__main__":
    asyncio.run(main())

2.4 Compliance & Ethics

run_config = CrawlerRunConfig(
    check_robots_txt=True,  # Enable robots.txt compliance
    user_agent="MyBot/1.0"  # Identify your crawler
)

3. LLMConfig - Setting up LLM providers

LLMConfig is useful to pass LLM provider config to strategies and functions that rely on LLMs to do extraction, filtering, schema generation etc. Currently it can be used in the following -

1. LLMExtractionStrategy
2. LLMContentFilter
3. JsonCssExtractionStrategy.generate_schema
4. JsonXPathExtractionStrategy.generate_schema
5. AdaptiveConfig.embedding_llm_config (embedding model for adaptive crawling)
6. AdaptiveConfig.query_llm_config (chat completion model for query expansion in adaptive crawling)

3.1 Parameters

2. API token of LLM provider eg: api_token = "gsk_1ClHGGJ7Lpn4WGybR7vNWGdyb3FY7zXEw3SCiy0BAVM9lL8CQv"
3. Environment variable - use with prefix "env:" eg:api_token = "env: GROQ_API_KEY" | API token to use for the given provider | base_url |Optional. Custom API endpoint | If your provider has a custom endpoint | backoff_base_delay |Optional. int (default: 2) | Seconds to wait before the first retry when the provider throttles a request. | backoff_max_attempts |Optional. int (default: 3) | Total tries (initial call + retries) before surfacing an error. | backoff_exponential_factor |Optional. int (default: 2) | Multiplier that increases the wait time for each retry (delay = base_delay * factor^attempt).

3.2 Example Usage

llm_config = LLMConfig(
    provider="openai/gpt-4o-mini",
    api_token=os.getenv("OPENAI_API_KEY"),
    backoff_base_delay=1, # optional
    backoff_max_attempts=5, # optional
    backoff_exponential_factor=3, # optional
)

4. Putting It All Together

• Use BrowserConfig for global browser settings: engine, headless, proxy, user agent.
• Use CrawlerRunConfig for each crawl’s context: how to filter content, handle caching, wait for dynamic elements, or run JS.
• Pass both configs to AsyncWebCrawler (the BrowserConfig) and then to arun() (the CrawlerRunConfig).
• Use LLMConfig for LLM provider configurations that can be used across all extraction, filtering, schema generation, and adaptive crawling tasks. Can be used in - LLMExtractionStrategy, LLMContentFilter, JsonCssExtractionStrategy.generate_schema, JsonXPathExtractionStrategy.generate_schema, and AdaptiveConfig (embedding_llm_config / query_llm_config)

# Create a modified copy with the clone() method
stream_cfg = run_cfg.clone(
    stream=True,
    cache_mode=CacheMode.BYPASS
)

# Or set project-wide defaults once at startup
BrowserConfig.set_defaults(headless=True, text_mode=True)
CrawlerRunConfig.set_defaults(cache_mode=CacheMode.BYPASS)