.. _Configuration:

Additional Parameters

It is possible to customize the behavior of the graphs by setting some configuration options. Some interesting ones are:

• verbose: If set to True, some debug information will be printed to the console.
• headless: If set to False, the web browser will be opened on the URL requested and close right after the HTML is fetched.
• max_results: The maximum number of results to be fetched from the search engine. Useful in SearchGraph.
• output_path: The path where the output files will be saved. Useful in SpeechGraph.
• loader_kwargs: A dictionary with additional parameters to be passed to the Loader class, such as proxy.
• burr_kwargs: A dictionary with additional parameters to enable Burr graphical user interface.
• max_images: The maximum number of images to be analyzed. Useful in OmniScraperGraph and OmniSearchGraph.
• cache_path: The path where the cache files will be saved. If already exists, the cache will be loaded from this path.
• additional_info: Add additional text to default prompts defined in the graphs. .. _Burr:

Burr Integration ^^^^^^^^^^^^^^^^

Burr is an open source python library that allows the creation and management of state machine applications. Discover more about it here <https://github.com/DAGWorks-Inc/burr>_. It is possible to enable a local hosted webapp to visualize the scraping pipelines and the data flow. First, we need to install the burr library as follows:

.. code-block:: bash

pip install scrapegraphai[burr]

and then run the graphical user interface as follows:

.. code-block:: bash

burr

To log your graph execution in the platform, you need to set the burr_kwargs parameter in the graph configuration as follows:

.. code-block:: python

graph_config = {
    "llm":{...},
    "burr_kwargs": {
        "project_name": "test-scraper",
        "app_instance_id":"some_id",
    }
}

.. _Proxy:

Proxy Rotation ^^^^^^^^^^^^^^

It is possible to rotate the proxy by setting the proxy option in the graph configuration. We provide a free proxy service which is based on free-proxy <https://pypi.org/project/free-proxy/>_ library and can be used as follows:

.. code-block:: python

graph_config = {
    "llm":{...},
    "loader_kwargs": {
        "proxy" : {
            "server": "broker",
            "criteria": {
                "anonymous": True,
                "secure": True,
                "countryset": {"IT"},
                "timeout": 10.0,
                "max_shape": 3
            },
        },
    },
}

Do you have a proxy server? You can use it as follows:

.. code-block:: python

graph_config = {
    "llm":{...},
    "loader_kwargs": {
        "proxy" : {
            "server": "http://your_proxy_server:port",
            "username": "your_username",
            "password": "your_password",
        },
    },
}