Parallel Web Systems Tool

This notebook demonstrates how to use the Parallel Web Systems tool integration with LlamaIndex.

The tool provides access to Parallel AI's Search and Extract APIs:

Search API: Returns structured, compressed excerpts from web search results optimized for LLM consumption
Extract API: Converts public URLs into clean, LLM-optimized markdown

Installation

python

%pip install llama-index-tools-parallel-web-systems

Setup

Get your API key from Parallel AI Platform

python

import os
from getpass import getpass

# Set your API key
if not os.environ.get("PARALLEL_API_KEY"):
    os.environ["PARALLEL_API_KEY"] = getpass("Enter your Parallel AI API key: ")

Initialize the Tool

python

import os
from llama_index.tools.parallel_web_systems import ParallelWebSystemsToolSpec

# Initialize the tool
parallel_tool = ParallelWebSystemsToolSpec(
    api_key=os.environ["PARALLEL_API_KEY"]
)

Search API Example

Search the web using natural language objectives or keyword queries.

python

# Search with an objective
results = parallel_tool.search(
    objective="What are the latest developments in renewable energy?",
    max_results=5,
)

print(f"Found {len(results)} results\n")

for i, doc in enumerate(results, 1):
    print(f"--- Result {i} ---")
    print(f"Title: {doc.metadata.get('title', 'N/A')}")
    print(f"URL: {doc.metadata.get('url', 'N/A')}")
    print(f"Content preview: {doc.text[:300]}...\n")

python

# Search with keyword queries
results = parallel_tool.search(
    search_queries=["solar power 2024", "wind energy statistics"],
    max_results=3,
    mode="agentic",  # More concise, token-efficient results
)

for doc in results:
    print(f"Title: {doc.metadata.get('title')}")
    print(f"URL: {doc.metadata.get('url')}\n")

Extract API Example

Extract clean, structured content from web pages.

python

# Extract content from URLs with a focused objective
results = parallel_tool.extract(
    urls=["https://en.wikipedia.org/wiki/Artificial_intelligence"],
    objective="What are the main applications of AI?",
)

for doc in results:
    print(f"Title: {doc.metadata.get('title')}")
    print(f"URL: {doc.metadata.get('url')}")
    print(f"\nContent:\n{doc.text[:1000]}...")

python

# Extract full content from a URL
results = parallel_tool.extract(
    urls=["https://docs.llamaindex.ai/en/stable/"],
    full_content=True,
    excerpts=False,
)

if results:
    print(f"Extracted {len(results[0].text)} characters of content")
    print(f"\nPreview:\n{results[0].text[:500]}...")

Using with a LlamaIndex Agent

You can use the tool with a LlamaIndex agent for automated web research.

python

# Install OpenAI for the agent (optional)
%pip install llama-index-llms-openai llama-index-core

python

import os
from getpass import getpass

# Set OpenAI API key for the agent
if not os.environ.get("OPENAI_API_KEY"):
    os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ")

python

from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

# Create an agent with the Parallel Web Systems tool
agent = FunctionAgent(
    tools=parallel_tool.to_tool_list(),
    llm=OpenAI(model="gpt-4o"),
)

# Use the agent to perform web research
response = await agent.run(
    "Search the web for the latest news about LlamaIndex and summarize the key points."
)
print(response)