Zyte Serp Reader

Zyte Serp Reader allows you to access the organic results from google search. Given a query string, it provides the URLs of the top search results and the text string associated with those.

python

# %pip install llama-index llama-index-readers-zyte-serp

In this notebook we show how Zyte Serp Reader (along with web reader) can be used collect information about a particular topic. Given these documents we can perform queries on this topic.

Recently the Govt. of Ireland announced fiscal budget for 2024 and here we show how we can query information regarding the budget. First we get the relevant information using the Zyte Serp Reader, then the information from these URLs is extracted using web reader and finally a queries are answered using a openai chatgpt model.

python

import os
from llama_index.readers.zyte_serp import ZyteSerpReader
from llama_index.readers.web.zyte_web.base import ZyteWebReader

python

# This is needed to run it in juypter notebook
# import nest_asyncio
# nest_asyncio.apply()

python

zyte_api_key = os.environ.get("ZYTE_API_KEY")

Get relevant resources (using ZyteSerp)

Given a topic, we use the search results from google to get the links to the relevant pages.

python

topic = "Ireland Budget 2025"

python

serp_reader = ZyteSerpReader(api_key=zyte_api_key)

python

search_results = serp_reader.load_data(topic)

python

len(search_results)

python

for r in search_results[:4]:
    print(r.text)
    print(r.metadata)

python

urls = [r.text for r in search_results]

Seems we have a list of relevant URL with regard to our topic ("Ireland budget 2024"). Metadata also shows the text and rank associated with the search result entry. Next we get the content of these webpages using web reader.

Get topic content

Given the urls of the webpages which contain information about the topic, we get the content. Since the webpages contain a lot of non-relevant content, we can obtain the filtered content using the "article" mode of the ZyteWebReader which returns only the article content of from the webpage.

python

web_reader = ZyteWebReader(api_key=zyte_api_key, mode="article")
documents = web_reader.load_data(urls)

python

print(documents[0].text[:200])

python

len(documents)

Query engine

Here a very basic query is performed using VectorStoreIndex. Please make sure that the OPENAI_API_KEY env variable is set before running the following code.

python

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents)

python

query_engine = index.as_query_engine()
response = query_engine.query(
    "What kind of energy credits are provided in the budget?"
)
print(response)