ContextQMD
Back to Llama Index

Refine with Structured Answer Filtering

docs/examples/response_synthesizers/structured_refine.ipynb

0.14.213.8 KB
Original Source

<a href="https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/response_synthesizers/structured_refine.ipynb" target="_parent"></a>

Refine with Structured Answer Filtering

When using our Refine response synthesizer for response synthesis, it's crucial to filter out non-answers. An issue often encountered is the propagation of a single unhelpful response like "I don't have the answer", which can persist throughout the synthesis process and lead to a final answer of the same nature. This can occur even when there are actual answers present in other, more relevant sections.

These unhelpful responses can be filtered out by setting structured_answer_filtering to True. It is set to False by default since this currently only works best if you are using an OpenAI model that supports function calling.

If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

python
%pip install llama-index-llms-openai
python
!pip install llama-index

Load Data

python
texts = [
    "The president in the year 2040 is John Cena.",
    "The president in the year 2050 is Florence Pugh.",
    'The president in the year 2060 is Dwayne "The Rock" Johnson.',
]

Summarize

python
import os

os.environ["OPENAI_API_KEY"] = "sk-..."
python
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-3.5-turbo-0613")
python
from llama_index.core import get_response_synthesizer

summarizer = get_response_synthesizer(
    response_mode="refine", llm=llm, verbose=True
)
python
response = summarizer.get_response("who is president in the year 2050?", texts)

Failed Result

As you can see, we weren't able to get the correct answer from the input texts strings since the initial "I don't know" answer propogated through till the end of the response synthesis.

python
print(response)

Now we'll try again with structured_answer_filtering=True

python
from llama_index.core import get_response_synthesizer

summarizer = get_response_synthesizer(
    response_mode="refine",
    llm=llm,
    verbose=True,
    structured_answer_filtering=True,
)
python
response = summarizer.get_response("who is president in the year 2050?", texts)

Successful Result

As you can see, we were able to determine the correct answer from the given context by filtering the texts strings for the ones that actually contained the answer to our question.

python
print(response)

Non Function-calling LLMs

You may want to make use of this filtering functionality with an LLM that doesn't offer a function calling API.

In that case, the Refine module will automatically switch to using a structured output Program that doesn't rely on an external function calling API.

python
# we'll stick with OpenAI but use an older model that does not support function calling
instruct_llm = OpenAI(model="gpt-3.5-turbo-instruct")
python
from llama_index.core import get_response_synthesizer

summarizer = get_response_synthesizer(
    response_mode="refine",
    llm=instruct_llm,
    verbose=True,
    structured_answer_filtering=True,
)
python
response = summarizer.get_response("who is president in the year 2050?", texts)
print(response)

CompactAndRefine

Since CompactAndRefine is built on top of Refine, this response mode also supports structured answer filtering.

python
from llama_index.core import get_response_synthesizer

summarizer = get_response_synthesizer(
    response_mode="compact",
    llm=instruct_llm,
    verbose=True,
    structured_answer_filtering=True,
)
python
response = summarizer.get_response("who is president in the year 2050?", texts)
print(response)