Extract table of contents

Extract PageIndex, namely table of contents, from documents to provide long context RAG and improve retrieval.

---

During indexing, this technique uses LLM to extract and generate chapter information, which is added to each chunk to provide sufficient global context. At the retrieval stage, it first uses the chunks matched by search, then supplements missing chunks based on the PageIndex (table of contents) structure. This addresses issues caused by chunk fragmentation and insufficient context, improving answer quality.

:::danger WARNING Enabling PageIndex extraction requires significant memory, computational resources, and tokens. :::

Prerequisites

The system's default chat model is used to summarize clustered content. Before proceeding, ensure that you have a chat model properly configured:

Quickstart

1. Navigate to the Configuration page.

2. Enable PageIndex.

3. To use this technique during retrieval, do either of the following:

   • In the Chat setting panel of your chat app, switch on the PageIndex toggle.
   • If you are using an Agent, click the Retrieval agent component to specify the dataset(s) and switch on the PageIndex toggle.

Frequently asked questions

Will previously parsed files be searched using the directory enhancement feature once I enable PageIndex?

No. Only files parsed after you enable PageIndex will be searched using the directory enhancement feature. To apply this feature to files parsed before enabling PageIndex, you must reparse them.