docs/guides/dataset/configuration.md
The configuration page is used to maintain the core configuration of a dataset. Basic information includes Name, Language, Avatar, Description, Embedding model, PageRank, and Tag sets.
The parsing method determines how a dataset processes uploaded documents and how documents are converted into chunks. Parse type in configuration provides two entries: Built-in and Pipeline.
Built-in means using RAGFlow's built-in document parsing capabilities. Users can choose a suitable parsing method based on the document type. The system then reads document content, splits it into chunks, and generates the structure required for retrieval according to the selected method.
The following built-in parsing methods are available:
Tip: On the built-in parsing configuration page, click Built-in pipeline introduction on the right to view the supported file formats, detailed chunking rules, and examples for each parsing method.
Selection suggestion: choose the parsing method based on the form of the material itself. Use General first for regular documents; use Table first for tabular materials; choose the corresponding method for Q&A collections, manuals, papers, images, audio, and emails to reduce later chunk adjustment costs.
After selecting a Built-in parsing method, you can further configure document parsing, chunk splitting, and content enhancement parameters. Reasonable configuration helps improve the accuracy and recall of later retrieval.
Different built-in methods display different configuration items. Use the current interface as the source of truth.
In addition to the built-in parsers above, you can also use a vision-language model (VLM) that supports PDF parsing. To use a third-party vision model to parse PDFs, first configure the default VLM under Model Providers > Set default model. After configuration, select the corresponding model from the PDF parser drop-down list.
Usually, some vision models whose names contain identifiers such as VL or V support PDF parsing. For documents with complex layouts or mixed text and images, a more capable vision model may produce better parsing results, but it also incurs additional model calls and token consumption. The specific support and parsing effect depend on the model used.
The following configurations are mainly used for built-in methods that need to split text according to specified rules, such as General.
Some built-in methods provide the following content enhancement configurations, such as General, Manual, Paper, Book, and Laws.
When Table is selected, the interface provides a dedicated Column mode configuration used to control how table columns participate in generating chunk content and metadata.
Some built-in methods may also provide configuration for specific file formats, for example:
Different built-in methods use different parsing logic, so not all configuration items appear at the same time. After selecting a built-in method, configure only the parameters displayed in the current interface. In general, for ordinary text documents, focus on chunk splitting parameters such as Recommended chunk size, Delimiter for text, and Overlapped percent (%). For PDFs, papers, books, manuals, or legal documents, focus on PDF parser and the corresponding content enhancement configuration. For table data, focus on Column mode. When you need to further enrich retrieval information, use content enhancement features such as Auto metadata, Auto-keyword, and Auto-question according to actual requirements.
After configuration is complete, click Save to save the settings. When documents are parsed later, the system processes them according to the currently selected built-in method and its configuration.
Pipeline means using a custom Ingestion Pipeline as the dataset's parsing method. It is suitable for scenarios that require custom document processing logic or complex processing flows.
The pipeline must be created and configured in Agent > Ingestion Pipeline in advance. In the dataset, you do not need to configure the processing nodes inside the pipeline again. You only need to select the created pipeline.
Usage:
If no pipeline is currently available, you can use the entry provided in the pipeline area to go to Agent and create one.
Note: For pipeline creation, node configuration, and flow orchestration, see Ingestion Pipeline.
Auto Metadata is used to configure metadata automatically generated during document parsing. The system supports two types of metadata: Generation and Built-in. You can configure them separately as needed.
Note: Changes in Metadata generation settings only take effect for newly parsed documents later. They do not automatically update documents that have already completed parsing. To apply the new configuration to existing documents, parse the relevant documents again.
Generation is used to customize metadata fields that need to be generated from document content. Click Add to add a field and configure the following items as needed:
After saving, the system generates corresponding metadata from document content during subsequent document parsing based on these field definitions.
Built-in provides system-predefined metadata fields. You do not need to manually create or configure field rules. Use the switch on the right side of each corresponding field to choose whether to generate that metadata during document parsing. The following built-in fields are currently supported:
Select the fields to use, enable their switches, and then click Save to save the configuration. Generated metadata can be used in document management and retrieval filtering. To view or edit generated metadata, Metadata management.
When a dataset uses Table as the built-in parsing method, you can use Column mode to set the purpose of each table column during parsing and retrieval. A column can be included in chunk text for indexing, used only as metadata, or used for both.
When Auto is selected, all columns are included in chunk text and are also stored as metadata. This is RAGFlow's default setting. This mode is suitable when you do not need to distinguish the purpose of each column and want all table content to participate in retrieval and also be available as metadata.
When Manual is selected, RAGFlow identifies columns in the table and displays them one by one according to the original table column names. For example, CRIM, ZN, INDUS, CHAS, and similar names in the interface come from the column names in the current table. They are not predefined RAGFlow fields.
Users can assign one of the following roles to each column through the drop-down menu on the right side of each column name:
For example, if a table contains four columns, Title, Content, Category, and Year, you can set them according to actual use:
In this way, fields that do not need to participate in content retrieval can be prevented from entering chunk text, while these fields are still preserved as retrieval filter conditions.
Note: Column role configuration applies to the column structure of a table, not to individual cells. After configuration is modified, the new settings apply to subsequently parsed documents. For documents that have already completed parsing, the documents must be parsed again before the new column roles take effect.
The Data source area is used to associate the current dataset with data sources that have already been added.
Click Link data source and select the data source to associate from existing data sources. After association, the current dataset can use the data provided by that data source. This area is only used to establish and manage the association between the dataset and data sources. It does not provide data source creation or connection configuration.
Tip: To add a new data source, or configure data source connection information, synchronization methods, and other settings, see Data source. After adding the data source, return to the dataset configuration page to associate it.