createDocumentChunkerTool()

The createDocumentChunkerTool() function creates a tool for splitting documents into smaller chunks for efficient processing and retrieval. It supports different chunking strategies and configurable parameters.

Basic usage

import { createDocumentChunkerTool, MDocument } from '@mastra/rag'

const document = new MDocument({
  text: 'Your document content here...',
  metadata: { source: 'user-manual' },
})

const chunker = createDocumentChunkerTool({
  doc: document,
  params: {
    strategy: 'recursive',
    size: 512,
    overlap: 50,
    separator: '\n',
  },
})

const { chunks } = await chunker.execute()

Parameters

<PropertiesTable content={[ { name: 'doc', type: 'MDocument', description: 'The document to be chunked', isOptional: false, }, { name: 'params', type: 'ChunkParams', description: 'Configuration parameters for chunking', isOptional: true, defaultValue: 'Default chunking parameters', }, ]} />

ChunkParams

<PropertiesTable content={[ { name: 'strategy', type: "'recursive'", description: 'The chunking strategy to use', isOptional: true, defaultValue: "'recursive'", }, { name: 'size', type: 'number', description: 'Target size of each chunk in tokens/characters', isOptional: true, defaultValue: '512', }, { name: 'overlap', type: 'number', description: 'Number of overlapping tokens/characters between chunks', isOptional: true, defaultValue: '50', }, { name: 'separator', type: 'string', description: 'Character(s) to use as chunk separator', isOptional: true, defaultValue: "'\n'", }, ]} />

Returns

<PropertiesTable content={[ { name: 'chunks', type: 'DocumentChunk[]', description: 'Array of document chunks with their content and metadata', }, ]} />

Example with custom parameters

const technicalDoc = new MDocument({
  text: longDocumentContent,
  metadata: {
    type: 'technical',
    version: '1.0',
  },
})

const chunker = createDocumentChunkerTool({
  doc: technicalDoc,
  params: {
    strategy: 'recursive',
    size: 1024, // Larger chunks
    overlap: 100, // More overlap
    separator: '\n\n', // Split on double newlines
  },
})

const { chunks } = await chunker.execute()

// Process the chunks
chunks.forEach((chunk, index) => {
  console.log(`Chunk ${index + 1} length: ${chunk.content.length}`)
})

Tool details

The chunker is created as a Mastra tool with the following properties:

• Tool ID: Document Chunker {strategy} {size}
• Description: Chunks document using {strategy} strategy with size {size} and {overlap} overlap
• Input Schema: Empty object (no additional inputs required)
• Output Schema: Object containing the chunks array

Related

• MDocument
• createVectorQueryTool