ContextQMD
Back to Mongoose

Atlas Vector Search

docs/atlas-vector-search.md

9.9.412.7 KB
Original Source

Atlas Vector Search

Atlas Vector Search enables you to perform semantic searches on vector embeddings stored in MongoDB Atlas. Vector search allows you to find similar documents based on their meaning rather than exact keyword matches, which is essential for building modern AI applications like semantic search, recommendation systems, and RAG (Retrieval-Augmented Generation) applications.

Mongoose provides full support for creating vector search indexes and running vector search queries through the MongoDB aggregation pipeline.

The examples in this guide are based on MongoDB's mflix sample dataset.

Creating a Vector Search Index

To use vector search, you first need to create a vector search index on your collection. You can do this by calling schema.searchIndex() with type: 'vectorSearch' and defining the vector fields you want to index.

javascript
const movieSchema = new mongoose.Schema({
  title: String,
  plot: String,
  year: Number,
  genres: [String],
  // Field to store the embeddings for the plot. It can be whatever you choose.
  plot_embedding_voyage_3_large: {
    type: Buffer,  // embeddings are stored as binData (BSON binary vector, float32)
    validate: {
      // A 2048-dim float32 binary vector is 8194 bytes: a 2-byte vector header + 2048 x 4 bytes
      validator: v => v == null || v.length === 8194,
      message: 'plot_embedding must be nullish or a 2048-dimension float32 binary vector'
    }
  }
});

// Define a vector search index
movieSchema.searchIndex({
  name: 'vector_index',
  type: 'vectorSearch',
  definition: {
    fields: [{
      type: 'vector',
      path: 'plot_embedding_voyage_3_large',  // Where the embeddings are stored
      numDimensions: 2048,  // Must match your embedding model's dimensions
      similarity: 'cosine'  // Similarity function to use
    }]
  }
});

const Movie = mongoose.model('Movie', movieSchema);

await Movie.createSearchIndexes();  // Create the index

Using BinData vs Float Array

Embedding vectors are arrays of numbers representing a piece of text in a numeric form that can be compared and searched against other vectors. Vectors in MongoDB can be stored as a plain array of numbers (most embedding providers return this format), but MongoDB offers an optional, more compact and efficient storage format: Binary Vector.

This guide uses the Binary Vector format but you can also use a plain array of numbers instead if your application requires it.

javascript
const movieSchema = new mongoose.Schema({
  title: String,
  plot: String,
  year: Number,
  genres: [String],
  plot_embedding: {
    type: [Number], // Array instead of Buffer
    validate: {
      validator: v => v == null || v.length === 2048, // This would be the size when not a Buffer
      message: 'plot_embedding must be nullish or a 2048-length float32 array'
    }
  }
});

Index Definition Fields

The vector search index definition requires the following fields:

  • type: Must be 'vector' for vector fields
  • path: The field path containing the vector embeddings
  • numDimensions: Number of dimensions in your vector embeddings. This must match the output dimensions of your embedding model
  • similarity: The similarity function to use. Options are 'cosine', 'euclidean', or 'dotProduct'. See MongoDB's similarity metrics documentation for guidance on choosing the right metric.

Monitoring Index Status

After creating a vector search index, it takes time to build. Check the index status using Model.listSearchIndexes():

javascript
const indexes = await Movie.listSearchIndexes();
const vectorIndex = indexes.find(idx => idx.name === 'vector_index');

if (vectorIndex?.queryable) {
  console.log('Vector search index is ready!');
} else {
  console.log('Index is still building...');
}

Generating Embeddings

Vector embeddings are numerical representations of your data generated by embedding models. Mongoose doesn't generate embeddings for you - you need to generate them before saving documents or performing queries.

Using Atlas Automated Embeddings (Recommended)

MongoDB Atlas can automatically generate embeddings for your documents using Atlas Vector Search Automated Embeddings. Available in MongoDB Atlas, and on self-managed deployments running MongoDB 8.2+.

With Automated Embeddings, you define an autoEmbed index on a text field — no embedding field in your schema required. Atlas generates and manages the embeddings internally.

javascript
// Schema needs only the source text field — no embedding array needed
const movieSchema = new mongoose.Schema({
  title: String,
  plot: String
});

// Define an autoEmbed index — Atlas handles embedding generation via Voyage AI
movieSchema.searchIndex({
  name: 'auto_embed_index',
  type: 'vectorSearch',
  definition: {
    fields: [{
      type: 'autoEmbed',      // Atlas generates embeddings automatically
      modality: 'text',
      path: 'plot',           // Text field to embed
      model: 'voyage-4'       // Voyage AI model used at index- and query-time
    }]
  }
});

const Movie = mongoose.model('Movie', movieSchema);
await Movie.createSearchIndexes();

// Just save the document as usual — Atlas generates the embedding automatically
await Movie.create({ 
  title: 'Project Hail Mary', 
  plot: 'An amnesiac teacher and a musical, rock-spider-like alien team up on a spaceship to save their solar systems from a sun-eating organism.',
  year: 2026,
  genres: ['Adventure', 'Family', 'Sci-Fi']
});

// Query using plain text — Atlas generates the query embedding automatically
const results = await Movie.aggregate([
  {
    $vectorSearch: {
      index: 'auto_embed_index',
      path: 'plot',
      query: 'scientist and friend save dying sun',  // plain text, no embedding needed
      numCandidates: 100,
      limit: 10
    }
  },
  {
    // Rest of your aggregation pipeline...
  }
]);

Using Third-Party Embedding Models

If you need to generate embeddings in your application (for example, when using embedding models not supported by Atlas Automated Embeddings), you can use any embedding provider. Popular options include:

  • Voyage AI - High-quality embeddings optimized for retrieval, with models like voyage-4-large, voyage-4-little, and voyage-4-code
  • OpenAI Embeddings - Models like text-embedding-ada-002 and text-embedding-3-small
  • Anthropic Embeddings - Multiple models, including Voyage AI family.

Example workflow for application-generated embeddings:

javascript
const { Binary } = require('mongodb');  // Bundled with Mongoose

// In the sample_mflix dataset, there are two separate movies collections: embedded_movies
// has built-in embeddings that were generated with Voyage AI and movies does not.
const Movie = mongoose.model('Movie', movieSchema, 'embedded_movies');

// Use this function to generate embeddings with a third-party provider
async function generateEmbedding(text) {
  const voyageAPIKey = process.env.VOYAGE_API_KEY;
  const response = await fetch('https://ai.mongodb.com/v1/embeddings', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${voyageAPIKey}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      input: text,
      model: 'voyage-3-large',
      output_dimension: 2048
    })
  });

  if (!response.ok) {
    throw new Error(`Embeddings request failed: ${response.status} ${await response.text()}`);
  }

  const result = await response.json();
  return result.data[0].embedding;
}

// Generate embeddings before saving
async function saveMovieWithEmbedding(movieData) {
  const embeddings = await generateEmbedding(movieData.plot);

  // Convert the float array to a binData for efficient storage
  const embeddingVector = Binary.fromFloat32Array(new Float32Array(embeddings));

  const movie = new Movie({
    title: movieData.title,
    plot: movieData.plot,
    plot_embedding_voyage_3_large: embeddingVector
  });

  await movie.save();
  return movie;
}

Important considerations for application-generated embeddings:

  • The embedding dimensions must match your index configuration (numDimensions)
  • You must use the same embedding model for both indexing and querying
  • Embeddings must be normalized if using dotProduct similarity

Querying with $vectorSearch

Once your vector search index is created, and your embeddings are generated, you can perform vector similarity searches using the $vectorSearch aggregation stage.

javascript
// Your query embedding (if not using Automated Embeddings)
const queryEmbedding = await generateEmbedding('space adventure about friendship');

// Perform vector search
const results = await Movie.aggregate([
  {
    $vectorSearch: {
      index: 'vector_index',     // Name of your vector search index
      path: 'plot_embedding_voyage_3_large',     // Field containing the vectors
      queryVector: queryEmbedding, // Your query as an array of numbers (not binData when searching)
      numCandidates: 100,         // Number of candidates to consider (should be >= limit)
      limit: 10                   // Number of results to return
    }
  },
  {
    $project: {
      title: 1,
      plot: 1,
      score: { $meta: 'vectorSearchScore' }  // Include similarity score
    }
  }
]);

$vectorSearch Options

The $vectorSearch stage accepts the following options:

  • index (required): Name of the vector search index to use
  • path (required): Field path containing the vector embeddings
  • queryVector (required): Array of numbers representing your query embedding
  • limit (required): Maximum number of documents to return
  • numCandidates (optional): Number of candidate documents to examine. Must be >= limit. Recommended: set to at least 10-20x your limit for better accuracy
  • filter (optional): MQL filter expression to pre-filter documents before vector search
  • exact (optional): Boolean. If true, performs exact nearest neighbor search. If false or omitted, performs approximate search

For detailed information on these parameters and performance tuning, see the MongoDB Vector Search documentation.

Using Pre-filters

You can use the filter option to restrict your search to a subset of documents. Any field you filter on must be declared as a filter field in your index definition:

javascript
movieSchema.searchIndex({
  name: 'vector_index',
  type: 'vectorSearch',
  definition: {
    fields: [
      {
        type: 'vector',
        path: 'plot_embedding_voyage_3_large',
        numDimensions: 2048,
        similarity: 'cosine'
      },
      { type: 'filter', path: 'year' },    // Enable filtering on year
      { type: 'filter', path: 'genres' }   // Enable filtering on genres
    ]
  }
});

Once the filter fields are indexed, you can pre-filter your search:

javascript
const results = await Movie.aggregate([
  {
    $vectorSearch: {
      index: 'vector_index',
      path: 'plot_embedding_voyage_3_large',
      queryVector: queryEmbedding,
      numCandidates: 100,
      limit: 10,
      filter: {
        year: { $gte: 2010 },     // Only movies from 2010 or later
        genres: 'Family'          // Only family movies
      }
    }
  }
]);

The filter field accepts standard MongoDB query operators. For best performance, ensure the fields used in your filter are indexed. See MongoDB's pre-filtering documentation for more details.

See Also