ContextQMD
Back to Searchkick

Searchkick

README.md

6.1.052.7 KB
Original Source

Searchkick

:rocket: Intelligent search made easy

Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.

Searchkick handles:

  • stemming - tomatoes matches tomato
  • special characters - jalapeno matches jalapeño
  • extra whitespace - dishwasher matches dish washer
  • misspellings - zuchini matches zucchini
  • custom synonyms - pop matches soda

Plus:

  • query like SQL - no need to learn a new query language
  • reindex without downtime
  • easily personalize results for each user
  • autocomplete
  • “Did you mean” suggestions
  • supports many languages
  • works with Active Record and Mongoid

Check out Searchjoy for analytics and Autosuggest for query suggestions

:tangerine: Battle-tested at Instacart

Contents

Searchkick 6.0 was recently released! See how to upgrade

Getting Started

Install Elasticsearch or OpenSearch. For Homebrew, use:

sh
brew install opensearch
brew services start opensearch

Add these lines to your application’s Gemfile:

ruby
gem "searchkick"

gem "elasticsearch"   # select one
gem "opensearch-ruby" # select one

The latest version works with Elasticsearch 8 and 9 and OpenSearch 2 and 3. For Elasticsearch 7 and OpenSearch 1, use version 5.5.2 and this readme.

Add searchkick to models you want to search.

ruby
class Product < ApplicationRecord
  searchkick
end

Add data to the search index.

ruby
Product.reindex

And to query, use:

ruby
products = Product.search("apples")
products.each do |product|
  puts product.name
end

Searchkick supports the complete Elasticsearch Search API and OpenSearch Search API. As your search becomes more advanced, we recommend you use the search server DSL for maximum flexibility.

Querying

Query like SQL

ruby
Product.search("apples").where(in_stock: true).limit(10).offset(50)

Search specific fields

ruby
fields(:name, :brand)

Where

ruby
where(store_id: 1, expires_at: Time.now..)

These types of filters are supported

Order

ruby
order(_score: :desc) # most relevant first - default

All of these sort options are supported

Limit / offset

ruby
limit(20).offset(40)

Select

ruby
select(:name)

These source filtering options are supported

Results

Searches return a Searchkick::Relation object. This responds like an array to most methods.

ruby
results = Product.search("milk")
results.size
results.any?
results.each { |result| ... }

By default, ids are fetched from the search server and records are fetched from your database. To fetch everything from the search server, use:

ruby
Product.search("apples").load(false)

Get total results

ruby
results.total_count

Get the time the search took (in milliseconds)

ruby
results.took

Get the full response from the search server

ruby
results.response

Note: By default, Elasticsearch and OpenSearch limit paging to the first 10,000 results for performance. This applies to the total count as well.

Filtering

Equal

ruby
where(store_id: 1)

Not equal

ruby
where.not(store_id: 2)

Greater than (gt), less than (lt), greater than or equal (gte), less than or equal (lte)

ruby
where(expires_at: {gt: Time.now})

Range

ruby
where(orders_count: 1..10)

In

ruby
where(aisle_id: [25, 30])

Not in

ruby
where.not(aisle_id: [25, 30])

Contains all

ruby
where(user_ids: {all: [1, 3]})

Like

ruby
where(category: {like: "%frozen%"})

Case-insensitive like

ruby
where(category: {ilike: "%frozen%"})

Regular expression

ruby
where(category: /frozen .+/)

Prefix

ruby
where(category: {prefix: "frozen"})

Exists

ruby
where(store_id: {exists: true})

Combine filters with OR

ruby
where(_or: [{in_stock: true}, {backordered: true}])

Boosting

Boost important fields

ruby
fields("title^10", "description")

Boost by the value of a field (field must be numeric)

ruby
boost_by(:orders_count) # give popular documents a little boost
boost_by(orders_count: {factor: 10}) # default factor is 1

Boost matching documents

ruby
boost_where(user_id: 1)
boost_where(user_id: {value: 1, factor: 100}) # default factor is 1000
boost_where(user_id: [{value: 1, factor: 100}, {value: 2, factor: 200}])

Boost by recency

ruby
boost_by_recency(created_at: {scale: "7d", decay: 0.5})

You can also boost by:

Get Everything

Use a * for the query.

ruby
Product.search("*")

Pagination

Plays nicely with kaminari and will_paginate.

ruby
# controller
@products = Product.search("milk").page(params[:page]).per_page(20)

View with kaminari

erb
<%= paginate @products %>

View with will_paginate

erb
<%= will_paginate @products %>

Partial Matches

By default, results must match all words in the query.

ruby
Product.search("fresh honey") # fresh AND honey

To change this, use:

ruby
Product.search("fresh honey").operator("or") # fresh OR honey

By default, results must match the entire word - back will not match backpack. You can change this behavior with:

ruby
class Product < ApplicationRecord
  searchkick word_start: [:name]
end

And to search (after you reindex):

ruby
Product.search("back").fields(:name).match(:word_start)

Available options are:

OptionMatchesExample
:wordentire wordapple matches apple
:word_startstart of wordapp matches apple
:word_middleany part of wordppl matches apple
:word_endend of wordple matches apple
:text_startstart of textgre matches green apple, app does not match
:text_middleany part of texteen app matches green apple
:text_endend of textple matches green apple, een does not match

The default is :word. The most matches will happen with :word_middle.

To specify different matching for different fields, use:

ruby
Product.search(query).fields({name: :word_start}, {brand: :word_middle})

Exact Matches

To match a field exactly (case-sensitive), use:

ruby
Product.search(query).fields({name: :exact})

Phrase Matches

To only match the exact order, use:

ruby
Product.search("fresh honey").match(:phrase)

Stemming and Language

Searchkick stems words by default for better matching. apple and apples both stem to appl, so searches for either term will have the same matches.

Searchkick defaults to English for stemming. To change this, use:

ruby
class Product < ApplicationRecord
  searchkick language: "german"
end

See the list of languages. A few languages require plugins:

You can also use a Hunspell dictionary for stemming.

ruby
class Product < ApplicationRecord
  searchkick stemmer: {type: "hunspell", locale: "en_US"}
end

Disable stemming with:

ruby
class Image < ApplicationRecord
  searchkick stem: false
end

Exclude certain words from stemming with:

ruby
class Image < ApplicationRecord
  searchkick stem_exclusion: ["apples"]
end

Or change how words are stemmed:

ruby
class Image < ApplicationRecord
  searchkick stemmer_override: ["apples => other"]
end

Synonyms

ruby
class Product < ApplicationRecord
  searchkick search_synonyms: [["pop", "soda"], ["burger", "hamburger"]]
end

Call Product.reindex after changing synonyms. Synonyms are applied at search time before stemming, and can be a single word or multiple words.

For directional synonyms, use:

ruby
search_synonyms: ["lightbulb => halogenlamp"]

Dynamic Synonyms

The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex. We recommend placing synonyms in a file on the search server (in the config directory). This allows you to reload synonyms without reindexing.

txt
pop, soda
burger, hamburger

Then use:

ruby
class Product < ApplicationRecord
  searchkick search_synonyms: "synonyms.txt"
end

And reload with:

ruby
Product.search_index.reload_synonyms

Misspellings

By default, Searchkick handles misspelled queries by returning results with an edit distance of one.

You can change this with:

ruby
Product.search("zucini").misspellings(edit_distance: 2) # zucchini

To prevent poor precision and improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.

ruby
Product.search("zuchini").misspellings(below: 5)

If there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.

Turn off misspellings with:

ruby
Product.search("zuchini").misspellings(false) # no zucchini

Specify which fields can include misspellings with:

ruby
Product.search("zucini").fields(:name, :color).misspellings(fields: [:name])

When doing this, you must also specify fields to search

Bad Matches

If a user searches butter, they may also get results for peanut butter. To prevent this, use:

ruby
Product.search("butter").exclude("peanut butter")

You can map queries and terms to exclude with:

ruby
exclude_queries = {
  "butter" => ["peanut butter"],
  "cream" => ["ice cream", "whipped cream"]
}

Product.search(query).exclude(exclude_queries[query])

You can demote results by boosting by a factor less than one:

ruby
Product.search("butter").boost_where(category: {value: "pantry", factor: 0.5})

Emoji

Search :ice_cream::cake: and get ice cream cake!

Add this line to your application’s Gemfile:

ruby
gem "gemoji-parser"

And use:

ruby
Product.search("🍨🍰").emoji

Indexing

Control what data is indexed with the search_data method. Call Product.reindex after changing this method.

ruby
class Product < ApplicationRecord
  belongs_to :department

  def search_data
    {
      name: name,
      department_name: department.name,
      on_sale: sale_price.present?
    }
  end
end

Searchkick uses find_in_batches to import documents. To eager load associations, use the search_import scope.

ruby
class Product < ApplicationRecord
  scope :search_import, -> { includes(:department) }
end

By default, all records are indexed. To control which records are indexed, use the should_index? method.

ruby
class Product < ApplicationRecord
  def should_index?
    active # only index active records
  end
end

If a reindex is interrupted, you can resume it with:

ruby
Product.reindex(resume: true)

For large data sets, try parallel reindexing.

To Reindex, or Not to Reindex

Reindex

  • when you install or upgrade searchkick
  • change the search_data method
  • change the searchkick method

No need to reindex

  • app starts

Strategies

There are four strategies for keeping the index synced with your database.

  1. Inline (default)

Anytime a record is inserted, updated, or deleted

  1. Asynchronous

Use background jobs for better performance

ruby
class Product < ApplicationRecord
  searchkick callbacks: :async
end

Jobs are added to a queue named searchkick.

  1. Queuing

Push ids of records that need updated to a queue and reindex in the background in batches. This is more performant than the asynchronous method, which updates records individually. See how to set up.

  1. Manual

Turn off automatic syncing

ruby
class Product < ApplicationRecord
  searchkick callbacks: false
end

And reindex a record or relation manually.

ruby
product.reindex
# or
store.products.reindex(mode: :async)

You can also do bulk updates.

ruby
Searchkick.callbacks(:bulk) do
  Product.find_each(&:update_fields)
end

Or temporarily skip updates.

ruby
Searchkick.callbacks(false) do
  Product.find_each(&:update_fields)
end

Or override the model’s strategy.

ruby
product.reindex(mode: :async) # :inline or :queue

Associations

Data is not automatically synced when an association is updated. If this is desired, add a callback to reindex:

ruby
class Image < ApplicationRecord
  belongs_to :product

  after_commit :reindex_product

  def reindex_product
    product.reindex
  end
end

Default Scopes

If you have a default scope that filters records, use the should_index? method to exclude them from indexing:

ruby
class Product < ApplicationRecord
  default_scope { where(deleted_at: nil) }

  def should_index?
    deleted_at.nil?
  end
end

If you want to index and search filtered records, set:

ruby
class Product < ApplicationRecord
  searchkick unscope: true
end

The best starting point to improve your search by far is to track searches and conversions. Searchjoy makes it easy.

ruby
Product.search("apple").track(user_id: current_user.id)

See the docs for how to install and use. Focus on top searches with a low conversion rate.

Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches. This can make a huge difference on the quality of your search.

Add conversion data with:

ruby
class Product < ApplicationRecord
  has_many :conversions, class_name: "Searchjoy::Conversion", as: :convertable
  has_many :searches, class_name: "Searchjoy::Search", through: :conversions

  searchkick conversions_v2: [:conversions] # name of field

  def search_data
    {
      name: name,
      conversions: searches.group(:query).distinct.count(:user_id)
      # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
    }
  end
end

Reindex and set up a cron job to add new conversions daily. For zero downtime deployment, temporarily set conversions_v2(false) in your search calls until the data is reindexed.

Performant Conversions

A performant way to do conversions is to cache them to prevent N+1 queries. For Postgres, create a migration with:

ruby
add_column :products, :search_conversions, :jsonb

For MySQL, use :json, and for others, use :text with a JSON serializer.

Next, update your model. Create a separate method for conversion data so you can use partial reindexing.

ruby
class Product < ApplicationRecord
  searchkick conversions_v2: [:conversions]

  def search_data
    {
      name: name,
      category: category
    }.merge(conversions_data)
  end

  def conversions_data
    {
      conversions: search_conversions || {}
    }
  end
end

Deploy and reindex your data. For zero downtime deployment, temporarily set conversions_v2(false) in your search calls until the data is reindexed.

ruby
Product.reindex

Then, create a job to update the conversions column and reindex records with new conversions. Here’s one you can use for Searchjoy:

ruby
class UpdateConversionsJob < ApplicationJob
  def perform(class_name, since: nil, update: true, reindex: true)
    model = Searchkick.load_model(class_name)

    # get records that have a recent conversion
    recently_converted_ids =
      Searchjoy::Conversion.where(convertable_type: class_name, created_at: since..)
        .order(:convertable_id).distinct.pluck(:convertable_id)

    # split into batches
    recently_converted_ids.in_groups_of(1000, false) do |ids|
      if update
        # fetch conversions
        conversions =
          Searchjoy::Conversion.where(convertable_id: ids, convertable_type: class_name)
            .joins(:search).where.not(searchjoy_searches: {user_id: nil})
            .group(:convertable_id, :query).distinct.count(:user_id)

        # group by record
        conversions_by_record = {}
        conversions.each do |(id, query), count|
          (conversions_by_record[id] ||= {})[query] = count
        end

        # update conversions column
        model.transaction do
          conversions_by_record.each do |id, conversions|
            model.where(id: id).update_all(search_conversions: conversions)
          end
        end
      end

      if reindex
        # reindex conversions data
        model.where(id: ids).reindex(:conversions_data, ignore_missing: true)
      end
    end
  end
end

Run the job:

ruby
UpdateConversionsJob.perform_now("Product")

And set it up to run daily.

ruby
UpdateConversionsJob.perform_later("Product", since: 1.day.ago)

Personalized Results

Order results differently for each user. For example, show a user’s previously purchased products before other results.

ruby
class Product < ApplicationRecord
  def search_data
    {
      name: name,
      orderer_ids: orders.pluck(:user_id) # boost this product for these users
    }
  end
end

Reindex and search with:

ruby
Product.search("milk").boost_where(orderer_ids: current_user.id)

Instant Search / Autocomplete

Autocomplete predicts what a user will type, making the search experience faster and easier.

Note: To autocomplete on search terms rather than results, check out Autosuggest.

Note 2: If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s much faster to load all records into JavaScript and autocomplete there (eliminates network requests).

First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing fast queries.

ruby
class Movie < ApplicationRecord
  searchkick word_start: [:title, :director]
end

Reindex and search with:

ruby
Movie.search("jurassic pa").fields(:title).match(:word_start)

Use a front-end library like typeahead.js to show the results.

Here’s how to make it work with Rails

First, add a route and controller action.

ruby
class MoviesController < ApplicationController
  def autocomplete
    render json: Movie.search(params[:query]).fields("title^5", "director")
      .match(:word_start).limit(10).load(false).misspellings(below: 5).map(&:title)
  end
end

Note: Use load(false) and misspellings(below: n) (or misspellings(false)) for best performance.

Then add the search box and JavaScript code to a view.

html
<input type="text" id="query" name="query" />

<script src="jquery.js"></script>
<script src="typeahead.bundle.js"></script>
<script>
  var movies = new Bloodhound({
    datumTokenizer: Bloodhound.tokenizers.whitespace,
    queryTokenizer: Bloodhound.tokenizers.whitespace,
    remote: {
      url: '/movies/autocomplete?query=%QUERY',
      wildcard: '%QUERY'
    }
  });
  $('#query').typeahead(null, {
    source: movies
  });
</script>

Suggestions

ruby
class Product < ApplicationRecord
  searchkick suggest: [:name] # fields to generate suggestions
end

Reindex and search with:

ruby
products = Product.search("peantu butta").suggest
products.suggestions # ["peanut butter"]

Aggregations

Aggregations provide aggregated search data.

ruby
products = Product.search("chuck taylor").aggs(:product_type, :gender, :brand)
products.aggs

By default, where conditions apply to aggregations.

ruby
Product.search("wingtips").where(color: "brandy").aggs(:size)
# aggregations for brandy wingtips are returned

Change this with:

ruby
Product.search("wingtips").where(color: "brandy").aggs(:size).smart_aggs(false)
# aggregations for all wingtips are returned

Set where conditions for each aggregation separately with:

ruby
Product.search("wingtips").aggs(size: {where: {color: "brandy"}})

Limit

ruby
Product.search("apples").aggs(store_id: {limit: 10})

Order

ruby
Product.search("wingtips").aggs(color: {order: {"_key" => "asc"}}) # alphabetically

All of these options are supported

Ranges

ruby
price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]
Product.search("*").aggs(price: {ranges: price_ranges})

Minimum document count

ruby
Product.search("apples").aggs(store_id: {min_doc_count: 2})

Script support

ruby
Product.search("*").aggs(color: {script: {source: "'Color: ' + _value"}})

Date histogram

ruby
Product.search("pear").aggs(products_per_year: {date_histogram: {field: :created_at, interval: :year}})

For other aggregation types, including sub-aggregations, use body_options:

ruby
Product.search("orange").body_options(aggs: {price: {histogram: {field: :price, interval: 10}}})

Highlight

Specify which fields to index with highlighting.

ruby
class Band < ApplicationRecord
  searchkick highlight: [:name]
end

Highlight the search query in the results.

ruby
bands = Band.search("cinema").highlight

View the highlighted fields with:

ruby
bands.with_highlights.each do |band, highlights|
  highlights[:name] # "Two Door <em>Cinema</em> Club"
end

To change the tag, use:

ruby
Band.search("cinema").highlight(tag: "<strong>")

To highlight and search different fields, use:

ruby
Band.search("cinema").fields(:name).highlight(fields: [:description])

By default, the entire field is highlighted. To get small snippets instead, use:

ruby
bands = Band.search("cinema").highlight(fragment_size: 20)
bands.with_highlights(multiple: true).each do |band, highlights|
  highlights[:name].join(" and ")
end

Additional options can be specified for each field:

ruby
Band.search("cinema").fields(:name).highlight(fields: {name: {fragment_size: 200}})

You can find available highlight options in the Elasticsearch or OpenSearch reference.

Similar Items

Find similar items

ruby
product = Product.first
product.similar.fields(:name).where(size: "12 oz")

Geospatial Searches

ruby
class Restaurant < ApplicationRecord
  searchkick locations: [:location]

  def search_data
    attributes.merge(location: {lat: latitude, lon: longitude})
  end
end

Reindex and search with:

ruby
Restaurant.search("pizza").where(location: {near: {lat: 37, lon: -114}, within: "100mi"}) # or 160km

Bounded by a box

ruby
Restaurant.search("sushi").where(location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}})

Note: top_right and bottom_left also work

Bounded by a polygon

ruby
Restaurant.search("dessert").where(location: {geo_polygon: {points: [{lat: 38, lon: -123}, {lat: 39, lon: -123}, {lat: 37, lon: 122}]}})

Boost By Distance

Boost results by distance - closer results are boosted more

ruby
Restaurant.search("noodles").boost_by_distance(location: {origin: {lat: 37, lon: -122}})

Also supports additional options

ruby
Restaurant.search("wings").boost_by_distance(location: {origin: {lat: 37, lon: -122}, function: "linear", scale: "30mi", decay: 0.5})

Geo Shapes

You can also index and search geo shapes.

ruby
class Restaurant < ApplicationRecord
  searchkick geo_shape: [:bounds]

  def search_data
    attributes.merge(
      bounds: {
        type: "envelope",
        coordinates: [{lat: 4, lon: 1}, {lat: 2, lon: 3}]
      }
    )
  end
end

See the Elasticsearch documentation for details.

Find shapes intersecting with the query shape

ruby
Restaurant.search("soup").where(bounds: {geo_shape: {type: "polygon", coordinates: [[{lat: 38, lon: -123}, ...]]}})

Falling entirely within the query shape

ruby
Restaurant.search("salad").where(bounds: {geo_shape: {type: "circle", relation: "within", coordinates: {lat: 38, lon: -123}, radius: "1km"}})

Not touching the query shape

ruby
Restaurant.search("burger").where(bounds: {geo_shape: {type: "envelope", relation: "disjoint", coordinates: [{lat: 38, lon: -123}, {lat: 37, lon: -122}]}})

Inheritance

Searchkick supports single table inheritance.

ruby
class Dog < Animal
end

In your parent model, set:

ruby
class Animal < ApplicationRecord
  searchkick inheritance: true
end

The parent and child model can both reindex.

ruby
Animal.reindex
Dog.reindex # equivalent, all animals reindexed

And to search, use:

ruby
Animal.search("*")                # all animals
Dog.search("*")                   # just dogs
Animal.search("*").type(Cat, Dog) # just cats and dogs

Notes:

  1. The suggest option retrieves suggestions from the parent at the moment.

    ruby
    Dog.search("airbudd").suggest # suggestions for all animals

  2. This relies on a type field that is automatically added to the indexed document. Be wary of defining your own type field in search_data, as it will take precedence.

Debugging Queries

To help with debugging queries, you can use:

ruby
Product.search("soap").debug

This prints useful info to stdout.

See how the search server scores your queries with:

ruby
Product.search("soap").explain.response

See how the search server tokenizes your queries with:

ruby
Product.search_index.tokens("Dish Washer Soap", analyzer: "searchkick_index")
# ["dish", "dishwash", "washer", "washersoap", "soap"]

Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search")
# ["dishwashersoap"] - no match

Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search2")
# ["dishwash", "soap"] - match!!

Partial matches

ruby
Product.search_index.tokens("San Diego", analyzer: "searchkick_word_start_index")
# ["s", "sa", "san", "d", "di", "die", "dieg", "diego"]

Product.search_index.tokens("dieg", analyzer: "searchkick_word_search")
# ["dieg"] - match!!

See the complete list of analyzers.

Testing

As you iterate on your search, it’s a good idea to add tests.

For performance, only enable Searchkick callbacks for the tests that need it.

Rails

Add to your test/test_helper.rb:

ruby
module ActiveSupport
  class TestCase
    parallelize_setup do |worker|
      Searchkick.index_suffix = worker

      # reindex models for parallel tests
      Product.reindex
    end
  end
end

# reindex models for non-parallel tests
Product.reindex

# and disable callbacks
Searchkick.disable_callbacks

And use:

ruby
class ProductTest < ActiveSupport::TestCase
  setup do
    Searchkick.enable_callbacks
  end

  teardown do
    Searchkick.disable_callbacks
  end

  test "search" do
    Product.create!(name: "Apple")
    Product.search_index.refresh
    assert_equal ["Apple"], Product.search("apple").map(&:name)
  end
end

Minitest

Add to your test/test_helper.rb:

ruby
# reindex models
Product.reindex

# and disable callbacks
Searchkick.disable_callbacks

And use:

ruby
class ProductTest < Minitest::Test
  def setup
    Searchkick.enable_callbacks
  end

  def teardown
    Searchkick.disable_callbacks
  end

  def test_search
    Product.create!(name: "Apple")
    Product.search_index.refresh
    assert_equal ["Apple"], Product.search("apple").map(&:name)
  end
end

RSpec

Add to your spec/spec_helper.rb:

ruby
RSpec.configure do |config|
  config.before(:suite) do
    # reindex models
    Product.reindex

    # and disable callbacks
    Searchkick.disable_callbacks
  end

  config.around(:each, search: true) do |example|
    Searchkick.callbacks(nil) do
      example.run
    end
  end
end

And use:

ruby
describe Product, search: true do
  it "searches" do
    Product.create!(name: "Apple")
    Product.search_index.refresh
    assert_equal ["Apple"], Product.search("apple").map(&:name)
  end
end

Factory Bot

Define a trait for each model:

ruby
FactoryBot.define do
  factory :product do
    trait :reindex do
      after(:create) do |product, _|
        product.reindex(refresh: true)
      end
    end
  end
end

And use:

ruby
FactoryBot.create(:product, :reindex)

GitHub Actions

Check out setup-elasticsearch for an easy way to install Elasticsearch:

yml
    - uses: ankane/setup-elasticsearch@v1

And setup-opensearch for an easy way to install OpenSearch:

yml
    - uses: ankane/setup-opensearch@v1

Deployment

For the search server, Searchkick uses ENV["ELASTICSEARCH_URL"] for Elasticsearch and ENV["OPENSEARCH_URL"] for OpenSearch. This defaults to http://localhost:9200.

Elastic Cloud

Create an initializer config/initializers/elasticsearch.rb with:

ruby
ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"

Then deploy and reindex:

sh
rake searchkick:reindex:all

Amazon OpenSearch Service

Create an initializer config/initializers/opensearch.rb with:

ruby
ENV["OPENSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"

To use signed requests, include in your Gemfile:

ruby
gem "faraday_middleware-aws-sigv4"

and add to your initializer:

ruby
Searchkick.aws_credentials = {
  access_key_id: ENV["AWS_ACCESS_KEY_ID"],
  secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
  region: "us-east-1"
}

Then deploy and reindex:

sh
rake searchkick:reindex:all

Heroku

Choose an add-on: Bonsai, SearchBox, or Elastic Cloud.

For Elasticsearch on Bonsai:

sh
heroku addons:create bonsai
heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`

For OpenSearch on Bonsai:

sh
heroku addons:create bonsai --engine=opensearch
heroku config:set OPENSEARCH_URL=`heroku config:get BONSAI_URL`

For SearchBox:

sh
heroku addons:create searchbox:starter
heroku config:set ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`

For Elastic Cloud (previously Found):

sh
heroku addons:create foundelasticsearch
heroku addons:open foundelasticsearch

Visit the Shield page and reset your password. You’ll need to add the username and password to your url. Get the existing url with:

sh
heroku config:get FOUNDELASTICSEARCH_URL

And add elastic:password@ right after https:// and add port 9243 at the end:

sh
heroku config:set ELASTICSEARCH_URL=https://elastic:[email protected]:9243

Then deploy and reindex:

sh
heroku run rake searchkick:reindex:all

Self-Hosted and Other

Create an initializer with:

ruby
ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
# or
ENV["OPENSEARCH_URL"] = "https://user:password@host:port"

Then deploy and reindex:

sh
rake searchkick:reindex:all

Data Protection

We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send personal data of your users to the search server.

Bonsai, Elastic Cloud, and Amazon OpenSearch Service all support encryption at rest and HTTPS.

Automatic Failover

Create an initializer with multiple hosts:

ruby
ENV["ELASTICSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"
# or
ENV["OPENSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"

Client Options

Create an initializer with:

ruby
Searchkick.client_options[:reload_connections] = true

See the docs for Elasticsearch or Opensearch for a complete list of options.

Lograge

Add the following to config/environments/production.rb:

ruby
config.lograge.custom_options = lambda do |event|
  options = {}
  options[:search] = event.payload[:searchkick_runtime] if event.payload[:searchkick_runtime].to_f > 0
  options
end

See Production Rails for other good practices.

Performance

Persistent HTTP Connections

Significantly increase performance with persistent HTTP connections. Add Typhoeus to your Gemfile and it’ll automatically be used.

ruby
gem "typhoeus"

To reduce log noise, create an initializer with:

ruby
Ethon.logger = Logger.new(nil)

Searchable Fields

By default, all string fields are searchable (can be used in fields option). Speed up indexing and reduce index size by only making some fields searchable.

ruby
class Product < ApplicationRecord
  searchkick searchable: [:name]
end

Filterable Fields

By default, all string fields are filterable (can be used in where option). Speed up indexing and reduce index size by only making some fields filterable.

ruby
class Product < ApplicationRecord
  searchkick filterable: [:brand]
end

Note: Non-string fields are always filterable and should not be passed to this option.

Parallel Reindexing

For large data sets, you can use background jobs to parallelize reindexing.

ruby
Product.reindex(mode: :async)
# {index_name: "products_production_20250111210018065"}

Once the jobs complete, promote the new index with:

ruby
Product.search_index.promote(index_name)

You can optionally track the status with Redis:

ruby
Searchkick.redis = Redis.new

And use:

ruby
Searchkick.reindex_status(index_name)

You can also have Searchkick wait for reindexing to complete

ruby
Product.reindex(mode: :async, wait: true)

You can use your background job framework to control concurrency. For Solid Queue, create an initializer with:

ruby
module SearchkickBulkReindexConcurrency
  extend ActiveSupport::Concern

  included do
    limits_concurrency to: 3, key: ""
  end
end

Rails.application.config.after_initialize do
  Searchkick::BulkReindexJob.include(SearchkickBulkReindexConcurrency)
end

This will allow only 3 jobs to run at once.

Refresh Interval

You can specify a longer refresh interval while reindexing to increase performance.

ruby
Product.reindex(mode: :async, refresh_interval: "30s")

Note: This only makes a noticeable difference with parallel reindexing.

When promoting, have it restored to the value in your mapping (defaults to 1s).

ruby
Product.search_index.promote(index_name, update_refresh_interval: true)

Queuing

Push ids of records needing reindexing to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using connection_pool.

ruby
Searchkick.redis = ConnectionPool.new { Redis.new }

And ask your models to queue updates.

ruby
class Product < ApplicationRecord
  searchkick callbacks: :queue
end

Then, set up a background job to run.

ruby
Searchkick::ProcessQueueJob.perform_later(class_name: "Product")

You can check the queue length with:

ruby
Product.search_index.reindex_queue.length

For more tips, check out Keeping Elasticsearch in Sync.

Routing

Searchkick supports routing, which can significantly speed up searches.

ruby
class Business < ApplicationRecord
  searchkick routing: true

  def search_routing
    city_id
  end
end

Reindex and search with:

ruby
Business.search("ice cream").routing(params[:city_id])

Partial Reindexing

Reindex a subset of attributes to reduce time spent generating search data and cut down on network traffic.

ruby
class Product < ApplicationRecord
  def search_data
    {
      name: name,
      category: category
    }.merge(prices_data)
  end

  def prices_data
    {
      price: price,
      sale_price: sale_price
    }
  end
end

And use:

ruby
Product.reindex(:prices_data)

Ignore errors for missing documents with:

ruby
Product.reindex(:prices_data, ignore_missing: true)

Advanced

Searchkick makes it easy to use the Elasticsearch or OpenSearch DSL on its own.

Advanced Mapping

Create a custom mapping:

ruby
class Product < ApplicationRecord
  searchkick mappings: {
    properties: {
      name: {type: "keyword"}
    }
  }
end

Note: If you use a custom mapping, you'll need to use custom searching as well.

To keep the mappings and settings generated by Searchkick, use:

ruby
class Product < ApplicationRecord
  searchkick merge_mappings: true, mappings: {...}
end

And use the body option to search:

ruby
products = Product.search.body(query: {match: {name: "milk"}})

View the response with:

ruby
products.response

To modify the query generated by Searchkick, use:

ruby
products = Product.search("milk").body_options(min_score: 1)

or

ruby
products =
  Product.search("apples") do |body|
    body[:min_score] = 1
  end

Client

To access the Elasticsearch::Client or OpenSearch::Client directly, use:

ruby
Searchkick.client

To batch search requests for performance, use:

ruby
products = Product.search("snacks")
coupons = Coupon.search("snacks")
Searchkick.multi_search([products, coupons])

Then use products and coupons as typical results.

Note: Errors are not raised as with single requests. Use the error method on each query to check for errors.

Multiple Models

Search across multiple models with:

ruby
Searchkick.search("milk").models(Product, Category)

Boost specific models with:

ruby
indices_boost(Category => 2, Product => 1)

Multi-Tenancy

Check out this great post on the Apartment gem. Follow a similar pattern if you use another gem.

Scroll API

Searchkick also supports the scroll API. Scrolling is not intended for real time user requests, but rather for processing large amounts of data.

ruby
Product.search("*").scroll("1m") do |batch|
  # process batch ...
end

You can also scroll batches manually.

ruby
products = Product.search("*").scroll("1m")
while products.any?
  # process batch ...

  products = products.scroll
end

products.clear_scroll

Deep Paging

By default, Elasticsearch and OpenSearch limit paging to the first 10,000 results. Here’s why. We don’t recommend changing this, but if you really need all results, you can use:

ruby
class Product < ApplicationRecord
  searchkick deep_paging: true
end

If you just need an accurate total count, you can instead use:

ruby
Product.search("pears").body_options(track_total_hits: true)

Nested Data

To query nested data, use dot notation.

ruby
Product.search("san").fields("store.city").where("store.zip_code" => 12345)

Available for Elasticsearch 8.6+ and OpenSearch 2.4+

ruby
class Product < ApplicationRecord
  searchkick knn: {embedding: {dimensions: 3, distance: "cosine"}}
end

Also supports euclidean and inner_product

Reindex and search with:

ruby
Product.search.knn(field: :embedding, vector: [1, 2, 3]).limit(10)

HNSW Options

Nearest neighbor search uses HNSW for indexing.

Specify m and ef_construction

ruby
class Product < ApplicationRecord
  searchkick knn: {embedding: {dimensions: 3, distance: "cosine", m: 16, ef_construction: 100}}
end

Specify ef_search

ruby
Product.search.knn(field: :embedding, vector: [1, 2, 3], ef_search: 40).limit(10)

First, add nearest neighbor search to your model

ruby
class Product < ApplicationRecord
  searchkick knn: {embedding: {dimensions: 768, distance: "cosine"}}
end

Generate an embedding for each record (you can use an external service or a library like Informers)

ruby
embed = Informers.pipeline("embedding", "Snowflake/snowflake-arctic-embed-m-v1.5")
embed_options = {model_output: "sentence_embedding", pooling: "none"} # specific to embedding model

Product.find_each do |product|
  embedding = embed.(product.name, **embed_options)
  product.update!(embedding: embedding)
end

For search, generate an embedding for the query (the query prefix is specific to the embedding model)

ruby
query_prefix = "Represent this sentence for searching relevant passages: "
query_embedding = embed.(query_prefix + query, **embed_options)

And perform nearest neighbor search

ruby
Product.search.knn(field: :embedding, vector: query_embedding).limit(20)

See a full example

Perform keyword search and semantic search in parallel

ruby
keyword_search = Product.search(query).limit(20)
semantic_search = Product.search.knn(field: :embedding, vector: query_embedding).limit(20)
Searchkick.multi_search([keyword_search, semantic_search])

To combine the results, use Reciprocal Rank Fusion (RRF)

ruby
Searchkick::Reranking.rrf(keyword_search, semantic_search).first(5)

Or a reranking model

ruby
rerank = Informers.pipeline("reranking", "mixedbread-ai/mxbai-rerank-xsmall-v1")
results = (keyword_search.to_a + semantic_search.to_a).uniq
rerank.(query, results.map(&:name)).first(5).map { |v| results[v[:doc_id]] }

See a full example

Reference

Reindex one record

ruby
product = Product.find(1)
product.reindex

Reindex multiple records

ruby
Product.where(store_id: 1).reindex

Reindex associations

ruby
store.products.reindex

Remove old indices

ruby
Product.search_index.clean_indices

Use custom settings

ruby
class Product < ApplicationRecord
  searchkick settings: {number_of_shards: 3}
end

Use a different index name

ruby
class Product < ApplicationRecord
  searchkick index_name: "products_v2"
end

Use a dynamic index name

ruby
class Product < ApplicationRecord
  searchkick index_name: -> { "#{name.tableize}-#{I18n.locale}" }
end

Prefix the index name

ruby
class Product < ApplicationRecord
  searchkick index_prefix: "datakick"
end

For all models

ruby
Searchkick.index_prefix = "datakick"

Use a different term for boosting by conversions

ruby
Product.search("banana").conversions_v2(term: "organic banana")

Define multiple conversion fields

ruby
class Product < ApplicationRecord
  has_many :searches, class_name: "Searchjoy::Search"

  searchkick conversions_v2: ["unique_conversions", "total_conversions"]

  def search_data
    {
      name: name,
      unique_conversions: searches.group(:query).distinct.count(:user_id),
      total_conversions: searches.group(:query).count
    }
  end
end

And specify which to use

ruby
Product.search("banana") # boost by both fields (default)
Product.search("banana").conversions_v2("total_conversions") # only boost by total_conversions
Product.search("banana").conversions_v2(false) # no conversion boosting

Change timeout

ruby
Searchkick.timeout = 15 # defaults to 10

Set a lower timeout for searches

ruby
Searchkick.search_timeout = 3

Change the search method name

ruby
Searchkick.search_method_name = :lookup

Change the queue name

ruby
Searchkick.queue_name = :search_reindex # defaults to :searchkick

Change the queue name or priority for a model

ruby
class Product < ApplicationRecord
  searchkick job_options: {queue: "critical", priority: 10}
end

Change the queue name or priority for a specific call

ruby
Product.reindex(mode: :async, job_options: {queue: "critical", priority: 10})

Change the parent job

ruby
Searchkick.parent_job = "ApplicationJob" # defaults to "ActiveJob::Base"

Eager load associations

ruby
Product.search("milk").includes(:brand, :stores)

Eager load different associations by model

ruby
Searchkick.search("*").models(Product, Store).model_includes(Product => [:store], Store => [:product])

Run additional scopes on results

ruby
Product.search("milk").scope_results(->(r) { r.with_attached_images })

Set opaque id for slow logs

ruby
Product.search("milk").opaque_id("some-id")
# or
Searchkick.multi_search(searches, opaque_id: "some-id")

Specify default fields to search

ruby
class Product < ApplicationRecord
  searchkick default_fields: [:name]
end

Turn off special characters

ruby
class Product < ApplicationRecord
  # A will not match Ä
  searchkick special_characters: false
end

Turn on stemming for conversions

ruby
class Product < ApplicationRecord
  searchkick stem_conversions: true
end

Make search case-sensitive

ruby
class Product < ApplicationRecord
  searchkick case_sensitive: true
end

Note: If misspellings are enabled (default), results with a single character case difference will match. Turn off misspellings if this is not desired.

Change import batch size

ruby
class Product < ApplicationRecord
  searchkick batch_size: 200 # defaults to 1000
end

Create index without importing

ruby
Product.reindex(import: false)

Use a different id

ruby
class Product < ApplicationRecord
  def search_document_id
    custom_id
  end
end

Add request parameters like search_type

ruby
Product.search("carrots").request_params(search_type: "dfs_query_then_fetch")

Set options across all models

ruby
Searchkick.model_options = {
  batch_size: 200
}

Reindex conditionally

ruby
class Product < ApplicationRecord
  searchkick callback_options: {if: :search_data_changed?}

  def search_data_changed?
    previous_changes.include?("name")
  end
end

Reindex all models - Rails only

sh
rake searchkick:reindex:all

Turn on misspellings after a certain number of characters

ruby
Product.search("api").misspellings(prefix_length: 2) # api, apt, no ahi

BigDecimal values are indexed as floats by default so they can be used for boosting. Convert them to strings to keep full precision.

ruby
class Product < ApplicationRecord
  def search_data
    {
      units: units.to_s("F")
    }
  end
end

Gotchas

Consistency

Elasticsearch and OpenSearch are eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the refresh method to have it show up immediately.

ruby
product.save!
Product.search_index.refresh

Inconsistent Scores

Due to the distributed nature of Elasticsearch and OpenSearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:

ruby
class Product < ApplicationRecord
  searchkick settings: {number_of_shards: 1}
end

For convenience, this is set by default in the test environment.

Upgrading

6.0

Searchkick 6 brings a new query builder API:

ruby
Product.search("apples").where(in_stock: true).limit(10).offset(50)

All existing options can be used as methods, or you can continue to use the existing API.

This release also significantly improves the performance of searches when using conversions. To upgrade conversions without downtime, add conversions_v2 to your model and an additional field to search_data:

ruby
class Product < ApplicationRecord
  searchkick conversions: [:conversions], conversions_v2: [:conversions_v2]

  def search_data
    conversions = searches.group(:query).distinct.count(:user_id)
    {
      conversions: conversions,
      conversions_v2: conversions
    }
  end
end

Reindex, then remove conversions:

ruby
class Product < ApplicationRecord
  searchkick conversions_v2: [:conversions_v2]

  def search_data
    {
      conversions_v2: searches.group(:query).distinct.count(:user_id)
    }
  end
end

Other improvements include the option to ignore errors for missing documents with partial reindexing and more customization for background jobs. Check out the changelog for the full list of changes.

History

View the changelog

Thanks

Thanks to Karel Minarik for Elasticsearch Ruby and Tire, Jaroslav Kalistsuk for zero downtime reindexing, and Alex Leschenko for Elasticsearch autocomplete.

Contributing

Everyone is encouraged to help improve this project. Here are a few ways you can help:

To get started with development:

sh
git clone https://github.com/ankane/searchkick.git
cd searchkick
bundle install
bundle exec rake test

Feel free to open an issue to get feedback on your idea before spending too much time on it.