Intelligent search made easy with Rails and Elasticsearch
:rocket: Intelligent search made easy
Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.
Searchkick handles:
tomatoesmatches
tomato
jalapenomatches
jalapeño
dishwashermatches
dish washer
zuchinimatches
zucchini
popmatches
soda
Plus:
Check out Searchjoy for analytics and Autosuggest for query suggestions
:tangerine: Battle-tested at Instacart
Install Elasticsearch. For Homebrew, use:
brew install elasticsearch brew services start elasticsearch
Add this line to your application’s Gemfile:
gem 'searchkick'
The latest version works with Elasticsearch 6 and 7. For Elasticsearch 5, use version 3.1.3 and this readme.
Add searchkick to models you want to search.
class Product < ApplicationRecord searchkick end
Add data to the search index.
Product.reindex
And to query, use:
products = Product.search("apples") products.each do |product| puts product.name end
Searchkick supports the complete Elasticsearch Search API. As your search becomes more advanced, we recommend you use the Elasticsearch DSL for maximum flexibility.
Query like SQL
Product.search "apples", where: {in_stock: true}, limit: 10, offset: 50
Search specific fields
fields: [:name, :brand]
Where
where: { expires_at: {gt: Time.now}, # lt, gte, lte also available orders_count: 1..10, # equivalent to {gte: 1, lte: 10} aisle_id: [25, 30], # in store_id: {not: 2}, # not aisle_id: {not: [25, 30]}, # not in user_ids: {all: [1, 3]}, # all elements in array category: {like: "%frozen%"}, # like category: /frozen .+/, # regexp category: {prefix: "frozen"}, # prefix store_id: {exists: true}, # exists _or: [{in_stock: true}, {backordered: true}], _and: [{in_stock: true}, {backordered: true}], _not: {store_id: 1} # negate a condition }
Order
order: {_score: :desc} # most relevant first - default
All of these sort options are supported
Limit / offset
limit: 20, offset: 40
Select
select: [:name]
These source filtering options are supported
Searches return a
Searchkick::Resultsobject. This responds like an array to most methods.
results = Product.search("milk") results.size results.any? results.each { |result| ... }
By default, ids are fetched from Elasticsearch and records are fetched from your database. To fetch everything from Elasticsearch, use:
Product.search("apples", load: false)
Get total results
results.total_count
Get the time the search took (in milliseconds)
results.took
Get the full response from Elasticsearch
results.response
Note: By default, Elasticsearch limits paging to the first 10,000 results for performance. With Elasticsearch 7, this applies to the total count as well.
Boost important fields
fields: ["title^10", "description"]
Boost by the value of a field (field must be numeric)
boost_by: [:orders_count] # give popular documents a little boost boost_by: {orders_count: {factor: 10}} # default factor is 1
Boost matching documents
boost_where: {user_id: 1} boost_where: {user_id: {value: 1, factor: 100}} # default factor is 1000 boost_where: {user_id: [{value: 1, factor: 100}, {value: 2, factor: 200}]}
Boost by recency
boost_by_recency: {created_at: {scale: "7d", decay: 0.5}}
You can also boost by:
Use a
*for the query.
Product.search "*"
Plays nicely with kaminari and will_paginate.
# controller @products = Product.search "milk", page: params[:page], per_page: 20
View with kaminari
View with will_paginate
By default, results must match all words in the query.
Product.search "fresh honey" # fresh AND honey
To change this, use:
Product.search "fresh honey", operator: "or" # fresh OR honey
By default, results must match the entire word -
backwill not match
backpack. You can change this behavior with:
class Product < ApplicationRecord searchkick word_start: [:name] end
And to search (after you reindex):
Product.search "back", fields: [:name], match: :word_start
Available options are:
Option |
Matches | Example |
---|
:word| entire word |
applematches
apple
:word_start| start of word |
appmatches
apple
:word_middle| any part of word |
pplmatches
apple
:word_end| end of word |
plematches
apple
:text_start| start of text |
grematches
green apple,
appdoes not match
:text_middle| any part of text |
een appmatches
green apple
:text_end| end of text |
plematches
green apple,
eendoes not match
The default is
:word. The most matches will happen with
:word_middle.
To match a field exactly (case-sensitive), use:
User.search query, fields: [{email: :exact}, :name]
To only match the exact order, use:
User.search "fresh honey", match: :phrase
Searchkick stems words by default for better matching.
appleand
applesboth stem to
appl, so searches for either term will have the same matches.
Searchkick defaults to English for stemming. To change this, use:
class Product < ApplicationRecord searchkick language: "german" end
See the list of languages. A few languages require plugins:
chinese- analysis-ik plugin
chinese2- analysis-smartcn plugin
japanese- analysis-kuromoji plugin
korean- analysis-openkoreantext plugin
korean2- analysis-nori plugin
polish- analysis-stempel plugin
ukrainian- analysis-ukrainian plugin
vietnamese- analysis-vietnamese plugin
You can also use a Hunspell dictionary for stemming.
class Product < ApplicationRecord searchkick stemmer: {type: "hunspell", locale: "en_US"} end
Disable stemming with:
class Image < ApplicationRecord searchkick stem: false end
Exclude certain words from stemming with:
class Image < ApplicationRecord searchkick stem_exclusion: ["apples"] end
Or change how words are stemmed:
class Image < ApplicationRecord searchkick stemmer_override: ["apples => other"] end
class Product < ApplicationRecord searchkick search_synonyms: [["pop", "soda"], ["burger", "hamburger"]] end
Call
Product.reindexafter changing synonyms. Synonyms are applied at search time before stemming, and can be a single word or multiple words.
For directional synonyms, use:
search_synonyms: ["lightbulb => halogenlamp"]
The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex.
For Elasticsearch 7.3+, we recommend placing synonyms in a file on the Elasticsearch server (in the
configdirectory). This allows you to reload synonyms without reindexing.
pop, soda burger, hamburger
Then use:
search_synonyms: "synonyms.txt"
Add elasticsearch-xpack to your Gemfile:
gem 'elasticsearch-xpack', '>= 7.8.0'
And use:
Product.search_index.reload_synonyms
You can use a library like ActsAsTaggableOn and do:
class Product < ApplicationRecord acts_as_taggable scope :search_import, -> { includes(:tags) }def search_data { name_tagged: "#{name} #{tags.map(&:name).join(" ")}" } end end
Search with:
Product.search query, fields: [:name_tagged]
By default, Searchkick handles misspelled queries by returning results with an edit distance of one.
You can change this with:
Product.search "zucini", misspellings: {edit_distance: 2} # zucchini
To prevent poor precision and improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.
Product.search "zuchini", misspellings: {below: 5}
If there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.
Turn off misspellings with:
Product.search "zuchini", misspellings: false # no zucchini
Specify which fields can include misspellings with:
Product.search "zucini", fields: [:name, :color], misspellings: {fields: [:name]}
When doing this, you must also specify fields to search
If a user searches
butter, they may also get results for
peanut butter. To prevent this, use:
Product.search "butter", exclude: ["peanut butter"]
You can map queries and terms to exclude with:
exclude_queries = { "butter" => ["peanut butter"], "cream" => ["ice cream", "whipped cream"] }Product.search query, exclude: exclude_queries[query]
You can demote results by boosting by a factor less than one:
Product.search("butter", boost_where: {category: {value: "pantry", factor: 0.5}})
Search :ice_cream::cake: and get
ice cream cake!
Add this line to your application’s Gemfile:
gem 'gemoji-parser'
And use:
Product.search "🍨🍰", emoji: true
Control what data is indexed with the
search_datamethod. Call
Product.reindexafter changing this method.
class Product < ApplicationRecord belongs_to :departmentdef search_data { name: name, department_name: department.name, on_sale: sale_price.present? } end end
Searchkick uses
find_in_batchesto import documents. To eager load associations, use the
search_importscope.
class Product < ApplicationRecord scope :search_import, -> { includes(:department) } end
By default, all records are indexed. To control which records are indexed, use the
should_index?method together with the
search_importscope.
class Product < ApplicationRecord scope :search_import, -> { where(active: true) }def should_index? active # only index active records end end
If a reindex is interrupted, you can resume it with:
Product.reindex(resume: true)
For large data sets, try parallel reindexing.
search_datamethod
searchkickmethod
There are four strategies for keeping the index synced with your database.
Anytime a record is inserted, updated, or deleted
Use background jobs for better performance
class Product < ApplicationRecord searchkick callbacks: :async end
Jobs are added to a queue named
searchkick.
Push ids of records that need updated to a queue and reindex in the background in batches. This is more performant than the asynchronous method, which updates records individually. See how to set up.
Turn off automatic syncing
class Product < ApplicationRecord searchkick callbacks: false end
You can also do bulk updates.
Searchkick.callbacks(:bulk) do User.find_each(&:update_fields) end
Or temporarily skip updates.
Searchkick.callbacks(false) do User.find_each(&:update_fields) end
Data is not automatically synced when an association is updated. If this is desired, add a callback to reindex:
class Image < ApplicationRecord belongs_to :productafter_commit :reindex_product
def reindex_product product.reindex end end
The best starting point to improve your search by far is to track searches and conversions. Searchjoy makes it easy.
Product.search "apple", track: {user_id: current_user.id}
See the docs for how to install and use.
Focus on:
Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
Add conversion data with:
class Product < ApplicationRecord has_many :searches, class_name: "Searchjoy::Search", as: :convertablesearchkick conversions: [:conversions] # name of field
def search_data { name: name, conversions: searches.group(:query).uniq.count(:user_id) # {"ice cream" => 234, "chocolate" => 67, "cream" => 2} } end end
Reindex and set up a cron job to add new conversions daily.
rake searchkick:reindex CLASS=Product
This can make a huge difference on the quality of your search.
For a more performant way to reindex conversion data, check out performant conversions.
Order results differently for each user. For example, show a user’s previously purchased products before other results.
class Product < ApplicationRecord def search_data { name: name, orderer_ids: orders.pluck(:user_id) # boost this product for these users } end end
Reindex and search with:
Product.search "milk", boost_where: {orderer_ids: current_user.id}
Autocomplete predicts what a user will type, making the search experience faster and easier.
Note: To autocomplete on search terms rather than results, check out Autosuggest.
Note 2: If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s much faster to load all records into JavaScript and autocomplete there (eliminates network requests).
First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.
class Movie < ApplicationRecord searchkick word_start: [:title, :director] end
Reindex and search with:
Movie.search "jurassic pa", fields: [:title], match: :word_start
Typically, you want to use a JavaScript library like typeahead.js or jQuery UI.
First, add a route and controller action.
class MoviesController < ApplicationController def autocomplete render json: Movie.search(params[:query], { fields: ["title^5", "director"], match: :word_start, limit: 10, load: false, misspellings: {below: 5} }).map(&:title) end end
Note: Use
load: falseand
misspellings: {below: n}(or
misspellings: false) for best performance.
Then add the search box and JavaScript code to a view.
class Product < ApplicationRecord searchkick suggest: [:name] # fields to generate suggestions end
Reindex and search with:
products = Product.search "peantu butta", suggest: true products.suggestions # ["peanut butter"]
Aggregations provide aggregated search data.
products = Product.search "chuck taylor", aggs: [:product_type, :gender, :brand] products.aggs
By default,
whereconditions apply to aggregations.
Product.search "wingtips", where: {color: "brandy"}, aggs: [:size] # aggregations for brandy wingtips are returned
Change this with:
Product.search "wingtips", where: {color: "brandy"}, aggs: [:size], smart_aggs: false # aggregations for all wingtips are returned
Set
whereconditions for each aggregation separately with:
Product.search "wingtips", aggs: {size: {where: {color: "brandy"}}}
Limit
Product.search "apples", aggs: {store_id: {limit: 10}}
Order
Product.search "wingtips", aggs: {color: {order: {"_key" => "asc"}}} # alphabetically
All of these options are supported
Ranges
price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}] Product.search "*", aggs: {price: {ranges: price_ranges}}
Minimum document count
Product.search "apples", aggs: {store_id: {min_doc_count: 2}}
Script support
Product.search "*", aggs: {color: {script: {source: "'Color: ' + _value"}}}
Date histogram
Product.search "pear", aggs: {products_per_year: {date_histogram: {field: :created_at, interval: :year}}}
For other aggregation types, including sub-aggregations, use
body_options:
Product.search "orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}}
Specify which fields to index with highlighting.
class Product < ApplicationRecord searchkick highlight: [:name] end
Highlight the search query in the results.
bands = Band.search "cinema", highlight: true
View the highlighted fields with:
bands.with_highlights.each do |band, highlights| highlights[:name] # "Two Door Cinema Club" end
To change the tag, use:
Band.search "cinema", highlight: {tag: ""}
To highlight and search different fields, use:
Band.search "cinema", fields: [:name], highlight: {fields: [:description]}
By default, the entire field is highlighted. To get small snippets instead, use:
bands = Band.search "cinema", highlight: {fragment_size: 20} bands.with_highlights(multiple: true).each do |band, highlights| highlights[:name].join(" and ") end
Additional options can be specified for each field:
Band.search "cinema", fields: [:name], highlight: {fields: {name: {fragment_size: 200}}}
You can find available highlight options in the Elasticsearch reference.
Find similar items.
product = Product.first product.similar(fields: [:name], where: {size: "12 oz"})
class Restaurant < ApplicationRecord searchkick locations: [:location]def search_data attributes.merge(location: {lat: latitude, lon: longitude}) end end
Reindex and search with:
Restaurant.search "pizza", where: {location: {near: {lat: 37, lon: -114}, within: "100mi"}} # or 160km
Bounded by a box
Restaurant.search "sushi", where: {location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}}}
Note:
top_rightand
bottom_leftalso work
Bounded by a polygon
Restaurant.search "dessert", where: {location: {geo_polygon: {points: [{lat: 38, lon: -123}, {lat: 39, lon: -123}, {lat: 37, lon: 122}]}}}
Boost results by distance - closer results are boosted more
Restaurant.search "noodles", boost_by_distance: {location: {origin: {lat: 37, lon: -122}}}
Also supports additional options
Restaurant.search "wings", boost_by_distance: {location: {origin: {lat: 37, lon: -122}, function: "linear", scale: "30mi", decay: 0.5}}
You can also index and search geo shapes.
class Restaurant < ApplicationRecord searchkick geo_shape: [:bounds]def search_data attributes.merge( bounds: { type: "envelope", coordinates: [{lat: 4, lon: 1}, {lat: 2, lon: 3}] } ) end end
See the Elasticsearch documentation for details.
Find shapes intersecting with the query shape
Restaurant.search "soup", where: {bounds: {geo_shape: {type: "polygon", coordinates: [[{lat: 38, lon: -123}, ...]]}}}
Falling entirely within the query shape
Restaurant.search "salad", where: {bounds: {geo_shape: {type: "circle", relation: "within", coordinates: [{lat: 38, lon: -123}], radius: "1km"}}}
Not touching the query shape
Restaurant.search "burger", where: {bounds: {geo_shape: {type: "envelope", relation: "disjoint", coordinates: [{lat: 38, lon: -123}, {lat: 37, lon: -122}]}}}
Searchkick supports single table inheritance.
class Dog < Animal end
In your parent model, set:
class Animal < ApplicationRecord searchkick inheritance: true end
The parent and child model can both reindex.
Animal.reindex Dog.reindex # equivalent, all animals reindexed
And to search, use:
Animal.search "*" # all animals Dog.search "*" # just dogs Animal.search "*", type: [Dog, Cat] # just cats and dogs
Notes:
The
suggestoption retrieves suggestions from the parent at the moment.
Dog.search "airbudd", suggest: true # suggestions for all animals
This relies on a
typefield that is automatically added to the indexed document. Be wary of defining your own
typefield in
search_data, as it will take precedence.
To help with debugging queries, you can use:
Product.search("soap", debug: true)
This prints useful info to
stdout.
See how Elasticsearch scores your queries with:
Product.search("soap", explain: true).response
See how Elasticsearch tokenizes your queries with:
Product.search_index.tokens("Dish Washer Soap", analyzer: "searchkick_index") # ["dish", "dishwash", "washer", "washersoap", "soap"]Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search")
["dishwashersoap"] - no match
Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search2")
["dishwash", "soap"] - match!!
Partial matches
Product.search_index.tokens("San Diego", analyzer: "searchkick_word_start_index") # ["s", "sa", "san", "d", "di", "die", "dieg", "diego"]Product.search_index.tokens("dieg", analyzer: "searchkick_word_search")
["dieg"] - match!!
See the complete list of analyzers.
As you iterate on your search, it’s a good idea to add tests.
For performance, only enable Searchkick callbacks for the tests that need it.
Rails 6 enables parallel tests by default. Add to your
test/test_helper.rb:
class ActiveSupport::TestCase parallelize_setup do |worker| Searchkick.index_suffix = worker# reindex models Product.reindex # and disable callbacks Searchkick.disable_callbacks
end end
And use:
class ProductTest < ActiveSupport::TestCase def setup Searchkick.enable_callbacks enddef teardown Searchkick.disable_callbacks end
def test_search Product.create!(name: "Apple") Product.search_index.refresh assert_equal ["Apple"], Product.search("apple").map(&:name) end end
Add to your
test/test_helper.rb:
# reindex models Product.reindexand disable callbacks
Searchkick.disable_callbacks
And use:
class ProductTest < Minitest::Test def setup Searchkick.enable_callbacks enddef teardown Searchkick.disable_callbacks end
def test_search Product.create!(name: "Apple") Product.search_index.refresh assert_equal ["Apple"], Product.search("apple").map(&:name) end end
Add to your
spec/spec_helper.rb:
RSpec.configure do |config| config.before(:suite) do # reindex models Product.reindex# and disable callbacks Searchkick.disable_callbacks
end
config.around(:each, search: true) do |example| Searchkick.callbacks(nil) do example.run end end end
And use:
describe Product, search: true do it "searches" do Product.create!(name: "Apple") Product.search_index.refresh assert_equal ["Apple"], Product.search("apple").map(&:name) end end
Use a trait and an after
createhook for each indexed model:
FactoryBot.define do factory :product do # ...# Note: This should be the last trait in the list so `reindex` is called # after all the other callbacks complete. trait :reindex do after(:create) do |product, _evaluator| product.reindex(refresh: true) end end
end end
use it
FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")
Check out setup-elasticsearch for an easy way to install Elasticsearch.
- uses: ankane/[email protected]
Searchkick uses
ENV["ELASTICSEARCH_URL"]for the Elasticsearch server. This defaults to
http://localhost:9200.
Create an initializer
config/initializers/elasticsearch.rbwith:
ENV["ELASTICSEARCH_URL"] = "https://user:[email protected]:port"
Then deploy and reindex:
rake searchkick:reindex:all
Choose an add-on: Bonsai, SearchBox, or Elastic Cloud.
For Bonsai:
heroku addons:create bonsai heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
For SearchBox:
heroku addons:create searchbox:starter heroku config:set ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`
For Elastic Cloud (previously Found):
heroku addons:create foundelasticsearch heroku addons:open foundelasticsearch
Visit the Shield page and reset your password. You’ll need to add the username and password to your url. Get the existing url with:
heroku config:get FOUNDELASTICSEARCH_URL
And add
elastic:[email protected]right after
https://and add port
9243at the end:
heroku config:set ELASTICSEARCH_URL=https://elastic:[email protected]:9243
Then deploy and reindex:
heroku run rake searchkick:reindex:all
Create an initializer
config/initializers/elasticsearch.rbwith:
ENV["ELASTICSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"
To use signed requests, include in your Gemfile:
gem 'faraday_middleware-aws-sigv4'
and add to your initializer:
Searchkick.aws_credentials = { access_key_id: ENV["AWS_ACCESS_KEY_ID"], secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"], region: "us-east-1" }
Then deploy and reindex:
rake searchkick:reindex:all
Create an initializer
config/initializers/elasticsearch.rbwith:
ENV["ELASTICSEARCH_URL"] = "https://user:[email protected]:port"
Then deploy and reindex:
rake searchkick:reindex:all
We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send personal data of your users to Elasticsearch.
Bonsai, Elastic Cloud, and Amazon Elasticsearch all support encryption at rest and HTTPS.
Create an initializer
config/initializers/elasticsearch.rbwith multiple hosts:
ENV["ELASTICSEARCH_URL"] = "https://user:[email protected],https://user:[email protected]"
See elasticsearch-transport for a complete list of options.
Add the following to
config/environments/production.rb:
config.lograge.custom_options = lambda do |event| options = {} options[:search] = event.payload[:searchkick_runtime] if event.payload[:searchkick_runtime].to_f > 0 options end
See Production Rails for other good practices.
Significantly increase performance with faster JSON generation. Add Oj to your Gemfile.
gem 'oj'
This speeds up all JSON generation and parsing in your application (automatically!)
Significantly increase performance with persistent HTTP connections. Add Typhoeus to your Gemfile and it’ll automatically be used.
gem 'typhoeus'
To reduce log noise, create an initializer with:
Ethon.logger = Logger.new(nil)
If you run into issues on Windows, check out this post.
By default, all string fields are searchable (can be used in
fieldsoption). Speed up indexing and reduce index size by only making some fields searchable.
class Product < ApplicationRecord searchkick searchable: [:name] end
By default, all string fields are filterable (can be used in
whereoption). Speed up indexing and reduce index size by only making some fields filterable.
class Product < ApplicationRecord searchkick filterable: [:brand] end
Note: Non-string fields are always filterable and should not be passed to this option.
For large data sets, you can use background jobs to parallelize reindexing.
Product.reindex(async: true) # {index_name: "products_production_20170111210018065"}
Once the jobs complete, promote the new index with:
Product.search_index.promote(index_name)
You can optionally track the status with Redis:
Searchkick.redis = Redis.new
And use:
Searchkick.reindex_status(index_name)
You can also have Searchkick wait for reindexing to complete
Product.reindex(async: {wait: true})
You can use ActiveJob::TrafficControl to control concurrency. Install the gem:
gem 'activejob-traffic_control', '>= 0.1.3'
And create an initializer with:
ActiveJob::TrafficControl.client = Searchkick.redisclass Searchkick::BulkReindexJob concurrency 3 end
This will allow only 3 jobs to run at once.
You can specify a longer refresh interval while reindexing to increase performance.
Product.reindex(async: true, refresh_interval: "30s")
Note: This only makes a noticable difference with parallel reindexing.
When promoting, have it restored to the value in your mapping (defaults to
1s).
Product.search_index.promote(index_name, update_refresh_interval: true)
Push ids of records needing reindexing to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using connection_pool.
Searchkick.redis = ConnectionPool.new { Redis.new }
And ask your models to queue updates.
class Product < ApplicationRecord searchkick callbacks: :queue end
Then, set up a background job to run.
Searchkick::ProcessQueueJob.perform_later(class_name: "Product")
You can check the queue length with:
Product.search_index.reindex_queue.length
For more tips, check out Keeping Elasticsearch in Sync.
Searchkick supports Elasticsearch’s routing feature, which can significantly speed up searches.
class Business < ApplicationRecord searchkick routing: truedef search_routing city_id end end
Reindex and search with:
Business.search "ice cream", routing: params[:city_id]
Reindex a subset of attributes to reduce time spent generating search data and cut down on network traffic.
class Product < ApplicationRecord def search_data { name: name }.merge(search_prices) enddef search_prices { price: price, sale_price: sale_price } end end
And use:
Product.reindex(:search_prices)
Split out conversions into a separate method so you can use partial reindexing, and cache conversions to prevent N+1 queries. Be sure to use a centralized cache store like Memcached or Redis.
class Product < ApplicationRecord def search_data { name: name }.merge(search_conversions) enddef search_conversions { conversions: Rails.cache.read("search_conversions:#{self.class.name}:#{id}") || {} } end end
Create a job to update the cache and reindex records with new conversions.
class ReindexConversionsJob < ApplicationJob def perform(class_name) # get records that have a recent conversion recently_converted_ids = Searchjoy::Search.where("convertable_type = ? AND converted_at > ?", class_name, 1.day.ago) .order(:convertable_id).uniq.pluck(:convertable_id)# split into groups recently_converted_ids.in_groups_of(1000, false) do |ids| # fetch conversions conversions = Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name) .group(:convertable_id, :query).uniq.count(:user_id) # group conversions by record conversions_by_record = {} conversions.each do |(id, query), count| (conversions_by_record[id] ||= {})[query] = count end # write to cache conversions_by_record.each do |id, conversions| Rails.cache.write("search_conversions:#{class_name}:#{id}", conversions) end # partial reindex class_name.constantize.where(id: ids).reindex(:search_conversions) end
end end
Run the job with:
ReindexConversionsJob.perform_later("Product")
Searchkick makes it easy to use the Elasticsearch DSL on its own.
Create a custom mapping:
class Product < ApplicationRecord searchkick mappings: { properties: { name: {type: "keyword"} } } end
Note: If you use a custom mapping, you'll need to use custom searching as well.
To keep the mappings and settings generated by Searchkick, use:
class Product < ApplicationRecord searchkick merge_mappings: true, mappings: {...} end
And use the
bodyoption to search:
products = Product.search body: {query: {match: {name: "milk"}}}
View the response with:
products.response
To modify the query generated by Searchkick, use:
products = Product.search "milk", body_options: {min_score: 1}
or
products = Product.search "apples" do |body| body[:min_score] = 1 end
Searchkick is built on top of the elasticsearch gem. To access the client directly, use:
Searchkick.client
To batch search requests for performance, use:
products = Product.search("snacks", execute: false) coupons = Coupon.search("snacks", execute: false) Searchkick.multi_search([products, coupons])
Then use
productsand
couponsas typical results.
Note: Errors are not raised as with single requests. Use the
errormethod on each query to check for errors.
Search across multiple models with:
Searchkick.search "milk", models: [Product, Category]
Boost specific models with:
indices_boost: {Category => 2, Product => 1}
Check out this great post on the Apartment gem. Follow a similar pattern if you use another gem.
Searchkick also supports the scroll API. Scrolling is not intended for real time user requests, but rather for processing large amounts of data.
Product.search("*", scroll: "1m").scroll do |batch| # process batch ... end
You can also scroll batches manually.
products = Product.search "*", scroll: "1m" while products.any? # process batch ...products = products.scroll end
products.clear_scroll
By default, Elasticsearch limits paging to the first 10,000 results. Here’s why. We don’t recommend changing this, but if you really need all results, you can use:
class Product < ApplicationRecord searchkick deep_paging: true end
If you just need an accurate total count with Elasticsearch 7, you can instead use:
Product.search("pears", body_options: {track_total_hits: true})
To query nested data, use dot notation.
User.search "san", fields: ["address.city"], where: {"address.zip_code" => 12345}
Reindex one record
product = Product.find(1) product.reindex
Reindex multiple records
Product.where(store_id: 1).reindex
Reindex associations
store.products.reindex
Remove old indices
Product.search_index.clean_indices
Use custom settings
class Product < ApplicationRecord searchkick settings: {number_of_shards: 3} end
Use a different index name
class Product < ApplicationRecord searchkick index_name: "products_v2" end
Use a dynamic index name
class Product < ApplicationRecord searchkick index_name: -> { "#{name.tableize}-#{I18n.locale}" } end
Prefix the index name
class Product < ApplicationRecord searchkick index_prefix: "datakick" end
For all models
Searchkick.index_prefix = "datakick"
Use a different term for boosting by conversions
Product.search("banana", conversions_term: "organic banana")
Multiple conversion fields
class Product < ApplicationRecord has_many :searches, class_name: "Searchjoy::Search"searchkick also supports multiple "conversions" fields
searchkick conversions: ["unique_user_conversions", "total_conversions"]
def search_data { name: name, unique_user_conversions: searches.group(:query).uniq.count(:user_id), # {"ice cream" => 234, "chocolate" => 67, "cream" => 2} total_conversions: searches.group(:query).count # {"ice cream" => 412, "chocolate" => 117, "cream" => 6} } end end
and during query time:
Product.search("banana") # boost by both fields (default) Product.search("banana", conversions: "total_conversions") # only boost by total_conversions Product.search("banana", conversions: false) # no conversion boosting
Change timeout
Searchkick.timeout = 15 # defaults to 10
Set a lower timeout for searches
Searchkick.search_timeout = 3
Change the search method name
Searchkick.search_method_name = :lookup
Change search queue name
Searchkick.queue_name = :search_reindex
Eager load associations
Product.search "milk", includes: [:brand, :stores]
Eager load different associations by model
Searchkick.search("*", models: [Product, Store], model_includes: {Product => [:store], Store => [:product]})
Run additional scopes on results
Product.search "milk", scope_results: ->(r) { r.with_attached_images }
Specify default fields to search
class Product < ApplicationRecord searchkick default_fields: [:name] end
Turn off special characters
class Product < ApplicationRecord # A will not match Ä searchkick special_characters: false end
Turn on stemming for conversions
class Product < ApplicationRecord searchkick stem_conversions: true end
Make search case-sensitive
class Product < ApplicationRecord searchkick case_sensitive: true end
Note: If misspellings are enabled (default), results with a single character case difference will match. Turn off misspellings if this is not desired.
Change import batch size
class Product < ApplicationRecord searchkick batch_size: 200 # defaults to 1000 end
Create index without importing
Product.reindex(import: false)
Use a different id
class Product < ApplicationRecord def search_document_id custom_id end end
Lazy searching
products = Product.search("carrots", execute: false) products.each { ... } # search not executed until here
Add request parameters, like
search_typeand
query_cache
Product.search("carrots", request_params: {search_type: "dfs_query_then_fetch"})
Set options across all models
Searchkick.model_options = { batch_size: 200 }
Reindex conditionally
class Product < ApplicationRecord searchkick callbacks: falseadd the callbacks manually
after_commit :reindex, if: -> (model) { model.previous_changes.key?("name") } # use your own condition end
Reindex all models - Rails only
rake searchkick:reindex:all
Turn on misspellings after a certain number of characters
Product.search "api", misspellings: {prefix_length: 2} # api, apt, no ahi
Note: With this option, if the query length is the same as
prefix_length, misspellings are turned off
Product.search "ah", misspellings: {prefix_length: 2} # ah, no aha
Elasticsearch is eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the
refreshmethod to have it show up immediately.
product.save! Product.search_index.refresh
Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:
class Product < ApplicationRecord searchkick settings: {number_of_shards: 1} end
For convenience, this is set by default in the test environment.
View the changelog.
Thanks to Karel Minarik for Elasticsearch Ruby and Tire, Jaroslav Kalistsuk for zero downtime reindexing, and Alex Leschenko for Elasticsearch autocomplete.
Everyone is encouraged to help improve this project. Here are a few ways you can help:
To get started with development:
git clone https://github.com/ankane/searchkick.git cd searchkick bundle install bundle exec rake test
Feel free to open an issue to get feedback on your idea before spending too much time on it.