by mrkamel

mrkamel / search_flip

Full-Featured ElasticSearch Ruby Client with a Chainable DSL

254 Stars 5 Forks Last release: 29 days ago (3.1.2) MIT License 438 Commits 12 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:


Full-Featured Elasticsearch Ruby Client with a Chainable DSL

Build Status Gem Version

Using SearchFlip it is dead-simple to create index classes that correspond to Elasticsearch indices and to manipulate, query and aggregate these indices using a chainable, concise, yet powerful DSL. Finally, SearchFlip supports Elasticsearch 2.x, 5.x, 6.x, 7.x. Check section Feature Support for version dependent features."hello world", default_field: "title").where(visible: true).aggregate(:user_id).sort(id: "desc")

CommentIndex.aggregate(:user_id) do |aggregation| aggregation.aggregate(histogram: { date_histogram: { field: "created_at", interval: "month" }}) end

CommentIndex.range(:created_at, gt: - 1.week, lt: ["approved", "pending"])

Updating from previous SearchFlip versions

Checkout for detailed instructions.

Comparison with other gems

There are great ruby gems to work with Elasticsearch like e.g. searchkick and elasticsearch-ruby already. However, they don't have a chainable API. Compare yourself.

# elasticsearch-ruby
  query: {
    query_string: {
      query: "hello world",
      default_operator: "AND"

searchkick"hello world", where: { available: true }, order: { id: "desc" }, aggs: [:username])

search_flip"hello world").where(available: true).sort(id: "desc").aggregate(:username)

Finally, SearchFlip comes with a minimal set of dependencies (http-rb, hashie and oj only).

Reference Docs

SearchFlip has a great documentation. Check youself at


Add this line to your application's Gemfile:

gem 'search_flip'

and then execute

$ bundle

or install it via

$ gem install search_flip


You can change global config options like:

SearchFlip::Config[:environment] = "development"
SearchFlip::Config[:base_url] = ""

Available config options are:

  • index_prefix
    to have a prefix added to your index names automatically. This can be useful to separate the indices of e.g. testing and development environments.
  • base_url
    to tell SearchFlip how to connect to your cluster
  • bulk_limit
    a global limit for bulk requests
  • bulk_max_mb
    a global limit for the payload of bulk requests
  • auto_refresh
    tells SearchFlip to automatically refresh an index after import, index, delete, etc operations. This is e.g. useful for testing, etc. Defaults to false.


First, create a separate class for your index and include

class CommentIndex
  include SearchFlip::Index

Then tell the Index about the index name, the corresponding model and how to serialize the model for indexing.

class CommentIndex
  include SearchFlip::Index

def self.index_name "comments" end

def self.model Comment end

def self.serialize(comment) { id:, username: comment.username, title: comment.title, message: comment.message } end end

Optionally, you can specify a custom

, but note that starting with Elasticsearch 7, types are deprecated.
class CommentIndex
  # ...

def self.type_name "comment" end end

You can additionally specify an

which will automatically be applied to scopes, eg. ActiveRecord::Relation objects, passed to
, etc. This can be used to preload associations that are used when serializing records or to restrict the records you want to index.
class CommentIndex
  # ...

def self.index_scope(scope) scope.preload(:user) end end

CommentIndex.import(Comment.all) # => CommentIndex.import(Comment.all.preload(:user))

To specify a custom mapping:

class CommentIndex
  # ...

def self.mapping { properties: { # ... } } end



Please note that you need to specify the mapping without a type name, even for Elasticsearch versions before 7, as SearchFlip will add the type name automatically if neccessary.

To specify index settings:

def self.index_settings
    settings: {
      number_of_shards: 10,
      number_of_replicas: 2

Then you can interact with the index:


Index records (automatically uses the Bulk API):

CommentIndex.import([Comment.find(1), Comment.find(2)])
CommentIndex.import(Comment.where("created_at > ?", - 7.days))

Query records:

# => 2838"title:hello").records

=> [#<comment ...>, #<comment ...>, ...]

CommentIndex.where(username: "mrkamel").total_entries

=> 13


=> {1=>#<:result doc_count="37" ...>, 2=>... }

... </:result>

Please note that you can check the request that will be send to Elasticsearch by calling

on the query:"hello world").sort(id: "desc").aggregate(:username).request
# => {:query=>{:bool=>{:must=>[{:query_string=>{:query=>"hello world", :default_operator=>:AND}}]}}, ...}

Delete records:

# for Elasticsearch >= 2.x and < 5.x, the delete-by-query plugin is required
# for the following query:


or delete manually via the bulk API:

CommentIndex.bulk do |indexer| CommentIndex.match_all.find_each do |record| indexer.delete end end

When indexing or deleting documents, you can pass options to control the bulk indexing and you can use all options provided by the Bulk API:

CommentIndex.import(Comment.first, { bulk_limit: 1_000 }, op_type: "create", routing: "routing_key")

or directly

CommentIndex.create(Comment.first, { bulk_max_mb: 100 }, routing: "routing_key") CommentIndex.update(Comment.first, ...)

Checkout the Elasticsearch Bulk API docs for more info as well as SearchFlip::Bulk for a complete list of available options to control the bulk indexing of SearchFlip.

Working with Elasticsearch Aliases

You can use and manage Elasticsearch Aliases like the following:

class UserIndex
  include SearchFlip::Index

def self.index_name alias_name end

def self.alias_name "users" end end

Then, create an index, import the records and add the alias like:

new_user_index = UserIndex.with_settings(index_name: "users-#{SecureRandom.hex}")
new_user_index.import User.all
new_user.connection.update_aliases(actions: [
  add: { index: new_user_index.index_name, alias: new_user_index.alias_name }

If the alias already exists, you have to remove it as well first within


Please note:

with_settings(index_name: '...')
returns an anonymous (i.e. temporary) class which inherits from UserIndex and overwrites

Chainable Methods

SearchFlip supports even more advanced usages, like e.g. post filters, filtered aggregations or nested aggregations via simple to use API methods.

Query/Filter Criteria Methods

SearchFlip provides powerful methods to query/filter Elasticsearch:

  • where


method feels like ActiveRecord's
and adds a bool filter clause to the request:
CommentIndex.where(reviewed: true)
CommentIndex.where(likes: 0 .. 10_000)
CommentIndex.where(state: ["approved", "rejected"])
  • where_not


method is like
, but excluding the matching documents:
CommentIndex.where_not(id: [1, 2, 3])
  • range


to add a range filter query:
CommentIndex.range(:created_at, gt: - 1.week, lt:
  • filter


to add raw filter queries:
CommentIndex.filter(term: { state: "approved" })
  • should


to add raw should queries:
  { term: { state: "approved" } },
  { term: { user: "mrkamel" } },
  • must


to add raw must queries:
CommentIndex.must(term: { state: "approved" })
  • must_not


, but excluding the matching documents:
CommentIndex.must_not(term: { state: "approved" })
  • search

Adds a query string query, with AND as default operator:"hello world")"state:approved")"username:a*")"state:approved OR state:rejected")"hello world", default_operator: "OR")
  • exists


to add an
  • exists_not


, but excluding the matching documents:
  • match_all

Simply matches all documents:

  • all

Simply returns the criteria as is or an empty criteria when called on the index class directly. Useful for chaining.

  • to_query

Sometimes, you want to convert the constraints of a search flip query to a raw query to e.g. use it in a should clause:

  CommentIndex.range(:likes_count, gt: 10).to_query,"search term").to_query

It returns all added queries and filters, including post filters as a raw query:

CommentIndex.where(state: "new").search("text").to_query
# => {:bool=>{:filter=>[{:term=>{:state=>"new"}}], :must=>[{:query_string=>{:query=>"text", ...}}]}}

Post Query/Filter Criteria Methods

All query/filter criteria methods (

, etc.) are available in post filter mode as well, ie. filters/queries applied after aggregations are calculated. Checkout the Elasticsearch docs for further info.
query = CommentIndex.aggregate(:user_id)
query = query.post_where(reviewed: true)
query = query.post_search("username:a*")

Checkout PostFilterable for a complete API reference.


SearchFlip allows to elegantly specify nested aggregations, no matter how deeply nested:

query = OrderIndex.aggregate(:username, order: { revenue: "desc" }) do |aggregation|
  aggregation.aggregate(revenue: { sum: { field: "price" }})

Generally, aggregation results returned by Elasticsearch are returned as a

, which basically is a
, such that you can access them via:

Still, if you want to get the raw aggregations returned by Elasticsearch, access them without supplying any aggregation name to

query.aggregations # => returns the raw aggregation section

query.aggregations["username"]["buckets"].detect { |bucket| bucket["key"] == "mrkamel" }["revenue"]["value"] # => 238.50

Once again, the criteria methods (

, etc.) are available in aggregations as well:
query = OrderIndex.aggregate(average_price: {}) do |aggregation|
  aggregation = aggregation.match_all
  aggregation = aggregation.where(user_id: if current_user

aggregation.aggregate(average_price: { avg: { field: "price" }}) end


Even various criteria for top hits aggregations can be specified elegantly:

query = ProductIndex.aggregate(sponsored: { top_hits: {} }) do |aggregation|
  aggregation.sort(:rank).highlight(:title).source([:id, :title])

Checkout Aggregatable as well as Aggregation for a complete API reference.


query = CommentIndex.suggest(:suggestion, text: "helo", term: { field: "message" })
query.suggestions(:suggestion).first["text"] # => "hello"


CommentIndex.highlight([:title, :message])
CommentIndex.highlight(:title, require_field_match: false)
CommentIndex.highlight(title: { type: "fvh" })
query = CommentIndex.highlight(:title).search("hello")
query.results[0]._hit.highlight.title # => "hello world"

Other Criteria Methods

There are even more chainable criteria methods to make your life easier. For a full list, checkout the reference docs.

  • source

In case you want to restrict the returned fields, simply specify the fields via

CommentIndex.source([:id, :message]).search("hello world")
  • paginate

SearchFlip supports will_paginate and kaminari compatible pagination. Thus, you can either use

in combination with
CommentIndex.paginate(page: 3, per_page: 50)
  • profile


to enable query profiling:
query = CommentIndex.profile(true)
query.raw_response["profile"] # => { "shards" => ... }
  • preload

Uses the well known methods from ActiveRecord to load associated database records when fetching the respective records themselves. Works with other ORMs as well, if supported.


CommentIndex.preload(:user, :post).records
PostIndex.includes(comments: :user).records


CommentIndex.eager_load(:user, :post).records
PostIndex.eager_load(comments: :user).records


CommentIndex.includes(:user, :post).records
PostIndex.includes(comments: :user).records
  • find_in_batches

Used to fetch and yield records in batches using the ElasicSearch scroll API. The batch size and scroll API timeout can be specified."hello world").find_in_batches(batch_size: 100) do |batch|
  # ...
  • find_results_in_batches

Used like

, but yielding the raw results (as
objects) instead of database records."hello world").find_results_in_batches(batch_size: 100) do |batch|
  # ...
  • find_each


but yielding one record at a time."hello world").find_each(batch_size: 100) do |record|
  # ...
  • find_each_result


, but yielding one record at a time."hello world").find_each_result(batch_size: 100) do |batch|
  # ...
  • scroll

You can as well use the underlying scroll API directly, ie. without using higher level scrolling:

query = CommentIndex.scroll(timeout: "5m")

until query.records.empty?


query = query.scroll(id: query.scroll_id, timeout: "5m") end

  • failsafe


to prevent any exceptions from being raised for query string syntax errors or Elasticsearch being unavailable, etc."invalid/request").execute
# raises SearchFlip::ResponseError


=> #<:response ...>


  • merge

You can merge criterias, ie. combine the attributes (constraints, settings, etc) of two individual criterias:

CommentIndex.where(approved: true).merge("hello"))
# equivalent to: CommentIndex.where(approved: true).search("hello")
  • timeout

Specify a timeout to limit query processing time:

  • terminate_after

Activate early query termination to stop query processing after the specified number of records has been found:


For further details and a full list of methods, check out the reference docs.

  • custom

You can add a custom clause to the request via

CommentIndex.custom(custom_clause: '...')

This can be useful for Elasticsearch features not yet supported via criteria methods by SearchFlip, custom plugin clauses, etc.

Custom Criteria Methods

To add custom criteria methods, you can add class methods to your index class.

class HotelIndex
  # ...

def self.where_geo(lat:, lon:, distance:) filter(geo_distance: { distance: distance, location: { lat: lat, lon: lon } }) end end"bed and breakfast").where_geo(lat: 53.57532, lon: 10.01534, distance: '50km').aggregate(:rating)

Using multiple Elasticsearch clusters

To use multiple Elasticsearch clusters, specify a connection within your indices:

MyConnection = "")

class MyIndex include SearchFlip::Index

def self.connection MyConnection end end

This allows to use different clusters per index e.g. when migrating indices to new versions of Elasticsearch.

You can specify basic auth, additional headers, etc via:

http_client =

Basic Auth

http_client = http_client.basic_auth(user: "username", pass: "password")

Raw Auth Header

http_client = http_client.auth("Bearer VGhlIEhUVFAgR2VtLCBST0NLUw")

Proxy Settings

http_client = http_client.via("", 8080)

Custom headers

http_client = http_client.headers(key: "value") "...", http_client: http_client)

AWS Elasticsearch / Signed Requests

To use SearchFlip with AWS Elasticsearch and signed requests, you have to add

to your Gemfile and tell SearchFlip to use the
require "search_flip/aws_sigv4_plugin"

MyConnection = base_url: "", http_client: plugins: [ region: "...", access_key_id: "...", secret_access_key: "..." ) ] ) )

Again, in your index you need to specify this connection:

class MyIndex
  include SearchFlip::Index

def self.connection MyConnection end end

Routing and other index-time options


in case you want to use routing or pass other index-time options:
class CommentIndex
  include SearchFlip::Index

def self.index_options(comment) { routing: comment.user_id, version: comment.version, version_type: "external_gte" } end end

These options will be passed whenever records get indexed, deleted, etc.


SearchFlip supports instrumentation for request execution via

compatible instrumenters to e.g. allow global performance tracing, etc.

To use instrumentation, configure the instrumenter:

SearchFlip::Config[:instrumenter] = ActiveSupport::Notifications.notifier

Subsequently, you can subscribe to notifcations for

ActiveSupport::Notifications.subscribe("request.search_flip") do |name, start, finish, id, payload|
  payload[:index] # the index class
  payload[:request] # the request hash sent to Elasticsearch
  payload[:response] # the SearchFlip::Response object or nil in case of errors

A notification will be send for every request that is sent to Elasticsearch.

Non-ActiveRecord models

SearchFlip ships with built-in support for ActiveRecord models, but using non-ActiveRecord models is very easy. The model must implement a

class method and the Index class needs to implement
. The default implementations for the index class are as follows:
class MyIndex
  include SearchFlip::Index

def self.record_id(object) end

def self.fetch_records(ids) model.where(id: ids) end end

Thus, if your ORM supports

you are already good to go. Otherwise, simply add your custom implementation of those methods that work with whatever ORM you use.

Date and Timestamps in JSON

Elasticsearch requires dates and timestamps to have one of the formats listed here:


in ruby by default outputs something like:
# => "{\"time\":\"2018-02-22 18:19:33 UTC\"}"

This format is not compatible with Elasticsearch by default. If you're on Rails, ActiveSupport adds its own

methods to
, etc. However, ActiveSupport checks whether they are used in combination with
or not and adapt:
=> "\"2018-02-22T18:18:22.088Z\""

JSON.generate(time: => "{"time":"2018-02-22 18:18:59 UTC"}"

SearchFlip is using the Oj gem to generate JSON. More concretely, SearchFlip is using:

Oj.dump({ key: "value" }, mode: :custom, use_to_json: true)

This mitigates the issues if you're on Rails:

Oj.dump(, mode: :custom, use_to_json: true)
# => "\"2018-02-22T18:21:21.064Z\""

However, if you're not on Rails, you need to add

methods to
to get proper serialization. You can either add them on your own, via other libraries or by simply using:
require "search_flip/to_json"

Feature Support

  • for Elasticsearch 2.x, the delete-by-query plugin is required to delete records via queries
  • #track_total_hits
    is only available with Elasticsearch >= 7

Keeping your Models and Indices in Sync

Besides the most basic approach to get you started, SearchFlip currently doesn't ship with any means to automatically keep your models and indices in sync, because every method is very much bound to the concrete environment and depends on your concrete requirements. In addition, the methods to achieve model/index consistency can get arbitrarily complex and we want to keep this bloat out of the SearchFlip codebase.

class Comment < ActiveRecord::Base
  include SearchFlip::Model

notifies_index(CommentIndex) end

It uses

(if applicable,
otherwise) hooks to synchronously update the index when your model changes.

Semantic Versioning

SearchFlip is using Semantic Versioning: SemVer



  1. Fork it
  2. Create your feature branch (
    git checkout -b my-new-feature
  3. Commit your changes (
    git commit -am 'Add some feature'
  4. Push to the branch (
    git push origin my-new-feature
  5. Create new Pull Request

Running the test suite

Running the tests is super easy. The test suite uses sqlite, such that you only need to install Elasticsearch. You can install Elasticsearch on your own, or you can e.g. use docker-compose:

$ cd search_flip
$ sudo ES_IMAGE=elasticsearch:5.4 docker-compose up
$ rspec

That's it.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.