search_flip

by mrkamel

mrkamel / search_flip

Full-Featured ElasticSearch Ruby Client with a Chainable DSL

254 Stars 5 Forks Last release: 29 days ago (3.1.2) MIT License 438 Commits 12 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

search_flip

Full-Featured Elasticsearch Ruby Client with a Chainable DSL

Build Status Gem Version

Using SearchFlip it is dead-simple to create index classes that correspond to Elasticsearch indices and to manipulate, query and aggregate these indices using a chainable, concise, yet powerful DSL. Finally, SearchFlip supports Elasticsearch 2.x, 5.x, 6.x, 7.x. Check section Feature Support for version dependent features.

CommentIndex.search("hello world", default_field: "title").where(visible: true).aggregate(:user_id).sort(id: "desc")

CommentIndex.aggregate(:user_id) do |aggregation| aggregation.aggregate(histogram: { date_histogram: { field: "created_at", interval: "month" }}) end

CommentIndex.range(:created_at, gt: Date.today - 1.week, lt: Date.today).where(state: ["approved", "pending"])

Updating from previous SearchFlip versions

Checkout UPDATING.md for detailed instructions.

Comparison with other gems

There are great ruby gems to work with Elasticsearch like e.g. searchkick and elasticsearch-ruby already. However, they don't have a chainable API. Compare yourself.

# elasticsearch-ruby
Comment.search(
  query: {
    query_string: {
      query: "hello world",
      default_operator: "AND"
    }
  }
)

searchkick

Comment.search("hello world", where: { available: true }, order: { id: "desc" }, aggs: [:username])

search_flip

CommentIndex.search("hello world").where(available: true).sort(id: "desc").aggregate(:username)

Finally, SearchFlip comes with a minimal set of dependencies (http-rb, hashie and oj only).

Reference Docs

SearchFlip has a great documentation. Check youself at http://www.rubydoc.info/github/mrkamel/search_flip

Install

Add this line to your application's Gemfile:

gem 'search_flip'

and then execute

$ bundle

or install it via

$ gem install search_flip

Config

You can change global config options like:

SearchFlip::Config[:environment] = "development"
SearchFlip::Config[:base_url] = "http://127.0.0.1:9200"

Available config options are:

  • index_prefix
    to have a prefix added to your index names automatically. This can be useful to separate the indices of e.g. testing and development environments.
  • base_url
    to tell SearchFlip how to connect to your cluster
  • bulk_limit
    a global limit for bulk requests
  • bulk_max_mb
    a global limit for the payload of bulk requests
  • auto_refresh
    tells SearchFlip to automatically refresh an index after import, index, delete, etc operations. This is e.g. useful for testing, etc. Defaults to false.

Usage

First, create a separate class for your index and include

SearchFlip::Index
.
class CommentIndex
  include SearchFlip::Index
end

Then tell the Index about the index name, the corresponding model and how to serialize the model for indexing.

class CommentIndex
  include SearchFlip::Index

def self.index_name "comments" end

def self.model Comment end

def self.serialize(comment) { id: comment.id, username: comment.username, title: comment.title, message: comment.message } end end

Optionally, you can specify a custom

type_name
, but note that starting with Elasticsearch 7, types are deprecated.
class CommentIndex
  # ...

def self.type_name "comment" end end

You can additionally specify an

index_scope
which will automatically be applied to scopes, eg. ActiveRecord::Relation objects, passed to
#import
,
#index
, etc. This can be used to preload associations that are used when serializing records or to restrict the records you want to index.
class CommentIndex
  # ...

def self.index_scope(scope) scope.preload(:user) end end

CommentIndex.import(Comment.all) # => CommentIndex.import(Comment.all.preload(:user))

To specify a custom mapping:

class CommentIndex
  # ...

def self.mapping { properties: { # ... } } end

...

end

Please note that you need to specify the mapping without a type name, even for Elasticsearch versions before 7, as SearchFlip will add the type name automatically if neccessary.

To specify index settings:

def self.index_settings
  {
    settings: {
      number_of_shards: 10,
      number_of_replicas: 2
    }
  }
end

Then you can interact with the index:

CommentIndex.create_index
CommentIndex.index_exists?
CommentIndex.delete_index
CommentIndex.update_mapping
CommentIndex.close_index
CommentIndex.open_index

Index records (automatically uses the Bulk API):

CommentIndex.import(Comment.all)
CommentIndex.import(Comment.first)
CommentIndex.import([Comment.find(1), Comment.find(2)])
CommentIndex.import(Comment.where("created_at > ?", Time.now - 7.days))

Query records:

CommentIndex.total_entries
# => 2838

CommentIndex.search("title:hello").records

=> [#<comment ...>, #<comment ...>, ...]

CommentIndex.where(username: "mrkamel").total_entries

=> 13

CommentIndex.aggregate(:username).aggregations(:username)

=> {1=>#<:result doc_count="37" ...>, 2=>... }

... </:result>

Please note that you can check the request that will be send to Elasticsearch by calling

#request
on the query:
CommentIndex.search("hello world").sort(id: "desc").aggregate(:username).request
# => {:query=>{:bool=>{:must=>[{:query_string=>{:query=>"hello world", :default_operator=>:AND}}]}}, ...}

Delete records:

# for Elasticsearch >= 2.x and < 5.x, the delete-by-query plugin is required
# for the following query:

CommentIndex.match_all.delete

or delete manually via the bulk API:

CommentIndex.bulk do |indexer| CommentIndex.match_all.find_each do |record| indexer.delete record.id end end

When indexing or deleting documents, you can pass options to control the bulk indexing and you can use all options provided by the Bulk API:

CommentIndex.import(Comment.first, { bulk_limit: 1_000 }, op_type: "create", routing: "routing_key")

or directly

CommentIndex.create(Comment.first, { bulk_max_mb: 100 }, routing: "routing_key") CommentIndex.update(Comment.first, ...)

Checkout the Elasticsearch Bulk API docs for more info as well as SearchFlip::Bulk for a complete list of available options to control the bulk indexing of SearchFlip.

Working with Elasticsearch Aliases

You can use and manage Elasticsearch Aliases like the following:

class UserIndex
  include SearchFlip::Index

def self.index_name alias_name end

def self.alias_name "users" end end

Then, create an index, import the records and add the alias like:

new_user_index = UserIndex.with_settings(index_name: "users-#{SecureRandom.hex}")
new_user_index.create_index
new_user_index.import User.all
new_user.connection.update_aliases(actions: [
  add: { index: new_user_index.index_name, alias: new_user_index.alias_name }
])

If the alias already exists, you have to remove it as well first within

update_aliases
.

Please note:

with_settings(index_name: '...')
returns an anonymous (i.e. temporary) class which inherits from UserIndex and overwrites
index_name
.

Chainable Methods

SearchFlip supports even more advanced usages, like e.g. post filters, filtered aggregations or nested aggregations via simple to use API methods.

Query/Filter Criteria Methods

SearchFlip provides powerful methods to query/filter Elasticsearch:

  • where

The

.where
method feels like ActiveRecord's
where
and adds a bool filter clause to the request:
CommentIndex.where(reviewed: true)
CommentIndex.where(likes: 0 .. 10_000)
CommentIndex.where(state: ["approved", "rejected"])
  • where_not

The

.where_not
method is like
.where
, but excluding the matching documents:
CommentIndex.where_not(id: [1, 2, 3])
  • range

Use

.range
to add a range filter query:
CommentIndex.range(:created_at, gt: Date.today - 1.week, lt: Date.today)
  • filter

Use

.filter
to add raw filter queries:
CommentIndex.filter(term: { state: "approved" })
  • should

Use

.should
to add raw should queries:
CommentIndex.should([
  { term: { state: "approved" } },
  { term: { user: "mrkamel" } },
])
  • must

Use

.must
to add raw must queries:
CommentIndex.must(term: { state: "approved" })
  • must_not

Like

must
, but excluding the matching documents:
CommentIndex.must_not(term: { state: "approved" })
  • search

Adds a query string query, with AND as default operator:

CommentIndex.search("hello world")
CommentIndex.search("state:approved")
CommentIndex.search("username:a*")
CommentIndex.search("state:approved OR state:rejected")
CommentIndex.search("hello world", default_operator: "OR")
  • exists

Use

exists
to add an
exists
query:
CommentIndex.exists(:state)
  • exists_not

Like

exists
, but excluding the matching documents:
CommentIndex.exists_not(:state)
  • match_all

Simply matches all documents:

CommentIndex.match_all
  • all

Simply returns the criteria as is or an empty criteria when called on the index class directly. Useful for chaining.

CommentIndex.all
  • to_query

Sometimes, you want to convert the constraints of a search flip query to a raw query to e.g. use it in a should clause:

CommentIndex.should([
  CommentIndex.range(:likes_count, gt: 10).to_query,
  CommentIndex.search("search term").to_query
])

It returns all added queries and filters, including post filters as a raw query:

CommentIndex.where(state: "new").search("text").to_query
# => {:bool=>{:filter=>[{:term=>{:state=>"new"}}], :must=>[{:query_string=>{:query=>"text", ...}}]}}

Post Query/Filter Criteria Methods

All query/filter criteria methods (

#where
,
#where_not
,
#range
, etc.) are available in post filter mode as well, ie. filters/queries applied after aggregations are calculated. Checkout the Elasticsearch docs for further info.
query = CommentIndex.aggregate(:user_id)
query = query.post_where(reviewed: true)
query = query.post_search("username:a*")

Checkout PostFilterable for a complete API reference.

Aggregations

SearchFlip allows to elegantly specify nested aggregations, no matter how deeply nested:

query = OrderIndex.aggregate(:username, order: { revenue: "desc" }) do |aggregation|
  aggregation.aggregate(revenue: { sum: { field: "price" }})
end

Generally, aggregation results returned by Elasticsearch are returned as a

SearchFlip::Result
, which basically is a
Hashie::Mash
, such that you can access them via:
query.aggregations(:username)["mrkamel"].revenue.value

Still, if you want to get the raw aggregations returned by Elasticsearch, access them without supplying any aggregation name to

#aggregations
:
query.aggregations # => returns the raw aggregation section

query.aggregations["username"]["buckets"].detect { |bucket| bucket["key"] == "mrkamel" }["revenue"]["value"] # => 238.50

Once again, the criteria methods (

#where
,
#range
, etc.) are available in aggregations as well:
query = OrderIndex.aggregate(average_price: {}) do |aggregation|
  aggregation = aggregation.match_all
  aggregation = aggregation.where(user_id: current_user.id) if current_user

aggregation.aggregate(average_price: { avg: { field: "price" }}) end

query.aggregations(:average_price).average_price.value

Even various criteria for top hits aggregations can be specified elegantly:

query = ProductIndex.aggregate(sponsored: { top_hits: {} }) do |aggregation|
  aggregation.sort(:rank).highlight(:title).source([:id, :title])
end

Checkout Aggregatable as well as Aggregation for a complete API reference.

Suggestions

query = CommentIndex.suggest(:suggestion, text: "helo", term: { field: "message" })
query.suggestions(:suggestion).first["text"] # => "hello"

Highlighting

CommentIndex.highlight([:title, :message])
CommentIndex.highlight(:title).highlight(:description)
CommentIndex.highlight(:title, require_field_match: false)
CommentIndex.highlight(title: { type: "fvh" })
query = CommentIndex.highlight(:title).search("hello")
query.results[0]._hit.highlight.title # => "hello world"

Other Criteria Methods

There are even more chainable criteria methods to make your life easier. For a full list, checkout the reference docs.

  • source

In case you want to restrict the returned fields, simply specify the fields via

#source
:
CommentIndex.source([:id, :message]).search("hello world")
  • paginate
    ,
    page
    ,
    per

SearchFlip supports will_paginate and kaminari compatible pagination. Thus, you can either use

#paginate
or
#page
in combination with
#per
:
CommentIndex.paginate(page: 3, per_page: 50)
CommentIndex.page(3).per(50)
  • profile

Use

#profile
to enable query profiling:
query = CommentIndex.profile(true)
query.raw_response["profile"] # => { "shards" => ... }
  • preload
    ,
    eager_load
    and
    includes

Uses the well known methods from ActiveRecord to load associated database records when fetching the respective records themselves. Works with other ORMs as well, if supported.

Using

#preload
:
CommentIndex.preload(:user, :post).records
PostIndex.includes(comments: :user).records

or

#eager_load
CommentIndex.eager_load(:user, :post).records
PostIndex.eager_load(comments: :user).records

or

#includes
CommentIndex.includes(:user, :post).records
PostIndex.includes(comments: :user).records
  • find_in_batches

Used to fetch and yield records in batches using the ElasicSearch scroll API. The batch size and scroll API timeout can be specified.

CommentIndex.search("hello world").find_in_batches(batch_size: 100) do |batch|
  # ...
end
  • find_results_in_batches

Used like

find_in_batches
, but yielding the raw results (as
SearchFlip::Result
objects) instead of database records.
CommentIndex.search("hello world").find_results_in_batches(batch_size: 100) do |batch|
  # ...
end
  • find_each

Like

#find_in_batches
but yielding one record at a time.
CommentIndex.search("hello world").find_each(batch_size: 100) do |record|
  # ...
end
  • find_each_result

Like

#find_results_in_batches
, but yielding one record at a time.
CommentIndex.search("hello world").find_each_result(batch_size: 100) do |batch|
  # ...
end
  • scroll

You can as well use the underlying scroll API directly, ie. without using higher level scrolling:

query = CommentIndex.scroll(timeout: "5m")

until query.records.empty?

...

query = query.scroll(id: query.scroll_id, timeout: "5m") end

  • failsafe

Use

#failsafe
to prevent any exceptions from being raised for query string syntax errors or Elasticsearch being unavailable, etc.
CommentIndex.search("invalid/request").execute
# raises SearchFlip::ResponseError

...

CommentIndex.search("invalid/request").failsafe(true).execute

=> #<:response ...>

</:response>

  • merge

You can merge criterias, ie. combine the attributes (constraints, settings, etc) of two individual criterias:

CommentIndex.where(approved: true).merge(CommentIndex.search("hello"))
# equivalent to: CommentIndex.where(approved: true).search("hello")
  • timeout

Specify a timeout to limit query processing time:

CommentIndex.timeout("3s").execute
  • terminate_after

Activate early query termination to stop query processing after the specified number of records has been found:

CommentIndex.terminate_after(10).execute

For further details and a full list of methods, check out the reference docs.

  • custom

You can add a custom clause to the request via

custom
CommentIndex.custom(custom_clause: '...')

This can be useful for Elasticsearch features not yet supported via criteria methods by SearchFlip, custom plugin clauses, etc.

Custom Criteria Methods

To add custom criteria methods, you can add class methods to your index class.

class HotelIndex
  # ...

def self.where_geo(lat:, lon:, distance:) filter(geo_distance: { distance: distance, location: { lat: lat, lon: lon } }) end end

HotelIndex.search("bed and breakfast").where_geo(lat: 53.57532, lon: 10.01534, distance: '50km').aggregate(:rating)

Using multiple Elasticsearch clusters

To use multiple Elasticsearch clusters, specify a connection within your indices:

MyConnection = SearchFlip::Connection.new(base_url: "http://elasticsearch.host:9200")

class MyIndex include SearchFlip::Index

def self.connection MyConnection end end

This allows to use different clusters per index e.g. when migrating indices to new versions of Elasticsearch.

You can specify basic auth, additional headers, etc via:

http_client = SearchFlip::HTTPClient.new

Basic Auth

http_client = http_client.basic_auth(user: "username", pass: "password")

Raw Auth Header

http_client = http_client.auth("Bearer VGhlIEhUVFAgR2VtLCBST0NLUw")

Proxy Settings

http_client = http_client.via("proxy.host", 8080)

Custom headers

http_client = http_client.headers(key: "value")

SearchFlip::Connection.new(base_url: "...", http_client: http_client)

AWS Elasticsearch / Signed Requests

To use SearchFlip with AWS Elasticsearch and signed requests, you have to add

aws-sdk-core
to your Gemfile and tell SearchFlip to use the
SearchFlip::AwsSigv4Plugin
:
require "search_flip/aws_sigv4_plugin"

MyConnection = SearchFlip::Connection.new( base_url: "https://your-elasticsearch-cluster.es.amazonaws.com", http_client: SearchFlip::HTTPClient.new( plugins: [ SearchFlip::AwsSigv4Plugin.new( region: "...", access_key_id: "...", secret_access_key: "..." ) ] ) )

Again, in your index you need to specify this connection:

class MyIndex
  include SearchFlip::Index

def self.connection MyConnection end end

Routing and other index-time options

Override

index_options
in case you want to use routing or pass other index-time options:
class CommentIndex
  include SearchFlip::Index

def self.index_options(comment) { routing: comment.user_id, version: comment.version, version_type: "external_gte" } end end

These options will be passed whenever records get indexed, deleted, etc.

Instrumentation

SearchFlip supports instrumentation for request execution via

ActiveSupport::Notifications
compatible instrumenters to e.g. allow global performance tracing, etc.

To use instrumentation, configure the instrumenter:

SearchFlip::Config[:instrumenter] = ActiveSupport::Notifications.notifier

Subsequently, you can subscribe to notifcations for

request.search_flip
:
ActiveSupport::Notifications.subscribe("request.search_flip") do |name, start, finish, id, payload|
  payload[:index] # the index class
  payload[:request] # the request hash sent to Elasticsearch
  payload[:response] # the SearchFlip::Response object or nil in case of errors
end

A notification will be send for every request that is sent to Elasticsearch.

Non-ActiveRecord models

SearchFlip ships with built-in support for ActiveRecord models, but using non-ActiveRecord models is very easy. The model must implement a

find_each
class method and the Index class needs to implement
Index.record_id
and
Index.fetch_records
. The default implementations for the index class are as follows:
class MyIndex
  include SearchFlip::Index

def self.record_id(object) object.id end

def self.fetch_records(ids) model.where(id: ids) end end

Thus, if your ORM supports

.find_each
,
#id
and
#where
you are already good to go. Otherwise, simply add your custom implementation of those methods that work with whatever ORM you use.

Date and Timestamps in JSON

Elasticsearch requires dates and timestamps to have one of the formats listed here: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html#strict-date-time.

However,

JSON.generate
in ruby by default outputs something like:
JSON.generate(time: Time.now.utc)
# => "{\"time\":\"2018-02-22 18:19:33 UTC\"}"

This format is not compatible with Elasticsearch by default. If you're on Rails, ActiveSupport adds its own

#to_json
methods to
Time
,
Date
, etc. However, ActiveSupport checks whether they are used in combination with
JSON.generate
or not and adapt:
Time.now.utc.to_json
=> "\"2018-02-22T18:18:22.088Z\""

JSON.generate(time: Time.now.utc) => "{"time":"2018-02-22 18:18:59 UTC"}"

SearchFlip is using the Oj gem to generate JSON. More concretely, SearchFlip is using:

Oj.dump({ key: "value" }, mode: :custom, use_to_json: true)

This mitigates the issues if you're on Rails:

Oj.dump(Time.now, mode: :custom, use_to_json: true)
# => "\"2018-02-22T18:21:21.064Z\""

However, if you're not on Rails, you need to add

#to_json
methods to
Time
,
Date
and
DateTime
to get proper serialization. You can either add them on your own, via other libraries or by simply using:
require "search_flip/to_json"

Feature Support

  • for Elasticsearch 2.x, the delete-by-query plugin is required to delete records via queries
  • #track_total_hits
    is only available with Elasticsearch >= 7

Keeping your Models and Indices in Sync

Besides the most basic approach to get you started, SearchFlip currently doesn't ship with any means to automatically keep your models and indices in sync, because every method is very much bound to the concrete environment and depends on your concrete requirements. In addition, the methods to achieve model/index consistency can get arbitrarily complex and we want to keep this bloat out of the SearchFlip codebase.

class Comment < ActiveRecord::Base
  include SearchFlip::Model

notifies_index(CommentIndex) end

It uses

after_commit
(if applicable,
after_save
,
after_destroy
and
after_touch
otherwise) hooks to synchronously update the index when your model changes.

Semantic Versioning

SearchFlip is using Semantic Versioning: SemVer

Links

Contributing

  1. Fork it
  2. Create your feature branch (
    git checkout -b my-new-feature
    )
  3. Commit your changes (
    git commit -am 'Add some feature'
    )
  4. Push to the branch (
    git push origin my-new-feature
    )
  5. Create new Pull Request

Running the test suite

Running the tests is super easy. The test suite uses sqlite, such that you only need to install Elasticsearch. You can install Elasticsearch on your own, or you can e.g. use docker-compose:

$ cd search_flip
$ sudo ES_IMAGE=elasticsearch:5.4 docker-compose up
$ rspec

That's it.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.