:mag: Learn how to use ElasticSearch to power a great search experience for your project/product/website.
In the next 30 mins you will learn how to use ElasticSearch to power a great search experience for your project/product/website.
For anything more than a basic website, people (visiting/using your site/app) expect to be able to search through your content (blog posts, recipes, products, reviews, etc.)
You could use google custom search to provide this functionality and side-step having to run your own (cluster of) search server(s)... But I suspect your project/customer wants/needs more control over the search experience and that's why you're reading this intro?
Simple/Short answer: Pick the Best tool for the job.
In the past we've used MongoDB's full-text-search (and even wrote a tutorial for it!), MySQL full-text-search to reasonable success (Deal Searcher V.1 @Groupon) and many of our Rails friends swear by Postgres full-text-search but none of these databases were designed from scratch to provide scalable full-text search. So, if you want search, Elasticsearch!
Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. i.e. awesomeness in a box!
Read more: http://www.elasticsearch.org/overview/elasticsearch/
Feeling bewildered by that buzzword fest? let's break it down:
Real-Time: a system in which input data is processed within milliseconds so that it is available virtually immediately as feedback to the process from which it is coming - i.e. things happen without a noticeable delay. An example of "real time" is instant messaging.
see: https://en.wikipedia.org/wiki/Real-time_computing
"Near" Real-Time: means there is a small (but noticeable) delay. You can insert/update a record in the "index" and it will be searchable in less than a second. (It is not immediate, but its close, so they say "Near" Real Time) And example of "near real time" is email (not quite instant)
Full-Text Search: means when you search through the records in an ElasticSearch
database (cluster) your search term(s) will be searched for everywhere in the desired field(s) of the document.
For example: Imagine you have a blog and each blog post has: Title, Intro, Body and Comments section.
When searching for a particular string e.g: "this is awesomeness", you could search in all-the-fields
which could return a result in one of the comments.
read more: https://en.wikipedia.org/wiki/Fulltextsearch
Distributed means you can have several ElasticSearch nodes in different data centers or regions
to improve retrieval reliability.
see: https://en.wikipedia.org/?title=Distributed_computing
Having a REST API means you can access your ElasticSearch cluster using standard HTTP requests. ˜
There are a few options for running ElasticSearch:
A. Boot a Virtual Machine with ES and all its dependencies (using Vagrant)
B. Install the "binary" package for your Operating System.
C. Don't install anything and just use a free heroku instance! (See: Heroku section below)
ElasticSearch requires Java 8, so if you want to install ElasticSearch ("natively") on your local machine you will need to have Java running... We prefer not to have Java running on our personal machines (because its chronically insecure) so we created a Vagrant box to consistently boot ES (using a VM!) ... see below.
If you aren't using Vagrant, read our Vagrant tutorial now: https://github.com/docdis/learn-vagrant
If you are already using Vagrant, simply clone this repo:
git clone [email protected]:docdis/learn-elasticsearch.git && cd learn-elasticsearch
Then run this command (in your terminal):
vagrant up
Note: expect the installation to take a few minutes, go for a walk, or skip to the Tutorial section below and start watching the video.
If you don't mind having Java running on your Mac, you can use Homebrew to install ES:
brew install elasticsearch
To have launchd start elasticsearch at login:
ln -sfv /usr/local/opt/elasticsearch/*.plist ~/Library/LaunchAgentsThen to load elasticsearch now:
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.elasticsearch.plistOr, if you don't want/need launchctl, you can just run:
elasticsearch --config=/usr/local/opt/elasticsearch/config/elasticsearch.yml
see: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-service-win.html
(but, seriously, try Vagrant!)
To confirm that everything is working as expected, open your terminal and run the following command:
curl -XGET http://localhost:9200
You should expect to see something similar to:
Once you have installed ElasticSearch (following the instructions above)
Visit: https://www.elastic.co/webinars/getting-started-with-elasticsearch (register using fake data if you want to avoid email spam) and watch the video.
curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{"user":"kimchy","post_date":"2009-11-15T14:12:12","message" : "trying out Elasticsearch"}'
If you want to following along with the ElasticSearch getting started video:
Insert a record:
sh curl -XPUT 'http://localhost:9200/vehicles/tv/one' -d '{"color":"green","driver":{"born":"1959-09-07","name":"Walter White"},"make":"Pontiac","model":"Aztek","value_usd":5000.0, "year":2003}'
Check the mapping for the index:
sh curl http://localhost:9200/vehicles/_mapping?pretty
To delete an index you accidentally created:
sh curl -XDELETE 'http://localhost:9200/vehicles/'Search:
js curl 'localhost:9200/vehicles/tv/_search?q=_id:one&pretty'Insert another document/record:
sh curl -XPUT 'http://localhost:9200/vehicles/tv/two' -d '{"color":"black","driver":{"born":"1949-01-09","name":"Michael Knight"},"make":"Pontiac","model":"Trans Am","value_usd":9999999.00, "year":1982}'
curl 'http://localhost:9200/vehicles/_search?q=pontiac&pretty'
The Update API is quite well documented: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html
see:
/nodejsfolder for sample scripts you can run in node.js
This section is about using ElasticSearch within the
Elixirprogramming language. If you are new to
Elixir, see: github.com/dwyl/learn-elixir (you're in for a treat!)
Once you know a bit about Elixir, writing to an ElasticSearch cluster is quite straight forward thanks to @Zatvobor's module
tirexssee: https://github.com/Zatvobor/tirexs#getting-started
We've included a simple Write/Read example in
/elixir/lib/elastic.exand
/elixir/lib/elastic_test.ex
To try it out on your local computer, simply run the following command(s):
git clone [email protected]:dwyl/learn-elasticsearch.git cd learn-elasticsearch mix deps.get mix test
Tip: you can copy paste the whole block and run all the commands in order.
ELK is a Logging Stack comprised of ElasticSearch, LogStash & Kibana
I chose elasticsearch to power the search for a project I lead at News after careful consideration of Solr. There are great heroku addons (we used Bonsai because they have a free dev tier) and the quality of the search results is superb.
see ERRORS.md
need to research this
There are over a hundred modules for ElasticSearch on NPM
see: http://node-modules.com/search?q=elasticsearch
While writing this post we tried the following modules:
We got frustrated using the other modules, so we wrote a better one: https://github.com/dwyl/esta
http://www.elasticsearch.org/guide/en/elasticsearch/client/community/current/front-ends.html
Where your cluster looses communication and you end up with two masters.
If you prefer not to administer your own database/cluster there are a few services you can use: