R client for the Elasticsearch HTTP API
A general purpose R interface to Elasticsearch
This client is developed following the latest stable releases, currently
v7.10.0. It is generally compatible with older versions of Elasticsearch. Unlike the Python client, we try to keep as much compatibility as possible within a single version of this client, as that's an easier setup in R world.
You're fine running ES locally on your machine, but be careful just throwing up ES on a server with a public IP address - make sure to think about security.
Stable version from CRAN
install.packages("elastic")
Development version from GitHub
remotes::install_github("ropensci/elastic")
library('elastic')
w/ Docker
Pull the official elasticsearch image
# elasticsearch needs to have a version tag. We're pulling 7.10.1 here docker pull elasticsearch:7.10.1
Then start up a container
docker run -d -p 9200:9200 elasticsearch:7.10.1
Then elasticsearch should be available on port 9200, try
curl localhost:9200and you should get the familiar message indicating ES is on.
If you're using boot2docker, you'll need to use the IP address in place of localhost. Get it by doing
boot2docker ip.
on OSX
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.0-darwin-x86_64.tar.gz
tar -zxvf elasticsearch-7.10.0-darwin-x86_64.tar.gz
sudo mv elasticsearch-7.10.0 /usr/local
cd /usr/local
elasticsearchdirectory:
rm -rf elasticsearch
sudo ln -s elasticsearch-7.10.0 elasticsearch(replace version with your version)
You can also install via Homebrew:
brew install elasticsearch
Note: for the 1.6 and greater upgrades of Elasticsearch, they want you to have java 8 or greater. I downloaded Java 8 from here http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html and it seemed to work great.
I am not totally clear on best practice here, but from what I understand, when you upgrade to a new version of Elasticsearch, place old
elasticsearch/dataand
elasticsearch/configdirectories into the new installation (
elasticsearch/dir). The new elasticsearch instance with replaced data and config directories should automatically update data to the new version and start working. Maybe if you use homebrew on a Mac to upgrade it takes care of this for you - not sure.
Obviously, upgrading Elasticsearch while keeping it running is a different thing (some help here from Elastic).
cd /usr/local/elasticsearch
bin/elasticsearch
I create a little bash shortcut called
esthat does both of the above commands in one step (
cd /usr/local/elasticsearch && bin/elasticsearch).
The function
connect()is used before doing anything else to set the connection details to your remote or local elasticsearch store. The details created by
connect()are written to your options for the current session, and are used by
elasticfunctions.
xIf you're following along here with a local instance of Elasticsearch, you'll use
xbelow to do more stuff.For AWS hosted elasticsearch, make sure to specify path = "" and the correct port - transport schema pair.
connect(host = , path = "", port = 80, transport_schema = "http") # or connect(host = , path = "", port = 443, transport_schema = "https")If you are using Elastic Cloud or an installation with authentication (X-pack), make sure to specify path = "", user = "", pwd = "" and the correct port - transport schema pair.
connect(host = , path = "", user="test", pwd = "1234", port = 9243, transport_schema = "https")
Get some data
Elasticsearch has a bulk load API to load data in fast. The format is pretty weird though. It's sort of JSON, but would pass no JSON linter. I include a few data sets in
elasticso it's easy to get up and running, and so when you run examples in this package they'll actually run the same way (hopefully).I have prepare a non-exported function useful for preparing the weird format that Elasticsearch wants for bulk data loads, that is somewhat specific to PLOS data (See below), but you could modify for your purposes. See
make_bulk_plos()andmake_bulk_gbif()here.Shakespeare data
Elasticsearch provides some data on Shakespeare plays. I've provided a subset of this data in this package. Get the path for the file specific to your machine:
shakespeareThen load the data into Elasticsearch:
make sure to create your connection object with
connect()# xIf you need some big data to play with, the shakespeare dataset is a good one to start with. You can get the whole thing and pop it into Elasticsearch (beware, may take up to 10 minutes or so.):
curl -XGET https://download.elastic.co/demos/kibana/gettingstarted/shakespeare_6.0.json > shakespeare.json curl -XPUT localhost:9200/_bulk --data-binary @shakespeare.jsonPublic Library of Science (PLOS) data
A dataset inluded in the
elasticpackage is metadata for PLOS scholarly articles. Get the file path, then load:if (index_exists(x, "plos")) index_delete(x, "plos") plosdatGlobal Biodiversity Information Facility (GBIF) data
A dataset inluded in the
elasticpackage is data for GBIF species occurrence records. Get the file path, then load:if (index_exists(x, "gbif")) index_delete(x, "gbif") gbifdatGBIF geo data with a coordinates element to allow
geo_shapequeriesif (index_exists(x, "gbifgeo")) index_delete(x, "gbifgeo") gbifgeoMore data sets
There are more datasets formatted for bulk loading in the
sckott/elastic_dataGitHub repository. Find it at https://github.com/sckott/elastic_data
Search
Search the
plosindex and only return 1 resultSearch(x, index = "plos", size = 1)$hits$hits #> [[1]] #> [[1]]$`_index` #> [1] "plos" #> #> [[1]]$`_type` #> [1] "_doc" #> #> [[1]]$`_id` #> [1] "0" #> #> [[1]]$`_score` #> [1] 1 #> #> [[1]]$`_source` #> [[1]]$`_source`$id #> [1] "10.1371/journal.pone.0007737" #> #> [[1]]$`_source`$title #> [1] "Phospholipase C-\u03b24 Is Essential for the Progression of the Normal Sleep Sequence and Ultradian Body Temperature Rhythms in Mice"Search the
plosindex, and query for antibody, limit to 1 resultSearch(x, index = "plos", q = "antibody", size = 1)$hits$hits #> [[1]] #> [[1]]$`_index` #> [1] "plos" #> #> [[1]]$`_type` #> [1] "_doc" #> #> [[1]]$`_id` #> [1] "813" #> #> [[1]]$`_score` #> [1] 5.18676 #> #> [[1]]$`_source` #> [[1]]$`_source`$id #> [1] "10.1371/journal.pone.0107638" #> #> [[1]]$`_source`$title #> [1] "Sortase A Induces Th17-Mediated and Antibody-Independent Immunity to Heterologous Serotypes of Group A Streptococci"Get documents
Get document with id=4
docs_get(x, index = 'plos', id = 4) #> $`_index` #> [1] "plos" #> #> $`_type` #> [1] "_doc" #> #> $`_id` #> [1] "4" #> #> $`_version` #> [1] 1 #> #> $`_seq_no` #> [1] 4 #> #> $`_primary_term` #> [1] 1 #> #> $found #> [1] TRUE #> #> $`_source` #> $`_source`$id #> [1] "10.1371/journal.pone.0107758" #> #> $`_source`$title #> [1] "Lactobacilli Inactivate Chlamydia trachomatis through Lactic Acid but Not H2O2"Get certain fields
docs_get(x, index = 'plos', id = 4, fields = 'id') #> $`_index` #> [1] "plos" #> #> $`_type` #> [1] "_doc" #> #> $`_id` #> [1] "4" #> #> $`_version` #> [1] 1 #> #> $`_seq_no` #> [1] 4 #> #> $`_primary_term` #> [1] 1 #> #> $found #> [1] TRUEGet multiple documents via the multiget API
Same index and different document ids
docs_mget(x, index = "plos", id = 1:2) #> $docs #> $docs[[1]] #> $docs[[1]]$`_index` #> [1] "plos" #> #> $docs[[1]]$`_type` #> [1] "_doc" #> #> $docs[[1]]$`_id` #> [1] "1" #> #> $docs[[1]]$`_version` #> [1] 1 #> #> $docs[[1]]$`_seq_no` #> [1] 1 #> #> $docs[[1]]$`_primary_term` #> [1] 1 #> #> $docs[[1]]$found #> [1] TRUE #> #> $docs[[1]]$`_source` #> $docs[[1]]$`_source`$id #> [1] "10.1371/journal.pone.0098602" #> #> $docs[[1]]$`_source`$title #> [1] "Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar" #> #> #> #> $docs[[2]] #> $docs[[2]]$`_index` #> [1] "plos" #> #> $docs[[2]]$`_type` #> [1] "_doc" #> #> $docs[[2]]$`_id` #> [1] "2" #> #> $docs[[2]]$`_version` #> [1] 1 #> #> $docs[[2]]$`_seq_no` #> [1] 2 #> #> $docs[[2]]$`_primary_term` #> [1] 1 #> #> $docs[[2]]$found #> [1] TRUE #> #> $docs[[2]]$`_source` #> $docs[[2]]$`_source`$id #> [1] "10.1371/journal.pone.0107757" #> #> $docs[[2]]$`_source`$title #> [1] "Cigarette Smoke Extract Induces a Phenotypic Shift in Epithelial Cells; Involvement of HIF1\u03b1 in Mesenchymal Transition"Parsing
You can optionally get back raw
jsonfromSearch(),docs_get(), anddocs_mget()setting parameterraw=TRUE.For example:
(out [1] "{\"docs\":[{\"_index\":\"plos\",\"_type\":\"_doc\",\"_id\":\"1\",\"_version\":1,\"_seq_no\":1,\"_primary_term\":1,\"found\":true,\"_source\":{\"id\":\"10.1371/journal.pone.0098602\",\"title\":\"Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar\"}},{\"_index\":\"plos\",\"_type\":\"_doc\",\"_id\":\"2\",\"_version\":1,\"_seq_no\":2,\"_primary_term\":1,\"found\":true,\"_source\":{\"id\":\"10.1371/journal.pone.0107757\",\"title\":\"Cigarette Smoke Extract Induces a Phenotypic Shift in Epithelial Cells; Involvement of HIF1\u03b1 in Mesenchymal Transition\"}}]}" #> attr(,"class") #> [1] "elastic_mget"Then parse
jsonlite::fromJSON(out) #> $docs #> _index _type _id _version _seq_no _primary_term found #> 1 plos _doc 1 1 1 1 TRUE #> 2 plos _doc 2 1 2 1 TRUE #> _source.id #> 1 10.1371/journal.pone.0098602 #> 2 10.1371/journal.pone.0107757 #> _source.title #> 1 Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar #> 2 Cigarette Smoke Extract Induces a Phenotypic Shift in Epithelial Cells; Involvement of HIF1\u03b1 in Mesenchymal TransitionKnown pain points
HEADrequests don't seem to work, not sure why
GETrequests, a number of functions that require
POSTrequests obviously then won't work. A big one is
Search(), but you can use
Search_uri()to get around this, which uses
GETinstead of
POST, but you can't pass a more complicated query via the body
A screencast introducing the package: vimeo.com/124659179
elasticin R doing
citation(package = 'elastic')