Become an AceSign inSign up
ropensci/ elastic (v1.2.0) 9 months ago
r-package elasticsearch data-science R Database HTTP etl
View on GitHub
Need help with elastic?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

ropensci
232 Stars 59 Forks Other 856 Commits 0 Opened issues

Description

R client for the Elasticsearch HTTP API

Services available

!
?

Need anything else?

Contributors list

elastic

Project Status: Active – The project has reached a stable, usable state and is being actively developed. R-check cran checks rstudio mirror downloads cran version <!-- codecov.io -->

A general purpose R interface to Elasticsearch

Elasticsearch info

Compatibility

This client is developed following the latest stable releases, currently 

v7.10.0
. It is generally compatible with older versions of Elasticsearch. Unlike the Python client, we try to keep as much compatibility as possible within a single version of this client, as that's an easier setup in R world.

Security

You're fine running ES locally on your machine, but be careful just throwing up ES on a server with a public IP address - make sure to think about security.

  • Elastic has paid products - but probably only applicable to enterprise users
  • DIY security - there are a variety of techniques for securing your Elasticsearch installation. A number of resources are collected in a blog post - tools include putting your ES behind something like Nginx, putting basic auth on top of it, using https, etc.

Installation

Stable version from CRAN

install.packages("elastic")

Development version from GitHub

remotes::install_github("ropensci/elastic")

library('elastic')

Install Elasticsearch

w/ Docker

Pull the official elasticsearch image

# elasticsearch needs to have a version tag. We're pulling 7.10.1 here
docker pull elasticsearch:7.10.1

Then start up a container

docker run -d -p 9200:9200 elasticsearch:7.10.1

Then elasticsearch should be available on port 9200, try 

curl localhost:9200
and you should get the familiar message indicating ES is on.

If you're using boot2docker, you'll need to use the IP address in place of localhost. Get it by doing 

boot2docker ip
.

on OSX

  • Download zip or tar file from Elasticsearch see here for download, e.g., 
    curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.0-darwin-x86_64.tar.gz
  • Extract: 
    tar -zxvf elasticsearch-7.10.0-darwin-x86_64.tar.gz
  • Move it: 
    sudo mv elasticsearch-7.10.0 /usr/local
  • Navigate to /usr/local: 
    cd /usr/local
  • Delete symlinked 
    elasticsearch
    directory: 
    rm -rf elasticsearch
  • Add shortcut: 
    sudo ln -s elasticsearch-7.10.0 elasticsearch
    (replace version with your version)

You can also install via Homebrew: 

brew install elasticsearch

Note: for the 1.6 and greater upgrades of Elasticsearch, they want you to have java 8 or greater. I downloaded Java 8 from here http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html and it seemed to work great.

Upgrading Elasticsearch

I am not totally clear on best practice here, but from what I understand, when you upgrade to a new version of Elasticsearch, place old 

elasticsearch/data
and 
elasticsearch/config
directories into the new installation (
elasticsearch/
dir). The new elasticsearch instance with replaced data and config directories should automatically update data to the new version and start working. Maybe if you use homebrew on a Mac to upgrade it takes care of this for you - not sure.

Obviously, upgrading Elasticsearch while keeping it running is a different thing (some help here from Elastic).

Start Elasticsearch

  • Navigate to elasticsearch: 
    cd /usr/local/elasticsearch
  • Start elasticsearch: 
    bin/elasticsearch

I create a little bash shortcut called 

es
that does both of the above commands in one step (
cd /usr/local/elasticsearch && bin/elasticsearch
).

Initialization

The function 

connect()
is used before doing anything else to set the connection details to your remote or local elasticsearch store. The details created by 
connect()
are written to your options for the current session, and are used by 
elastic
functions. 
x 



If you're following along here with a local instance of Elasticsearch, you'll use 


x
 below to 
do more stuff.




For AWS hosted elasticsearch, make sure to specify path = "" and the correct port - transport schema pair.


connect(host = , path = "", port = 80, transport_schema  = "http")
  # or
connect(host = , path = "", port = 443, transport_schema  = "https")



If you are using Elastic Cloud or an installation with authentication (X-pack), make sure to specify path = "", user = "", pwd = "" and the correct port - transport schema pair.


connect(host = , path = "", user="test", pwd = "1234", port = 9243, transport_schema  = "https")







Get some data



Elasticsearch has a bulk load API to load data in fast. The format is pretty weird though. It's sort of JSON, but would pass no JSON linter. I include a few data sets in 
elastic
 so it's easy to get up and running, and so when you run examples in this package they'll actually run the same way (hopefully).


I have prepare a non-exported function useful for preparing the weird format that Elasticsearch wants for bulk data loads, that is somewhat specific to PLOS data (See below), but you could modify for your purposes. See 
make_bulk_plos()
 and 
make_bulk_gbif()
 here.


Shakespeare data



Elasticsearch provides some data on Shakespeare plays. I've provided a subset of this data in this package. Get the path for the file specific to your machine:


shakespeare 

Then load the data into Elasticsearch:





make sure to create your connection object with 


connect()




# x 

If you need some big data to play with, the shakespeare dataset is a good one to start with. You can get the whole thing and pop it into Elasticsearch (beware, may take up to 10 minutes or so.):


curl -XGET https://download.elastic.co/demos/kibana/gettingstarted/shakespeare_6.0.json > shakespeare.json
curl -XPUT localhost:9200/_bulk --data-binary @shakespeare.json



Public Library of Science (PLOS) data



A dataset inluded in the 
elastic
 package is metadata for PLOS scholarly articles. Get the file path, then load:

if (index_exists(x, "plos")) index_delete(x, "plos")
plosdat 

Global Biodiversity Information Facility (GBIF) data



A dataset inluded in the 
elastic
 package is data for GBIF species occurrence records. Get the file path, then load:

if (index_exists(x, "gbif")) index_delete(x, "gbif")
gbifdat 

GBIF geo data with a coordinates element to allow 
geo_shape
 queries

if (index_exists(x, "gbifgeo")) index_delete(x, "gbifgeo")
gbifgeo 

More data sets



There are more datasets formatted for bulk loading in the 
sckott/elastic_data
 GitHub repository. Find it at https://github.com/sckott/elastic_data






Search



Search the 
plos
 index and only return 1 result

Search(x, index = "plos", size = 1)$hits$hits
#> [[1]]
#> [[1]]$`_index`
#> [1] "plos"
#> 
#> [[1]]$`_type`
#> [1] "_doc"
#> 
#> [[1]]$`_id`
#> [1] "0"
#> 
#> [[1]]$`_score`
#> [1] 1
#> 
#> [[1]]$`_source`
#> [[1]]$`_source`$id
#> [1] "10.1371/journal.pone.0007737"
#> 
#> [[1]]$`_source`$title
#> [1] "Phospholipase C-\u03b24 Is Essential for the Progression of the Normal Sleep Sequence and Ultradian Body Temperature Rhythms in Mice"



Search the 
plos
 index, and query for antibody, limit to 1 result

Search(x, index = "plos", q = "antibody", size = 1)$hits$hits
#> [[1]]
#> [[1]]$`_index`
#> [1] "plos"
#> 
#> [[1]]$`_type`
#> [1] "_doc"
#> 
#> [[1]]$`_id`
#> [1] "813"
#> 
#> [[1]]$`_score`
#> [1] 5.18676
#> 
#> [[1]]$`_source`
#> [[1]]$`_source`$id
#> [1] "10.1371/journal.pone.0107638"
#> 
#> [[1]]$`_source`$title
#> [1] "Sortase A Induces Th17-Mediated and Antibody-Independent Immunity to Heterologous Serotypes of Group A Streptococci"



Get documents



Get document with id=4


docs_get(x, index = 'plos', id = 4)
#> $`_index`
#> [1] "plos"
#> 
#> $`_type`
#> [1] "_doc"
#> 
#> $`_id`
#> [1] "4"
#> 
#> $`_version`
#> [1] 1
#> 
#> $`_seq_no`
#> [1] 4
#> 
#> $`_primary_term`
#> [1] 1
#> 
#> $found
#> [1] TRUE
#> 
#> $`_source`
#> $`_source`$id
#> [1] "10.1371/journal.pone.0107758"
#> 
#> $`_source`$title
#> [1] "Lactobacilli Inactivate Chlamydia trachomatis through Lactic Acid but Not H2O2"



Get certain fields


docs_get(x, index = 'plos', id = 4, fields = 'id')
#> $`_index`
#> [1] "plos"
#> 
#> $`_type`
#> [1] "_doc"
#> 
#> $`_id`
#> [1] "4"
#> 
#> $`_version`
#> [1] 1
#> 
#> $`_seq_no`
#> [1] 4
#> 
#> $`_primary_term`
#> [1] 1
#> 
#> $found
#> [1] TRUE



Get multiple documents via the multiget API



Same index and different document ids


docs_mget(x, index = "plos", id = 1:2)
#> $docs
#> $docs[[1]]
#> $docs[[1]]$`_index`
#> [1] "plos"
#> 
#> $docs[[1]]$`_type`
#> [1] "_doc"
#> 
#> $docs[[1]]$`_id`
#> [1] "1"
#> 
#> $docs[[1]]$`_version`
#> [1] 1
#> 
#> $docs[[1]]$`_seq_no`
#> [1] 1
#> 
#> $docs[[1]]$`_primary_term`
#> [1] 1
#> 
#> $docs[[1]]$found
#> [1] TRUE
#> 
#> $docs[[1]]$`_source`
#> $docs[[1]]$`_source`$id
#> [1] "10.1371/journal.pone.0098602"
#> 
#> $docs[[1]]$`_source`$title
#> [1] "Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar"
#> 
#> 
#> 
#> $docs[[2]]
#> $docs[[2]]$`_index`
#> [1] "plos"
#> 
#> $docs[[2]]$`_type`
#> [1] "_doc"
#> 
#> $docs[[2]]$`_id`
#> [1] "2"
#> 
#> $docs[[2]]$`_version`
#> [1] 1
#> 
#> $docs[[2]]$`_seq_no`
#> [1] 2
#> 
#> $docs[[2]]$`_primary_term`
#> [1] 1
#> 
#> $docs[[2]]$found
#> [1] TRUE
#> 
#> $docs[[2]]$`_source`
#> $docs[[2]]$`_source`$id
#> [1] "10.1371/journal.pone.0107757"
#> 
#> $docs[[2]]$`_source`$title
#> [1] "Cigarette Smoke Extract Induces a Phenotypic Shift in Epithelial Cells; Involvement of HIF1\u03b1 in Mesenchymal Transition"



Parsing



You can optionally get back raw 
json
 from 
Search()
, 
docs_get()
, and 
docs_mget()
 setting parameter 
raw=TRUE
.


For example:


(out  [1] "{\"docs\":[{\"_index\":\"plos\",\"_type\":\"_doc\",\"_id\":\"1\",\"_version\":1,\"_seq_no\":1,\"_primary_term\":1,\"found\":true,\"_source\":{\"id\":\"10.1371/journal.pone.0098602\",\"title\":\"Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar\"}},{\"_index\":\"plos\",\"_type\":\"_doc\",\"_id\":\"2\",\"_version\":1,\"_seq_no\":2,\"_primary_term\":1,\"found\":true,\"_source\":{\"id\":\"10.1371/journal.pone.0107757\",\"title\":\"Cigarette Smoke Extract Induces a Phenotypic Shift in Epithelial Cells; Involvement of HIF1\u03b1 in Mesenchymal Transition\"}}]}"
#> attr(,"class")
#> [1] "elastic_mget"



Then parse


jsonlite::fromJSON(out)
#> $docs
#>   _index _type _id _version _seq_no _primary_term found
#> 1   plos  _doc   1        1       1             1  TRUE
#> 2   plos  _doc   2        1       2             1  TRUE
#>                     _source.id
#> 1 10.1371/journal.pone.0098602
#> 2 10.1371/journal.pone.0107757
#>                                                                                                                                                _source.title
#> 1 Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar
#> 2                                Cigarette Smoke Extract Induces a Phenotypic Shift in Epithelial Cells; Involvement of HIF1\u03b1 in Mesenchymal Transition



Known pain points

  • On secure Elasticsearch servers: 
    • HEAD
      requests don't seem to work, not sure why
    • If you allow only 
      GET
      requests, a number of functions that require 
      POST
      requests obviously then won't work. A big one is 
      Search()
      , but you can use 
      Search_uri()
      to get around this, which uses 
      GET
      instead of 
      POST
      , but you can't pass a more complicated query via the body

Screencast

A screencast introducing the package: vimeo.com/124659179

Meta

  • Please report any issues or bugs
  • License: MIT
  • Get citation information for 
    elastic
    in R doing 
    citation(package = 'elastic')
  • Please note that this package is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.