H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
H2O is an in-memory platform for distributed, scalable machine learning. H2O uses familiar interfaces like R, Python, Scala, Java, JSON and the Flow notebook/web interface, and works seamlessly with big data technologies like Hadoop and Spark. H2O provides implementations of many popular algorithms such as Generalized Linear Models (GLM), Gradient Boosting Machines (including XGBoost), Random Forests, Deep Neural Networks, Stacked Ensembles, Naive Bayes, Generalized Additive Models (GAM), Cox Proportional Hazards, K-Means, PCA, Word2Vec, as well as a fully automatic machine learning algorithm (H2O AutoML).
H2O is extensible so that developers can add data transformations and custom algorithms of their choice and access them through all of those clients. H2O models can be downloaded and loaded into H2O memory for scoring, or exported into POJO or MOJO format for extemely fast scoring in production. More information can be found in the H2O User Guide.
H2O-3 (this repository) is the third incarnation of H2O, and the successor to H2O-2.
While most of this README is written for developers who do their own builds, most H2O users just download and use a pre-built version. If you are a Python or R user, the easiest way to install H2O is via PyPI or Anaconda (for Python) or CRAN (for R):
pip install h2o
install.packages("h2o")
For the latest stable, nightly, Hadoop (or Spark / Sparkling Water) releases, or the stand-alone H2O jar, please visit: https://h2o.ai/download
More info on downloading & installing H2O is available in the H2O User Guide.
Most people interact with three or four primary open source resources: GitHub (which you've already found), JIRA (for bug reports and issue tracking), Stack Overflow for H2O code/software-specific questions, and h2ostream (a Google Group / email discussion forum) for questions not suitable for Stack Overflow. There is also a Gitter H2O developer chat group, however for archival purposes & to maximize accessibility, we'd prefer that standard H2O Q&A be conducted on Stack Overflow.
(Note: There is only one issue tracking system for the project. GitHub issues are not enabled; you must use JIRA.)
You can browse and create new issues in our open source JIRA: http://jira.h2o.ai
Issuesmenu
Search for issues
Log Inbutton on the top right of the screen
Create an acccountnear the bottom of the login box
Createbutton on the menu to create an issue
Twitter -- follow us for updates and H2O news!
Awesome H2O -- share your H2O-powered creations with us
Every nightly build publishes R, Python, Java, and Scala artifacts to a build-specific repository. In particular, you can find Java artifacts in the maven/repo directory.
Here is an example snippet of a gradle build file using h2o-3 as a dependency. Replace x, y, z, and nnnn with valid numbers.
// h2o-3 dependency information def h2oBranch = 'master' def h2oBuildNumber = 'nnnn' def h2oProjectVersion = "x.y.z.${h2oBuildNumber}"repositories { // h2o-3 dependencies maven { url "https://s3.amazonaws.com/h2o-release/h2o-3/${h2oBranch}/${h2oBuildNumber}/maven/repo/" } }
dependencies { compile "ai.h2o:h2o-core:${h2oProjectVersion}" compile "ai.h2o:h2o-algos:${h2oProjectVersion}" compile "ai.h2o:h2o-web:${h2oProjectVersion}" compile "ai.h2o:h2o-app:${h2oProjectVersion}" }
Refer to the latest H2O-3 bleeding edge nightly build page for information about installing nightly build artifacts.
Refer to the h2o-droplets GitHub repository for a working example of how to use Java artifacts with gradle.
Note: Stable H2O-3 artifacts are periodically published to Maven Central (click here to search) but may substantially lag behind H2O-3 Bleeding Edge nightly builds.
Getting started with H2O development requires JDK 1.7, Node.js, Gradle, Python and R. We use the Gradle wrapper (called
gradlew) to ensure up-to-date local versions of Gradle and other dependencies are installed in your development directory.
Building
h2orequires a properly set up R environment with required packages and Python environment with the following packages:
grip colorama future tabulate requests wheel
To install these packages you can use pip or conda. If you have troubles installing these packages on Windows, please follow section Setup on Windows of this guide.
(Note: It is recommended to use some virtual environment such as VirtualEnv, to install all packages. )
To build H2O from the repository, perform the following steps.
# Build H2O git clone https://github.com/h2oai/h2o-3.git cd h2o-3 ./gradlew build -x testYou may encounter problems: e.g. npm missing. Install it: brew install npm
Start H2O
java -jar build/h2o.jar
Point browser to http://localhost:54321
git clone https://github.com/h2oai/h2o-3.git cd h2o-3 ./gradlew syncSmalldata ./gradlew syncRPackages ./gradlew build
Notes:
- Running tests starts five test JVMs that form an H2O cluster and requires at least 8GB of RAM (preferably 16GB of RAM).
- Running
./gradlew syncRPackagesis supported on Windows, OS X, and Linux, and is strongly recommended but not required../gradlew syncRPackagesensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using./gradlew syncRPackages. To set the ENV variable, use the following format (where `${WORKSPACE} can be any path):mkdir -p ${WORKSPACE}/Rlibrary export R_LIBS_USER=${WORKSPACE}/Rlibrary
git pull ./gradlew syncSmalldata ./gradlew syncRPackages ./gradlew clean ./gradlew build
We recommend using
./gradlew cleanafter each
git pull.
-x testat the end the gradle build command line. Tests typically run for 7-10 minutes on a Macbook Pro laptop with 4 CPUs (8 hyperthreads) and 16 GB of RAM.
./gradlew syncSmalldataas the first troubleshooting step. Syncing smalldata downloads data files from AWS S3 to the smalldata directory in your workspace. The sync is incremental. Do not check in these files. The smalldata directory is in .gitignore. If you do not run any tests, you do not need the smalldata directory.
./gradlew syncRPackagesis supported on Windows, OS X, and Linux, and is strongly recommended but not required.
./gradlew syncRPackagesensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using
./gradlew syncRPackages. To set the ENV variable, use the following format (where
${WORKSPACE}can be any path):
mkdir -p ${WORKSPACE}/Rlibrary export R_LIBS_USER=${WORKSPACE}/Rlibrary
./gradlew clean && ./gradlew build -x test && (export DO_FAST=1; ./gradlew dist) open target/docs-website/h2o-docs/index.html
From the command line, validate
pythonis using the newly installed package by using
which python(or
sudo which python). Update the Environment variable with the WinPython path.
pip install grip 'colorama>=0.3.8' future tabulate wheel
Install Java 1.7 and add the appropriate directory
C:\Program Files\Java\jdk1.7.0_65\binwith java.exe to PATH in Environment Variables. To make sure the command prompt is detecting the correct Java version, run:
javac -version
The CLASSPATH variable also needs to be set to the lib subfolder of the JDK:
CLASSPATH=////lib
Install Node.js and add the installed directory
C:\Program Files\nodejs, which must include node.exe and npm.cmd to PATH if not already prepended.
Install R and add the bin directory to your PATH if not already included.
Install the following R packages:
To install these packages from within an R session:
pkgsNote that libcurl is required for installation of the RCurl R package.
Note that this packages don't cover running tests, they for building H2O only.
Finally, install Rtools, which is a collection of command line tools to facilitate R development on Windows.
NOTE: During Rtools installation, do not install Cygwin.dll.
Step 6. Install Cygwin
NOTE: During installation of Cygwin, deselect the Python packages to avoid a conflict with the Python.org package.
Step 6b. Validate Cygwin
If Cygwin is already installed, remove the Python packages or ensure that Native Python is before Cygwin in the PATH variable.
Step 7. Update or validate the Windows PATH variable to include R, Java JDK, Cygwin.
Step 8. Git Clone h2o-3
If you don't already have a Git client, please install one. The default one can be found here http://git-scm.com/downloads. Make sure that command prompt support is enabled before the installation.
Download and update h2o-3 source codes:
git clone https://github.com/h2oai/h2o-3Step 9. Run the top-level gradle build:
cd h2o-3 ./gradlew.bat buildIf you encounter errors run again with
--stacktracefor more instructions on missing dependencies.4.4. Setup on OS X
If you don't have Homebrew, we recommend installing it. It makes package management for OS X easy.
Step 1. Install JDK
Install Java 1.7. To make sure the command prompt is detecting the correct Java version, run:
javac -versionStep 2. Install Node.js:
Using Homebrew:
brew install nodeOtherwise, install from the NodeJS website.
Step 3. Install R and the required packages:
Install R and add the bin directory to your PATH if not already included.
To install these packages from within an R session:
pkgsNote that libcurl is required for installation of the RCurl R package.
Note that this packages don't cover running tests, they for building H2O only.
Step 4. Install python and the required packages:
Install python:
brew install pythonInstall pip package manager:
sudo easy_install pipNext install required packages:
sudo pip install wheel requests 'colorama>=0.3.8' future tabulateStep 5. Git Clone h2o-3
OS X should already have Git installed. To download and update h2o-3 source codes:
git clone https://github.com/h2oai/h2o-3Step 6. Run the top-level gradle build:
cd h2o-3 ./gradlew buildNote: on a regular machine it may take very long time (about an hour) to run all the tests.
If you encounter errors run again with
--stacktracefor more instructions on missing dependencies.4.5. Setup on Ubuntu 14.04
Step 1. Install Node.js
curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash - sudo apt-get install -y nodejsStep 2. Install JDK:
Install Java 8. Installation instructions can be found here JDK installation. To make sure the command prompt is detecting the correct Java version, run:
javac -versionStep 3. Install R and the required packages:
Installation instructions can be found here R installation. Click “Download R for Linux”. Click “ubuntu”. Follow the given instructions.
To install the required packages, follow the same instructions as for OS X above.
Note: If the process fails to install RStudio Server on Linux, run one of the following:
sudo apt-get install libcurl4-openssl-devor
sudo apt-get install libcurl4-gnutls-devStep 4. Git Clone h2o-3
If you don't already have a Git client:
sudo apt-get install gitDownload and update h2o-3 source codes:
git clone https://github.com/h2oai/h2o-3Step 5. Run the top-level gradle build:
cd h2o-3 ./gradlew buildIf you encounter errors, run again using
--stacktracefor more instructions on missing dependencies.Make sure that you are not running as root, since
bowerwill reject such a run.4.6. Setup on Ubuntu 13.10
Step 1. Install Node.js
curl -sL https://deb.nodesource.com/setup_10.x | sudo bash - sudo apt-get install -y nodejsSteps 2-4. Follow steps 2-4 for Ubuntu 14.04 (above)
4.7. Setup on CentOS 7
cd /opt sudo wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz"sudo tar xzf jdk-7u79-linux-x64.tar.gz cd jdk1.7.0_79
sudo alternatives --install /usr/bin/java java /opt/jdk1.7.0_79/bin/java 2
sudo alternatives --install /usr/bin/jar jar /opt/jdk1.7.0_79/bin/jar 2 sudo alternatives --install /usr/bin/javac javac /opt/jdk1.7.0_79/bin/javac 2 sudo alternatives --set jar /opt/jdk1.7.0_79/bin/jar sudo alternatives --set javac /opt/jdk1.7.0_79/bin/javac
cd /opt
sudo wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm sudo rpm -ivh epel-release-7-5.noarch.rpm
sudo echo "multilib_policy=best" >> /etc/yum.conf sudo yum -y update
sudo yum -y install R R-devel git python-pip openssl-devel libxml2-devel libcurl-devel gcc gcc-c++ make openssl-devel kernel-devel texlive texinfo texlive-latex-fonts libX11-devel mesa-libGL-devel mesa-libGL nodejs npm python-devel numpy scipy python-pandas
sudo pip install scikit-learn grip tabulate statsmodels wheel
mkdir
/Rlibrary export JAVA_HOME=/opt/jdk1.7.0_79 export JRE_HOME=/opt/jdk1.7.0_79/jre export PATH=$PATH:/opt/jdk1.7.0_79/bin:/opt/jdk1.7.0_79/jre/bin export R_LIBS_USER=/Rlibraryinstall local R packages
R -e 'install.packages(c("RCurl","jsonlite","statmod","devtools","roxygen2","testthat"), dependencies=TRUE, repos="http://cran.rstudio.com/")'
cd git clone https://github.com/h2oai/h2o-3.git cd h2o-3
Build H2O
./gradlew syncSmalldata ./gradlew syncRPackages ./gradlew build -x test
5. Launching H2O after Building
To start the H2O cluster locally, execute the following on the command line:
java -jar build/h2o.jarA list of available start-up JVM and H2O options (e.g.
-Xmx,-nthreads,-ip), is available in the H2O User Guide.6. Building H2O on Hadoop
Pre-built H2O-on-Hadoop zip files are available on the download page. Each Hadoop distribution version has a separate zip file in h2o-3.
To build H2O with Hadoop support yourself, first install sphinx for python:
pip install sphinxThen start the build by entering the following from the top-level h2o-3 directory:(export BUILD_HADOOP=1; ./gradlew build -x test) ./gradlew distThis will create a directory called 'target' and generate zip files there. Note that
BUILD_HADOOPis the default behavior when the username isjenkins(refer tosettings.gradle); otherwise you have to request it, as shown above.Adding support for a new version of Hadoop
In the
h2o-hadoopdirectory, each Hadoop version has a build directory for the driver and an assembly directory for the fatjar.You need to:
- Add a new driver directory and assembly directory (each with a
build.gradlefile) inh2o-hadoop- Add these new projects to
h2o-3/settings.gradle- Add the new Hadoop version to
HADOOP_VERSIONSinmake-dist.sh- Add the new Hadoop version to the list in
h2o-dist/buildinfo.jsonSecure user impersonation
Hadoop supports secure user impersonation through its Java API. A kerberos-authenticated user can be allowed to proxy any username that meets specified criteria entered in the NameNode's core-site.xml file. This impersonation only applies to interactions with the Hadoop API or the APIs of Hadoop-related services that support it (this is not the same as switching to that user on the machine of origin).
Setting up secure user impersonation (for h2o):
- Create or find an id to use as proxy which has limited-to-no access to HDFS or related services; the proxy user need only be used to impersonate a user
- (Required if not using h2odriver) If you are not using the driver (e.g. you wrote your own code against h2o's API using Hadoop), make the necessary code changes to impersonate users (see org.apache.hadoop.security.UserGroupInformation)
- In either of Ambari/Cloudera Manager or directly on the NameNode's core-site.xml file, add 2/3 properties for the user we wish to use as a proxy (replace with the simple user name - not the fully-qualified principal name).
hadoop.proxyuser..hosts: the hosts the proxy user is allowed to perform impersonated actions on behalf of a valid user from hadoop.proxyuser..groups: the groups an impersonated user must belong to for impersonation to work with that proxy user hadoop.proxyuser..users: the users a proxy user is allowed to impersonate- Example:
hadoop.proxyuser.myproxyuser.hosts host1,host2 hadoop.proxyuser.myproxyuser.groups group1,group2 hadoop.proxyuser.myproxyuser.users user1,user2- Restart core services such as HDFS & YARN for the changes to take effect
Impersonated HDFS actions can be viewed in the hdfs audit log ('auth:PROXY' should appear in the
ugi=field in entries where this is applicable). YARN similarly should show 'auth:PROXY' somewhere in the Resource Manager UI.To use secure impersonation with h2o's Hadoop driver:
Before this is attempted, see Risks with impersonation, below
When using the h2odriver (e.g. when running with
hadoop jar ...), specify-principal,-keytab, and-run_as_user, in addition to any other arguments needed. If the configuration was successful, the proxy user will log in and impersonate the-run_as_useras long as that user is allowed by either the users or groups configuration property (configured above); this is enforced by HDFS & YARN, not h2o's code. The driver effectively sets its security context as the impersonated user so all supported Hadoop actions will be performed as that user (e.g. YARN, HDFS APIs support securely impersonated users, but others may not).Precautions to take when leveraging secure impersonation
hadoop.proxyuser..hostsproperty whenever possible or practical.
su, for instance)
hadoop.proxyuser..{hosts,groups,users}property to '*' can greatly increase exposure to security risk.
$ git diff diff --git a/h2o-app/build.gradle b/h2o-app/build.gradle index af3b929..097af85 100644 --- a/h2o-app/build.gradle +++ b/h2o-app/build.gradle @@ -8,5 +8,6 @@ dependencies { compile project(":h2o-algos") compile project(":h2o-core") compile project(":h2o-genmodel") + compile project(":h2o-persist-hdfs") }diff --git a/h2o-persist-hdfs/build.gradle b/h2o-persist-hdfs/build.gradle index 41b96b2..6368ea9 100644 --- a/h2o-persist-hdfs/build.gradle +++ b/h2o-persist-hdfs/build.gradle @@ -2,5 +2,6 @@ description = "H2O Persist HDFS"
dependencies { compile project(":h2o-core")
Sparkling Water combines two open-source technologies: Apache Spark and the H2O Machine Learning platform. It makes H2O’s library of advanced algorithms, including Deep Learning, GLM, GBM, K-Means, and Distributed Random Forest, accessible from Spark workflows. Spark users can select the best features from either platform to meet their Machine Learning needs. Users can combine Spark's RDD API and Spark MLLib with H2O’s machine learning algorithms, or use H2O independently of Spark for the model building process and post-process the results in Spark.
Sparkling Water Resources:
The main H2O documentation is the H2O User Guide. Visit http://docs.h2o.ai for the top-level introduction to documentation on H2O projects.
To generate the REST API documentation, use the following commands:
cd ~/h2o-3 cd py python ./generate_rest_api_docs.py # to generate Markdown only python ./generate_rest_api_docs.py --generate_html --github_user GITHUB_USER --github_password GITHUB_PASSWORD # to generate Markdown and HTML
The default location for the generated documentation is
build/docs/REST.
If the build fails, try
gradlew clean, then
git clean -f.
Documentation for each bleeding edge nightly build is available on the nightly build page.
If you use H2O as part of your workflow in a publication, please cite your H2O resource(s) using the following BibTex entry:
@Manual{h2o_package_or_module, title = {package_or_module_title}, author = {H2O.ai}, year = {year}, month = {month}, note = {version_information}, url = {resource_url}, }
Formatted H2O Software citation examples:
H2O algorithm booklets are available at the Documentation Homepage.
@Manual{h2o_booklet_name, title = {booklet_title}, author = {list_of_authors}, year = {year}, month = {month}, url = {link_url}, }
Formatted booklet citation examples:
Arora, A., Candel, A., Lanford, J., LeDell, E., and Parmar, V. (Oct. 2016). Deep Learning with H2O. http://docs.h2o.ai/h2o/latest-stable/h2o-docs/booklets/DeepLearningBooklet.pdf.
Click, C., Lanford, J., Malohlava, M., Parmar, V., and Roark, H. (Oct. 2016). Gradient Boosted Models with H2O. http://docs.h2o.ai/h2o/latest-stable/h2o-docs/booklets/GBMBooklet.pdf.
H2O has been built by a great many number of contributors over the years both within H2O.ai (the company) and the greater open source community. You can begin to contribute to H2O by answering Stack Overflow questions or filing bug reports. Please join us!
SriSatish Ambati Cliff Click Tom Kraljevic Tomas Nykodym Michal Malohlava Kevin Normoyle Spencer Aiello Anqi Fu Nidhi Mehta Arno Candel Josephine Wang Amy Wang Max Schloemer Ray Peck Prithvi Prabhu Brandon Hill Jeff Gambera Ariel Rao Viraj Parmar Kendall Harris Anand Avati Jessica Lanford Alex Tellez Allison Washburn Amy Wang Erik Eckstrand Neeraja Madabhushi Sebastian Vidrio Ben Sabrin Matt Dowle Mark Landry Erin LeDell Andrey Spiridonov Oleg Rogynskyy Nick Martin Nancy Jordan Nishant Kalonia Nadine Hussami Jeff Cramer Stacie Spreitzer Vinod Iyengar Charlene Windom Parag Sanghavi Navdeep Gill Lauren DiPerna Anmol Bal Mark Chan Nick Karpov Avni Wadhwa Ashrith Barthur Karen Hayrapetyan Jo-fai Chow Dmitry Larko Branden Murray Jakub Hava Wen Phan Magnus Stensmo Pasha Stetsenko Angela Bartz Mateusz Dymczyk Micah Stubbs Ivy Wang Terone Ward Leland Wilkinson Wendy Wong Nikhil Shekhar Pavel Pscheidl Michal Kurka Veronika Maurerova Jan Sterba Jan Jendrusak Sebastien Poirier Tomáš Frýda
Scientific Advisory Council
Stephen Boyd Rob Tibshirani Trevor Hastie
Systems, Data, FileSystems and Hadoop
Doug Lea Chris Pouliot Dhruba Borthakur
Jishnu Bhattacharjee, Nexus Venture Partners Anand Babu Periasamy Anand Rajaraman Ash Bhardwaj Rakesh Mathur Michael Marks Egbert Bierman Rajesh Ambati