Need help with hadoop-connectors?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

GoogleCloudDataproc
219 Stars 172 Forks Apache License 2.0 1.2K Commits 25 Opened issues

Description

Libraries and tools for interoperability between Hadoop-related open-source software and Google Cloud Platform.

Services available

!
?

Need anything else?

Contributors list

Apache Hadoop Connectors

GitHub release GitHub release date Code Quality: Java codecov

Libraries and tools for interoperability between Apache Hadoop related open-source software and Google Cloud Platform.

Google Cloud Storage connector for Apache Hadoop (HCFS)

Maven Central Maven Central Maven Central

The Google Cloud Storage connector for Hadoop enables running MapReduce jobs directly on data in Google Cloud Storage by implementing the Hadoop FileSystem interface. For details, see the README.

Google BigQuery connector for Apache Hadoop MapReduce

Maven Central Maven Central Maven Central

The Google BigQuery connector for Hadoop MapReduce enables running MapReduce jobs on data in BigQuery by implementing the

InputFormat
&
OutputFormat
interfaces. For more details see the documentation

Building the Cloud Storage and BigQuery connectors

Note that build requires Java 8 and fails with newer Java versions.

To build the connector for specific Hadoop version, run the following commands from the main directory:

# with Hadoop 2 and YARN support:
./mvnw -P hadoop2 clean package

with Hadoop 3 and YARN support:

./mvnw -P hadoop3 clean package

In order to verify test coverage for specific Hadoop version, run the following commands from the main directory:

# with Hadoop 2 and YARN support:
./mvnw -P hadoop2 -P coverage clean verify

with Hadoop 3 and YARN support:

./mvnw -P hadoop3 -P coverage clean verify

The Cloud Storage connector JAR can be found in

gcs/target/
. The BigQuery connector JAR can be found in
bigquery/target/
.

Adding the Cloud Storage and BigQuery connectors to your build

Maven group ID is

com.google.cloud.bigdataoss
and artifact ID for Cloud Storage connector is
gcs-connector
and for BigQuery connectors is
bigquery-connector
.

To add a dependency on one of the connectors using Maven, use the following:

  • Cloud Storage connector:

      com.google.cloud.bigdataoss
      gcs-connector
      hadoop2-2.2.0
    
    
  • BigQuery connector:

      com.google.cloud.bigdataoss
      bigquery-connector
      hadoop2-1.2.0
    
    

Resources

On Stack Overflow, use the tag

google-cloud-dataproc
for questions about the connectors in this repository. This tag receives responses from the Stack Overflow community and Google engineers, who monitor the tag and offer unofficial support.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.