airflow-gcp-examples

by alexvanboxel

alexvanboxel / airflow-gcp-examples

Repository with examples and smoke tests for the GCP Airflow operators and hooks

130 Stars 34 Forks Last release: Not found Apache License 2.0 8 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

airflow-gcp-examples

Repository with examples and smoke tests for the GCP Airflow operators and hooks.

Setup

This Google Cloud Examples does assume you will have a standard Airflow setup up and running. This tutorial does work perfectly locally as in a production setup because the only requirement is that you have a service key, that we'll explain next. But first a quick rundown of what you need:

  • Running Airflow (as of this writing you need Airflow master branch!!!)
  • Create a service account (Cloud Console)
  • Setup a Google Cloud Connection in Airflow
  • Setup variables that the DAG's will need
  • Copy the DAG's to your dags folder

Airflow setup

  • Checkout master of Airflow
  • pip install google-api-python-client
  • python setup.py install

Make sure you're running the LocalExecutor and have a decent database setup.

Google Cloud Service Key

Next create a service account where you want your smoke tests and examples to run in. Go to the console:

console

And create a service key. Provision a JSON private key and give it Editor's rights

console

Airflow Connection

In Airflow you need to define the gcp_smoke named connection to your project:

console

Supply the path to the downloaded private key, supply the project_id and define the minimum scope of https://www.googleapis.com/auth/cloud-platform

Airflow Variables

You need to setup variables that are used in the examples. You can tweak them to suite your environment.

console

variable

example value note
gcproject my-project Project where the examples will run in
gcqdataset airflow BigQuery dataset for examples
gcqtempset airflowtemp BiqQuery dataset with 1 day retentions
gcsbucket airflow-gcp-smoke Storage bucket
gcsroot data Storage root path (required, no start and end with slash)

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.