Need help with airflow-gcp-examples?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

136 Stars 35 Forks Apache License 2.0 8 Commits 3 Opened issues


Repository with examples and smoke tests for the GCP Airflow operators and hooks

Services available


Need anything else?

Contributors list

# 31,441
8 commits


Repository with examples and smoke tests for the GCP Airflow operators and hooks.


This Google Cloud Examples does assume you will have a standard Airflow setup up and running. This tutorial does work perfectly locally as in a production setup because the only requirement is that you have a service key, that we'll explain next. But first a quick rundown of what you need:

  • Running Airflow (as of this writing you need Airflow master branch!!!)
  • Create a service account (Cloud Console)
  • Setup a Google Cloud Connection in Airflow
  • Setup variables that the DAG's will need
  • Copy the DAG's to your dags folder

Airflow setup

  • Checkout master of Airflow
  • pip install google-api-python-client
  • python install

Make sure you're running the LocalExecutor and have a decent database setup.

Google Cloud Service Key

Next create a service account where you want your smoke tests and examples to run in. Go to the console:


And create a service key. Provision a JSON private key and give it Editor's rights


Airflow Connection

In Airflow you need to define the gcp_smoke named connection to your project:


Supply the path to the downloaded private key, supply the project_id and define the minimum scope of

Airflow Variables

You need to setup variables that are used in the examples. You can tweak them to suite your environment.



example value note
gcproject my-project Project where the examples will run in
gcqdataset airflow BigQuery dataset for examples
gcqtempset airflowtemp BiqQuery dataset with 1 day retentions
gcsbucket airflow-gcp-smoke Storage bucket
gcsroot data Storage root path (required, no start and end with slash)

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.