Repository with examples and smoke tests for the GCP Airflow operators and hooks
The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:
Repository with examples and smoke tests for the GCP Airflow operators and hooks.
This Google Cloud Examples does assume you will have a standard Airflow setup up and running. This tutorial does work perfectly locally as in a production setup because the only requirement is that you have a service key, that we'll explain next. But first a quick rundown of what you need:
Make sure you're running the LocalExecutor and have a decent database setup.
Next create a service account where you want your smoke tests and examples to run in. Go to the console:
And create a service key. Provision a JSON private key and give it Editor's rights
In Airflow you need to define the gcp_smoke named connection to your project:
Supply the path to the downloaded private key, supply the project_id and define the minimum scope of https://www.googleapis.com/auth/cloud-platform
You need to setup variables that are used in the examples. You can tweak them to suite your environment.
|gcproject||my-project||Project where the examples will run in|
|gcqdataset||airflow||BigQuery dataset for examples|
|gcqtempset||airflowtemp||BiqQuery dataset with 1 day retentions|
|gcsroot||data||Storage root path (required, no start and end with slash)|