Chaos Toolkit Extensions for Kubernetes

This project contains activities, such as probes and actions, you can call from your experiment through the Chaos Toolkit to perform Chaos Engineering against the Kubernetes API: killing a pod, removing a statefulset or node...

Install

To be used from your experiment, this package must be installed in the Python environment where chaostoolkit already lives.

$ pip install chaostoolkit-kubernetes

Usage

To use the probes and actions from this package, add the following to your experiment file:

{
    "title": "Do we remain available in face of pod going down?",
    "description": "We expect Kubernetes to handle the situation gracefully when a pod goes down",
    "tags": ["kubernetes"],
    "steady-state-hypothesis": {
        "title": "Verifying service remains healthy",
        "probes": [
            {
                "name": "all-our-microservices-should-be-healthy",
                "type": "probe",
                "tolerance": true,
                "provider": {
                    "type": "python",
                    "module": "chaosk8s.probes",
                    "func": "microservice_available_and_healthy",
                    "arguments": {
                        "name": "myapp"
                    }
                }
            }
        ]
    },
    "method": [
        {
            "type": "action",
            "name": "terminate-db-pod",
            "provider": {
                "type": "python",
                "module": "chaosk8s.pod.actions",
                "func": "terminate_pods",
                "arguments": {
                    "label_selector": "app=my-app",
                    "name_pattern": "my-app-[0-9]$",
                    "rand": true
                }
            },
            "pauses": {
                "after": 5
            }
        }
    ]
}

That's it! Notice how the action gives you the way to kill one pod randomly.

Please explore the documentation to see existing probes and actions.

Configuration

Use ~/.kube/config

If you have a valid entry in your

~/.kube/config

You may specify

KUBECONFIG

$ export KUBECONFIG=/tmp/my-config

Specify the Kubernetes context

Quite often, your Kubernetes configuration contains several entries and you need to define the one to use as a default context when not it isn't explicitely provided.

You may of course change your default using

kubectl config use-context KUBERNETES_CONTEXT

{
    "title": "Do we remain available in face of pod going down?",
    "description": "We expect Kubernetes to handle the situation gracefully when a pod goes down",
    "tags": ["kubernetes"],
    "secrets": {
        "k8s": {
            "KUBERNETES_CONTEXT": "..."
        }
    },
    "steady-state-hypothesis": {
        "title": "Verifying service remains healthy",
        "probes": [
            {
                "name": "all-our-microservices-should-be-healthy",
                "type": "probe",
                "tolerance": true,
                "secrets": ["k8s"],
                "provider": {
                    "type": "python",
                    "module": "chaosk8s.probes",
                    "func": "microservice_available_and_healthy",
                    "arguments": {
                        "name": "myapp"
                    }
                }
            }
        ]
    },
    "method": [
        {
            "type": "action",
            "name": "terminate-db-pod",
            "secrets": ["k8s"],
            "provider": {
                "type": "python",
                "module": "chaosk8s.pod.actions",
                "func": "terminate_pods",
                "arguments": {
                    "label_selector": "app=my-app",
                    "name_pattern": "my-app-[0-9]$",
                    "rand": true
                }
            },
            "pauses": {
                "after": 5
            }
        }
    ]
}

You need to specify the

KUBERNETES_CONTEXT

"secrets": ["k8s"]

Use a Pod's service account

When running from a pod (not your local machine or a CI for instance), the

./.kube/config

To let the extension know about this, simply set

CHAOSTOOLKIT_IN_POD

env:
- name: CHAOSTOOLKIT_IN_POD
  value: "true"

Pass all credentials in the experiment

Finally, you may pass explicitely all required credentials information to the experiment as follows:

Using an API key

{
    "secrets": {
        "kubernetes": {
            "KUBERNETES_HOST": "http://somehost",
            "KUBERNETES_API_KEY": {
                "type": "env",
                "key": "SOME_ENV_VAR"
            }
        }
    }
}

Using a username/password

{
    "secrets": {
        "kubernetes": {
            "KUBERNETES_HOST": "http://somehost",
            "KUBERNETES_USERNAME": {
                "type": "env",
                "key": "SOME_ENV_VAR"
            },
            "KUBERNETES_PASSWORD": {
                "type": "env",
                "key": "SOME_ENV_VAR"
            }
        }
    }
}

Using a TLS key/certificate

{
    "secrets": {
        "kubernetes": {
            "KUBERNETES_HOST": "http://somehost",
            "KUBERNETES_CERT_FILE": {
                "type": "env",
                "key": "SOME_ENV_VAR"
            },
            "KUBERNETES_KEY_FILE": {
                "type": "env",
                "key": "SOME_ENV_VAR"
            }
        }
    }
}

Managed Kubernetes Clusters Authentication

On some managed Kubernetes clusters, you also need to authenticate against the platform itself because the Kubernetes authentication is delegated to it.

Google Cloud Platform

In addition to your Kubernetes credentials (via the

~/.kube/config

$ gcloud auth login

But can also be achieved by defining the

GOOGLE_APPLICATION_CREDENTIALS

Contribute

If you wish to contribute more functions to this package, you are more than welcome to do so. Please, fork this project, write unit tests to cover the proposed changes, implement the changes, ensure they meet the formatting standards set out by

black

flake8

isort

Please refer to the formatting section for more information on the formatting standards.

The Chaos Toolkit projects require all contributors must sign a Developer Certificate of Origin on each commit they would like to merge into the master branch of the repository. Please, make sure you can abide by the rules of the DCO before submitting a PR.

Develop

If you wish to develop on this project, make sure to install the development dependencies. But first, create a virtual environment and then install those dependencies.

$ make install-dev

Now, you can edit the files and they will be automatically be seen by your environment, even when running from the

chaos

Tests

To run the tests for the project execute the following:

$ make tests

Formatting and Linting

We use a combination of

black

flake8

isort

Before raising a Pull Request, we recommend you run formatting against your code with:

$ make format

This will automatically format any code that doesn't adhere to the formatting standards.

As some things are not picked up by the formatting, we also recommend you run:

$ make lint

To ensure that any unused import statements/strings that are too long, etc. are also picked up.