kubergrunt

kubergrunt

Some of the features of

kubergrunt

• Configuring

  kubectl

  to authenticate with a given EKS cluster. Learn more about authenticating

  kubectl

  to EKS in the our production deployment guide.
• Managing Helm and associated TLS certificates on any Kubernetes cluster.
• Setting up Helm client with TLS certificates on any Kubernetes cluster.
• Generating TLS certificate key pairs and storing them as Kubernetes

  Secrets

  on any Kubernetes cluster.

Installation

The binaries are all built as part of the CI pipeline on each release of the package, and is appended to the corresponding release in the Releases Page. You can download the corresponding binary for your platform from the releases page.

Alternatively, you can install

kubergrunt

v0.5.13

gruntwork-install --binary-name "kubergrunt" --repo "https://github.com/gruntwork-io/kubergrunt" --tag "v0.5.13"

3rd party package managers

Note that third-party Kubergrunt packages may not be updated with the latest version, but are often close. Please check your version against the latest available on the Releases Page.

Chocolatey (Windows):

choco install kubergrunt

Building from source

The main package is in

cmd

go build -o bin/kubergrunt ./cmd

If you need to set a version on the binary (so that

kubergrunt --version

ldflags

go build -o kubergrunt -ldflags "-X main.VERSION=v0.7.6 -extldflags '-static'" ./cmd

Commands

The following commands are available as part of

kubergrunt

1. eks
   • verify
   • configure
   • token
   • oidc-thumbprint
   • deploy
   • sync-core-components
   • cleanup-security-group
   • schedule-coredns
   • drain
2. k8s
   • wait-for-ingress
   • kubectl
3. tls
   • gen
4. Deprecated commands
   • helm

eks

The

eks

kubergrunt

verify

This subcommand verifies that the specified EKS cluster is up and ready. An EKS cluster is considered ready when:

• The cluster status reaches ACTIVE state.
• The cluster Kubernetes API server endpoint responds to http requests.

When passing

--wait

--max-retries

--sleep-between-retries

--max-retries

--sleep-between-retries

kubergrunt eks verify --eks-cluster-arn $EKS_CLUSTER_ARN --wait --max-retries 10 --sleep-between-retries 15s

and the cluster was not active yet, this command will query the AWS API up to 10 times, waiting 15 seconds inbetween each try for a total of 150 seconds (2.5 minutes) before timing out.

Run

kubergrunt eks verify --help

Similar Commands:

• AWS CLI (

  aws eks wait

  ): This command will wait until the EKS cluster reaches the ACTIVE state. Note that oftentimes the Kubernetes API endpoint has a delay in accepting traffic even after reaching the ACTIVE state. We have observed it take up to 1.5 minutes after the cluster becomes ACTIVE before we can have a valid TCP connection with the Kubernetes API endpoint.

configure

This subcommand will setup the installed

kubectl

kubergrunt eks token

For example to setup a

kubectl

kubergrunt eks configure --eks-cluster-arn $EKS_CLUSTER_ARN

Run

kubergrunt eks configure --help

Similar Commands:

• AWS CLI (

  aws eks update-kubeconfig

  ): This command will configure

  kubeconfig

  in a similar manner. Instead of using

  kubergrunt eks token

  , this version will use the

  get-token

  subcommand built into the AWS CLI.

token

This subcommand is used by

kubectl

aws-iam-authenticator

kubergrunt

The

configure

kubergrunt eks

aws-iam-authenticator

kubectl

This subcommand also supports outputting the token in a format that is consumable by terraform as an external data source when you pass in the

--as-tf-data

kubernetes

# NOTE: Terraform does not allow you to interpolate resources in a provider config. We work around this by using the
# template_file data source as a means to compute the resource interpolations.
provider "kubernetes" {
  load_config_file       = false
  host                   = "${data.template_file.kubernetes_cluster_endpoint.rendered}"
  cluster_ca_certificate = "${base64decode(data.template_file.kubernetes_cluster_ca.rendered)}"
  token                  = "${lookup(data.external.kubernetes_token.result, "token_data")}"
}

data "template_file" "kubernetes_cluster_endpoint" {
  template = "${module.eks_cluster.eks_cluster_endpoint}"
}
data "template_file" "kubernetes_cluster_ca" {
  template = "${module.eks_cluster.eks_cluster_certificate_authority}"
}
data "external" "kubernetes_token" {
  program = ["kubergrunt", "--loglevel", "error", "eks", "token", "--as-tf-data", "--cluster-id", "${module.eks_cluster.eks_cluster_name}"]
}

This will configure the

kubernetes

Similar Commands:

• AWS CLI (

  aws eks get-token

  ): This command will do the same thing, but does not provide any specific optimizations for terraform.
• Terraform

  aws_eks_cluster_auth

  data source: This data source can be used to retrieve a temporary auth token for EKS in Terraform. This can only be used in Terraform.

• aws-iam-authenticator

  : This is a standalone binary that can be used to fetch a temporary auth token.

oidc-thumbprint

This subcommand will take the EKS OIDC Issuer URL and retrieve the root CA thumbprint. This is used to set the trust relation for any certificates signed by that CA for the issuer domain. This is necessary to setup the OIDC provider, which is used for the IAM Roles for Service Accounts feature of EKS.

You can read more about the general procedure for retrieving the root CA thumbprint of an OIDC Provider in the official documentation.

To retrieve the thumbprint, call the command with the issuer URL:

kubergrunt eks oidc-thumbprint --issuer-url $ISSUER_URL

This will output the thumbprint to stdout in JSON format, with the key

thumbprint

Run

kubergrunt eks oidc-thumbprint --help

Similar Commands:

• You can use

  openssl

  to retrieve the thumbprint as described by the official documentation.

• eksctl

  provides routines for directly configuring the OIDC provider so you don't need to retrieve the thumbprint.

deploy

This subcommand will initiate a rolling deployment of the current AMI config to the EC2 instances in your EKS cluster. This command will not deploy or update an application deployed on your Kubernetes cluster (e.g

Deployment

Pod

helm-kubernetes-services

Terraform and AWS do not provide a way to automatically roll out a change to the Instances in an EKS Cluster. Due to Terraform limitations (see here for a discussion), there is currently no way to implement this purely in Terraform code. Therefore, we've created this subcommand that can do a zero-downtime roll out for you.

To deploy a change (such as rolling out a new AMI) to all EKS workers using this command:

1. Make sure the

   cluster_max_size

   is at least twice the size of

   cluster_min_size

   . The extra capacity will be used to deploy the updated instances.
2. Update the Terraform code with your changes (e.g. update the

   cluster_instance_ami

   variable to a new AMI).
3. Run

   terraform apply

   .
4. Run the command:

kubergrunt eks deploy --region REGION --asg-name ASG_NAME

When you call the command, it will:

1. Double the desired capacity of the Auto Scaling Group that powers the EKS Cluster. This will launch new EKS workers with the new launch configuration.
2. Wait for the new nodes to be ready for Pod scheduling in Kubernetes. This includes waiting for the new nodes to be registered to any external load balancers managed by Kubernetes.
3. Cordon the old instances in the ASG so that they won't schedule new Pods.
4. Drain the pods scheduled on the old EKS workers (using the equivalent of

   kubectl drain

   ), so that they will be rescheduled on the new EKS workers.
5. Wait for all the pods to migrate off of the old EKS workers.
6. Set the desired capacity down to the original value and remove the old EKS workers from the ASG.

Note that to minimize service disruption from this command, your services should setup a PodDisruptionBudget, a readiness probe that fails on container shutdown events, and implement graceful handling of SIGTERM in the container. You can learn more about these features in our blog post series covering them.

Currently

kubergrunt

sync-core-components

This subcommand will sync the core components of an EKS cluster to match the deployed Kubernetes version by following the steps listed in the official documentation.

The core components managed by this command are:

• kube-proxy
• Amazon VPC CNI plug-in
• CoreDNS

By default, this command will rotate the images without waiting for the Pods to be redeployed. You can use the

--wait

Example:

kubergrunt eks sync-core-components --eks-cluster-arn EKS_CLUSTER_ARN

cleanup-security-group

This subcommand cleans up the leftover AWS-managed security groups that are associated with an EKS cluster you intend to destroy. It accepts -

--eks-cluster-arn

--security-group-id

--vpc-id

It also looks for other security groups associated with the EKS cluster, such as the security group created by the AWS Load Balancer Controller. To safely delete these resources, it detaches and deletes any associated AWS Elastic Network Interfaces.

Example:

kubergrunt eks cleanup-security-group --eks-cluster-arn EKS_CLUSTER_ARN --security-group-id SECURITY_GROUP_ID \
--vpc-id VPC_ID

schedule-coredns

This subcommand can be used to toggle the CoreDNS service between scheduling on Fargate and EC2 worker types. During the creation of an EKS cluster that uses Fargate,

schedule-coredns fargate

schedule-coredns ec2

This command is useful when creating Fargate only EKS clusters. By default, EKS will schedule the

coredns

You can also use this command in

local-exec

aws_eks_fargate_profile

Currently

fargate

ec2

schedule-coredns

Examples:

kubergrunt eks schedule-coredns fargate --eks-cluster-name EKS_CLUSTER_NAME --fargate-profile-arn FARGATE_PROFILE_ARN

kubergrunt eks schedule-coredns ec2 --eks-cluster-name EKS_CLUSTER_NAME --fargate-profile-arn FARGATE_PROFILE_ARN

drain

This subcommand can be used to drain Pods from the instances in the provided Auto Scaling Groups. This can be used to gracefully retire existing Auto Scaling Groups by ensuring the Pods are evicted in a manner that respects disruption budgets.

You can read more about the drain operation in the official documentation.

To drain the Auto Scaling Group

my-asg

us-east-2

kubergrunt eks drain --asg-name my-asg --region us-east-2

You can drain multiple ASGs by providing the

--name

kubergrunt eks drain --asg-name my-asg-a --name my-asg-b --name my-asg-c --region us-east-2

k8s

The

k8s

kubergrunt

wait-for-ingress

This subcommand waits for the Ingress endpoint to be provisioned. This will monitor the Ingress resource, continuously checking until the endpoint is allocated to the Ingress resource or times out. By default, this will try for 5 minutes (max retries 60 and time betweeen sleep of 5 seconds).

You can configure the timeout settings using the --max-retries and --sleep-between-retries CLI args. This will check for --max-retries times, sleeping for --sleep-between-retries inbetween tries.

For example, if you ran the command:

kubergrunt k8s wait-for-ingress \
    --ingress-name $INGRESS_NAME \
    --namespace $NAMESPACE \
    --max-retries 10 \
    --sleep-between-retries 15s

this command will query the Kubernetes API to check the

Ingress

Run

kubergrunt k8s wait-for-ingress --help

kubectl

This subcommand will call out to kubectl with a temporary file that acts as the kubeconfig, set up with the parameters

--kubectl-server-endpoint

--kubectl-certificate-authority

--kubectl-token

To forward args to kubectl, pass all the args you wish to forward after a

--

kubectl get pods -n kube-system

kubergrunt k8s kubectl \
  --kubectl-server-endpoint $SERVER_ENDPOINT \
  --kubectl-certificate-authority $SERVER_CA \
  --kubectl-token $TOKEN \
  -- get pods -n kube-system

Run

kubergrunt k8s kubectl --help

tls

The

tls

kubergrunt

gen

This subcommand will generate new TLS certificate key pairs based on the provided configuration arguments. Once the certificates are generated, they will be stored on your targeted Kubernetes cluster as Secrets. This supports features such as:

• Generating a new CA key pair and storing the generated key pair in your Kubernetes cluster.
• Issuing a new signed TLS certificate key pair using an existing CA stored in your Kubernetes cluster.
• Replacing the stored certificate key pair in your Kubernetes cluster with a newly generated one.
• Controlling which Namespace the Secrets are stored in.

For example, to generate a new CA key pair, issue a TLS certificate key pair, storing each of those as the Secrets

ca-keypair

tls-keypair

# Generate the CA key pair
kubergrunt tls gen \
    --namespace kube-system \
    --secret-name ca-keypair \
    --ca \
    --tls-common-name kiam-ca \
    --tls-org Gruntwork \
    --tls-org-unit IT \
    --tls-city Phoenix \
    --tls-state AZ \
    --tls-country US \
    --secret-annotation "gruntwork.io/version=v1"
# Generate a signed TLS key pair using the previously created CA
kubergrunt tls gen \
    --namespace kube-system \
    --secret-name tls-keypair \
    --ca-secret-name ca-keypair \
    --tls-common-name kiam-server \
    --tls-org Gruntwork \
    --tls-org-unit IT \
    --tls-city Phoenix \
    --tls-state AZ \
    --tls-country US \
    --secret-annotation "gruntwork.io/version=v1"

The first command will generate a CA key pair and store it as the Secret

ca-keypair

--ca

kubergrunt

The second command uses the generated CA key pair to issue a new TLS key pair. The

--ca-secret-name

kubergrunt

ca-keypair

This command should be run by a cluster administrator to ensure access to the Secrets are tightly controlled.

See the command help for all the available options:

kubergrunt tls gen --help

Deprecated commands

helm

The

helm

kubergrunt

v0.6.0

Who maintains this project?

kubergrunt

Gruntwork can help with:

• Setup, customization, and support for this project.
• Modules and submodules for other types of infrastructure in major cloud providers, such as VPCs, Docker clusters, databases, and continuous integration.
• Modules and Submodules that meet compliance requirements, such as HIPAA.
• Consulting & Training on AWS, GCP, Terraform, and DevOps.

How do I contribute?

Contributions are very welcome! Check out the Contribution Guidelines for instructions.

How is this project versioned?

This project follows the principles of Semantic Versioning. You can find each new release, along with the changelog, in the Releases Page.

During initial development, the major version will be 0 (e.g.,

0.x.y

1.0.0

License

Please see LICENSE and NOTICE for how the code in this repo is licensed.

Copyright © 2020 Gruntwork, Inc.