An end-to-end scenario showing how to use the Operator Framework.
This project is deprecated. Documentation for the Operator Framework has moved to https://operatorframework.io/
The Operator Framework (intro blog post) is an open source toolkit to manage Kubernetes native applications, called operators, in an effective, automated, and scalable way. Operators take advantage of Kubernetes's extensibility to deliver the automation advantages of cloud services like provisioning, scaling, and backup/restore while being able to run anywhere that Kubernetes can run.
This guide shows how to build a simple memcached operator and how to manage its lifecycle from install to update to a new version. For that, we will use two center pieces of the framework:
BEFORE YOU BEGIN: links to the Operator SDK repo in this document are pinned to the
masterbranch. Make sure you update the link such that it points to the correct Operator SDK repo version, which should match this repo's version or the
operator-sdk versionbeing used. For example, if you are using
operator-sdkv0.12.0, update all links from this repo to the SDK repo with
master -> v0.12.0. Otherwise you may see incorrect information.
The Operator SDK makes it easier to build Kubernetes native applications, a process that can require deep, application-specific operational knowledge. The SDK not only lowers that barrier, but it also helps reduce the amount of boilerplate code needed for many common management capabilities, such as metering or monitoring.
This section walks through an example of building a simple memcached operator using tools and libraries provided by the Operator SDK. This walkthrough is not exhaustive; for an in-depth explanation of these steps, see the SDK's user guide.
Requirements: Please make sure that the Operator SDK is installed on the development machine. Additionally, the Operator Lifecycle Manager must be installed in the cluster (1.8 or above to support the apps/v1beta2 API group) before running this guide.
memcached-operatorproject:
$ mkdir -p $GOPATH/src/github.com/example-inc/ $ cd $GOPATH/src/github.com/example-inc/ $ export GO111MODULE=on $ operator-sdk new memcached-operator $ cd memcached-operator
This creates the
memcached-operatorproject.
go mod tidy
NOTE: Learn more about the project directory structure from the SDK project layout documentation.
The main program for the operator
cmd/manager/main.goinitializes and runs the Manager.
The Manager will automatically register the scheme for all custom resources defined under
pkg/apis/...and run all controllers under
pkg/controller/....
The Manager can restrict the namespace that all controllers will watch for resources:
mgr, err := manager.New(cfg, manager.Options{ Namespace: namespace, })
By default this will be the namespace that the operator is running in. To watch all namespaces leave the namespace option empty:
mgr, err := manager.New(cfg, manager.Options{ Namespace: "", })
Add a new Custom Resource Definition (CRD) API called
Memcached, with APIVersion
cache.example.com/v1alpha1and Kind
Memcached.
$ operator-sdk add api --api-version=cache.example.com/v1alpha1 --kind=Memcached
This will scaffold the
Memcachedresource API under
pkg/apis/cache/v1alpha1/....
Modify the spec and status of the
MemcachedCustom Resource (CR) at
pkg/apis/cache/v1alpha1/memcached_types.go:
type MemcachedSpec struct { // Size is the size of the memcached deployment Size int32 `json:"size"` } type MemcachedStatus struct { // Nodes are the names of the memcached pods Nodes []string `json:"nodes"` }
After modifying the
*_types.gofile always run the following command to update the generated code for that resource type:
$ operator-sdk generate k8s
Also run the following command in order to automatically generate the CRDs:
$ operator-sdk generate crds
You can see the changes applied in
deploy/crds/cache.example.com_memcacheds_crd.yaml
Add a new Controller to the project that will watch and reconcile the
Memcachedresource:
$ operator-sdk add controller --api-version=cache.example.com/v1alpha1 --kind=Memcached
This will scaffold a new Controller implementation under
pkg/controller/memcached/....
For this example replace the generated Controller file
pkg/controller/memcached/memcached_controller.gowith the example
memcached_controller.goimplementation.
The example Controller executes the following reconciliation logic for each
MemcachedCR:
MemcachedCR spec
MemcachedCR status with the names of the memcached pods
The next two subsections explain how the Controller watches resources and how the reconcile loop is triggered. Skip to the Build section to see how to build and run the operator.
Inspect the Controller implementation at
pkg/controller/memcached/memcached_controller.goto see how the Controller watches resources.
The first watch is for the
Memcachedtype as the primary resource. For each Add/Update/Delete event the reconcile loop will be sent a reconcile
Request(a namespace/name key) for that
Memcachedobject:
err := c.Watch( &source.Kind{Type: &cachev1alpha1.Memcached{}}, &handler.EnqueueRequestForObject{}, )
The next watch is for Deployments but the event handler will map each event to a reconcile
Requestfor the owner of the Deployment. Which in this case is the
Memcachedobject for which the Deployment was created. This allows the controller to watch Deployments as a secondary resource.
err := c.Watch( &source.Kind{Type: &appsv1.Deployment{}}, &handler.EnqueueRequestForOwner{ IsController: true, OwnerType: &cachev1alpha1.Memcached{}}, )
Every Controller has a Reconciler object with a
Reconcile()method that implements the reconcile loop. The reconcile loop is passed the
Requestargument which is a Namespace/Name key used to lookup the primary resource object,
Memcached, from the cache:
func (r *ReconcileMemcached) Reconcile(request reconcile.Request) (reconcile.Result, error) { // Lookup the Memcached instance for this reconcile request memcached := &cachev1alpha1.Memcached{} err := r.client.Get(context.TODO(), request.NamespacedName, memcached) ... }
For a guide on Reconcilers, Clients, and interacting with resource Events, see the Client API doc.
Before running the operator, the CRD must be registered with the Kubernetes apiserver:
$ kubectl create -f deploy/crds/cache.example.com_memcacheds_crd.yaml
Once this is done, there are two ways to run the operator:
Build the memcached-operator image and push it to your registry. The following example uses https://quay.io as the registry.
$ operator-sdk build quay.io//memcached-operator:v0.0.1 $ sed -i 's|REPLACE_IMAGE|quay.io//memcached-operator:v0.0.1|g' deploy/operator.yaml $ docker push quay.io//memcached-operator:v0.0.1
Note If you are performing these steps on OSX, use the following
sedcommand instead:
$ sed -i "" 's|REPLACE_IMAGE|quay.io//memcached-operator:v0.0.1|g' deploy/operator.yaml
The above command will replace the string
REPLACE_IMAGEwith the
built above. Afterwards, verify that your:
operator.yamlfile was updated successfully.
serviceAccountName: memcached-operator containers: - name: memcached-operator # Replace this with the built image name image: quay.io//memcached-operator:v0.0.1 command: - memcached-operator imagePullPolicy: Always
IMPORTANT: Ensure that your cluster is able to pull the image pushed to your registry.
The Deployment manifest is generated at
deploy/operator.yaml. Be sure to update the deployment image as shown above since the default is just a placeholder.
Setup RBAC and deploy the memcached-operator:
$ kubectl create -f deploy/service_account.yaml $ kubectl create -f deploy/role.yaml $ kubectl create -f deploy/role_binding.yaml $ kubectl create -f deploy/operator.yaml
NOTE: To apply the RBAC you need to be logged in
system:admin. (E.g. By using for OCP:
oc login -u system:admin.)
Verify that the
memcached-operatorDeployment is up and running:
$ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE memcached-operator 1 1 1 1 1m
Verify that the
memcached-operatorpod is up and running:
$ kubectl get pod NAME READY STATUS RESTARTS AGE memcached-operator-7d76948766-nrcp7 1/1 Running 0 44s
IMPORTANT: Ensure that you built and pushed the image, and updated the
operator.yamlfile.
Verify that the operator is running successfully by checking its logs.
$ kubectl logs memcached-operator-7d76948766-nrcp7 {"level":"info","ts":1580855834.104447,"logger":"cmd","msg":"Operator Version: 0.0.1"} {"level":"info","ts":1580855834.1044931,"logger":"cmd","msg":"Go Version: go1.13.6"} {"level":"info","ts":1580855834.104505,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1580855834.1045163,"logger":"cmd","msg":"Version of operator-sdk: v0.15.1"} {"level":"info","ts":1580855834.1049826,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1580855834.4423697,"logger":"leader","msg":"No pre-existing lock was found."} {"level":"info","ts":1580855834.447401,"logger":"leader","msg":"Became the leader."} {"level":"info","ts":1580855834.7494223,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"} {"level":"info","ts":1580855834.7497423,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1580855835.3955405,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"memcached-operator-metrics","Service.Namespace":"default"} {"level":"info","ts":1580855835.7000446,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"no ServiceMonitor registered with the API"} {"level":"info","ts":1580855835.7005095,"logger":"cmd","msg":"Install prometheus-operator in your cluster to create ServiceMonitor objects","error":"no ServiceMonitor registered with the API"} {"level":"info","ts":1580855835.7007008,"logger":"cmd","msg":"Starting the Cmd."} {"level":"info","ts":1580855835.7014875,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1580855835.702304,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"memcached-controller","source":"kind source: /, Kind="} {"level":"info","ts":1580855835.803201,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"memcached-controller","source":"kind source: /, Kind="} {"level":"info","ts":1580855835.9041016,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"memcached-controller"} {"level":"info","ts":1580855835.9044445,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"memcached-controller","worker count":1}
The following error will occur if your cluster was unable to pull the image:
$ kubectl get pod NAME READY STATUS RESTARTS AGE memcached-operator-6b5dc697fb-t62cv 0/1 ImagePullBackOff 0 2m
Following the logs in the error scenario described above.
$ kubectl logs memcached-operator-6b5dc697fb-t62cv Error from server (BadRequest): container "memcached-operator" in pod "memcached-operator-6b5dc697fb-t62cv" is waiting to start: image can't be pulled
NOTE: Just for tests purposes make the image public and setting up the cluster to allow use insecure registry. ( E.g
--insecure-registry 172.30.0.0/16)
This method is preferred during development cycle to deploy and test faster.
Run the operator locally with the default kubernetes config file present at
$HOME/.kube/config:
$ operator-sdk run local INFO[0000] Running the operator locally; watching namespace "default" {"level":"info","ts":1593777657.892013,"logger":"cmd","msg":"Operator Version: 0.0.1"} {"level":"info","ts":1593777657.892079,"logger":"cmd","msg":"Go Version: go1.14.4"} {"level":"info","ts":1593777657.892084,"logger":"cmd","msg":"Go OS/Arch: darwin/amd64"} {"level":"info","ts":1593777657.892087,"logger":"cmd","msg":"Version of operator-sdk: v0.18.2"} ...
You can use a specific kubeconfig via the flag
--kubeconfig=.
Create the example
MemcachedCR that was generated at
deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml:
$ cat deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml apiVersion: "cache.example.com/v1alpha1" kind: "Memcached" metadata: name: "example-memcached" spec: size: 3$ kubectl apply -f deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml
Ensure that the
memcached-operatorcreates the deployment for the CR:
$ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE memcached-operator 1 1 1 1 2m example-memcached 3 3 3 3 1m
Check the pods and CR status to confirm the status is updated with the memcached pod names:
$ kubectl get pods NAME READY STATUS RESTARTS AGE example-memcached-6fd7c98d8-7dqdr 1/1 Running 0 1m example-memcached-6fd7c98d8-g5k7v 1/1 Running 0 1m example-memcached-6fd7c98d8-m7vn7 1/1 Running 0 1m memcached-operator-7cc7cfdf86-vvjqk 1/1 Running 0 2m
$ kubectl get memcached/example-memcached -o yaml apiVersion: cache.example.com/v1alpha1 kind: Memcached metadata: clusterName: "" creationTimestamp: 2018-03-31T22:51:08Z generation: 0 name: example-memcached namespace: default resourceVersion: "245453" selfLink: /apis/cache.example.com/v1alpha1/namespaces/default/memcacheds/example-memcached uid: 0026cc97-3536-11e8-bd83-0800274106a1 spec: size: 3 status: nodes: - example-memcached-6fd7c98d8-7dqdr - example-memcached-6fd7c98d8-g5k7v - example-memcached-6fd7c98d8-m7vn7
Change the
spec.sizefield in the memcached CR from 3 to 4 and apply the change:
$ cat deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml apiVersion: "cache.example.com/v1alpha1" kind: "Memcached" metadata: name: "example-memcached" spec: size: 4$ kubectl apply -f deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml
Confirm that the operator changes the deployment size:
$ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE example-memcached 4 4 4 4 5m
Delete the operator and its related resources:
$ kubectl delete -f deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml $ kubectl delete -f deploy/operator.yaml $ kubectl delete -f deploy/role_binding.yaml $ kubectl delete -f deploy/role.yaml $ kubectl delete -f deploy/service_account.yaml $ kubectl delete -f deploy/crds/cache.example.com_memcacheds_crd.yaml
The above walkthrough follows a similar implementation process to the one used to produce the
memcached-operatorin the SDK samples repo.
NOTE: This section of the Getting Started Guide is out-of-date. We're working on some improvements to Operator SDK to streamline the experience of using OLM. For further information see, for example, this enhancement proposal. In the meantime, you might find the following documentation helpful:
The previous section has covered manually running an operator. In the next sections, we will explore using the Operator Lifecycle Manager (OLM) which is what enables a more robust deployment model for operators being run in production environments.
OLM helps you to install, update, and generally manage the lifecycle of all of the operators (and their associated services) on a Kubernetes cluster. It runs as an Kubernetes extension and lets you use
kubectlfor all the lifecycle management functions without any additional tools.
NOTE: Various public, OLM-ready operator projects are available at operatorhub.io.
The first step to leveraging OLM is to create a Cluster Service Version (CSV) manifest. An operator manifest describes how to display, create and manage the application, in this case memcached, as a whole. It is required for OLM to function.
The Operator SDK CLI can generate CSV manifests via the following command:
$ operator-sdk generate csv --csv-version 0.0.1 --update-crds
Several fields must be updated after generating the CSV. See the CSV generation doc for a list of required fields, and the memcached-operator CSV for an example of a complete CSV.
NOTE: You are able to preview and validate your CSV manifest syntax in the operatorhub.io CSV Preview tool.
The next step is to ensure your project deploys correctly with OLM and runs as expected. Follow this testing guide to deploy and test your operator.
NOTE: Also, check out some of the new OLM integrations in operator-sdk:
operator-sdk olmto install and manage an OLM installation in your cluster.
operator-sdk run --olmto run your operator using the CSV generated by
operator-sdk generate csv.
operator-sdk bundleto create and validate operator bundle images.
We recommend running
operator-sdk scorecardagainst your operator to see whether your operator's OLM integration follows best practices. For further information on running the scorecard and results, see the scorecard documentation.
NOTE: the scorecard is undergoing changes to give informative and helpful feedback. The original scorecard functionality will still be available while and after changes are made.
Hopefully, this guide was an effective demonstration of the value of the Operator Framework for building and managing operators. There is much more that we left out in the interest of brevity. The Operator Framework and its components are open source, so please feel encouraged to jump into each individually and learn what else you can do. If you want to discuss your experience, have questions, or want to get involved, join the Operator Framework mailing list.