Building a Kubernetes operator using operator-sdk (golang)

Kubernetes operator-sdk

7 min read | by Jordi Prats

With the operator pattern we are going to define the state we want it to be an let the controller do changes on it so the current state matches the desired state

Using the operator-sdk we can focus on the code that makes this happen rather than on all the boiler-plate that is going to be involved on it

To start a operator-sdk project we are going to need some prerequisites:

Prerequisites

We can check the operator-sdk documentation on it's installation for go, but in a nutshell we'll need:

  • Go version 1.18 or later:
  • A test cluster, such as minikube

Install operator-sdk

To be able to create all the boiler-plate we'll need to install the operator-sdk cli:

export ARCH=$(case $(uname -m) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; *) echo -n $(uname -m) ;; esac)
export OS=$(uname | awk '{print tolower($0)}')
export OPERATOR_SDK_DL_URL=https://github.com/operator-framework/operator-sdk/releases/download/v1.22.0
curl -LO ${OPERATOR_SDK_DL_URL}/operator-sdk_${OS}_${ARCH}
chmod +x operator-sdk_${OS}_${ARCH} && sudo mv operator-sdk_${OS}_${ARCH} /usr/local/bin/operator-sdk

Init project

We are going to use the pet2cattle/operatorsdk-example repository as an example:

$ git clone git@github.com:pet2cattle/operatorsdk-example.git
Cloning into 'operatorsdk-example'...
warning: You appear to have cloned an empty repository.
$ cd operatorsdk-example

Then we'll be able to initialize it using operator-sdk init:

$ operator-sdk init --domain pet2cattle.com --repo github.com/pet2cattle/operatorsdk-example
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.12.1
go: downloading golang.org/x/time v0.0.0-20220210224613-90d013bbcef8
go: downloading golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
go: downloading golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
Update dependencies:
$ go mod tidy
go: downloading github.com/Azure/go-autorest/autorest v0.11.18
go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.13
go: downloading cloud.google.com/go v0.81.0
go: downloading github.com/benbjohnson/clock v1.1.0
go: downloading github.com/Azure/go-autorest v14.2.0+incompatible
go: downloading golang.org/x/crypto v0.0.0-20220214200702-86341886e292
go: downloading github.com/Azure/go-autorest/logger v0.2.1
go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.1
go: downloading github.com/Azure/go-autorest/tracing v0.6.0
go: downloading github.com/form3tech-oss/jwt-go v3.2.3+incompatible
go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0
Next: define a resource with:
$ operator-sdk create api

We can check commit 2822dd5 to check all the files that it creates

Create API

Then, we can use operator-sdk create api to create the CRD resources that we are going to use to define the desired state of out application. For a namespaced object we can use:

$ operator-sdk create api --group example --version v1 --kind Demo --resource --controller
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/demo_types.go
controllers/demo_controller.go
Update dependencies:
$ go mod tidy
go: downloading github.com/onsi/ginkgo/v2 v2.0.0
Running make:
$ make generate
mkdir -p /home/pet2cattle/git/operatorsdk-example/bin
GOBIN=/home/pet2cattle/git/operatorsdk-example/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.9.0
go: downloading sigs.k8s.io/controller-tools v0.9.0
go: downloading github.com/spf13/cobra v1.4.0
go: downloading golang.org/x/tools v0.1.10-0.20220218145154-897bd77cd717
go: downloading github.com/fatih/color v1.12.0
go: downloading github.com/gobuffalo/flect v0.2.5
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/mattn/go-colorable v0.1.8
go: downloading golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

Is we want to create a non-namespaced resource we'll need to add the --namespaced=false flag:

$ operator-sdk create api \
                          --group example \
                          --version v1 \
                          --kind Demo \
                          --resource \
                          --controller \
                          --namespaced=false

We can also check commit b7eeb09 to see what this is going under the hood for a namespaced object.

Define schema

At this point we'll need to start modifying the files that the operator-sdk cli have created.

First, we'll need to update the api/v1/demo_types.go file with whatever we want to to contain. For this example we are going to use the attribute name as an example:

// DemoSpec defines the desired state of Demo
type DemoSpec struct {
  Name string `json:"name"`
}

Then, we can update the sample instance using this file: config/samples/example_v1_expiration.yaml:

apiVersion: example.pet2cattle.com/v1
kind: Demo
metadata:
  name: demo-sample
spec:
  example: 'this-is-an-example'

See the change on github: commit 0cc82f7

Installing the CRD

Using make install we can install the CRD to the active cluster (we are using minikube for this example)

$ make install
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -s -- 3.8.7 /home/pet2cattle/git/operatorsdk-example/bin
{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:14:14Z GoOs:linux GoArch:amd64}
kustomize installed to /home/pet2cattle/git/operatorsdk-example/bin/kustomize
/home/pet2cattle/git/operatorsdk-example/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/demoes.example.pet2cattle.com created

Deploying a CRD instance

We can use kustomize to apply it for more complex projects:

$ kustomize build config/samples | kubectl apply -f -
demo.example.pet2cattle.com/demo-sample created

But for now, kubectl apply would work just as well:

$ kubectl apply -f config/samples/example_v1_expiration.yaml
demo.example.pet2cattle.com/demo-sample created

Spin up controller

Using make run we can spin up the controller, but at this stage it's just a placeholder: It's not going to do much:

$ make run
GOBIN=/home/pet2cattle/git/operatorsdk-example/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.9.0
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
1.6556603319415438e+09  INFO  controller-runtime.metrics  Metrics server is starting to listen  {"addr": ":8080"}
1.6556603319434564e+09  INFO  setup starting manager
1.6556603319438052e+09  INFO  Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.655660331943896e+09 INFO  Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6556603319438577e+09  INFO  Starting EventSource  {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo", "source": "kind source: *v1.Demo"}
1.6556603319439232e+09  INFO  Starting Controller {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo"}
1.6556603320442946e+09  INFO  Starting workers  {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo", "worker count": 1}
(...)

Check resource

We can check the CRD using the usual kubectl get demo

$ kubectl get demo
NAME          AGE
demo-sample   88s

Or even describe it:

$ kubectl describe demo demo-sample
Name:         demo-sample
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  example.pet2cattle.com/v1
Kind:         Demo
Metadata:
  Creation Timestamp:  2022-06-19T19:37:51Z
  Generation:          1
  Managed Fields:
    API Version:  example.pet2cattle.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:example:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2022-06-19T17:37:51Z
  Resource Version:  552
  UID:               67d2f0ed-d1d0-46a5-930a-70186717cfc6
Spec:
  Example:  this-is-an-example
Events:     <none>

Implement reconcile loop

If we want to actually implement something we'll have to update the Reconcile function on the controllers/expiration_controller.go file.

For this example we are just going to print some data to stdout:

func (r *ExpirationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  _ = log.FromContext(ctx)

  // Fetch the Expiration instance.
  expiration := &bestbyv1.Expiration{}
  if err := r.Get(ctx, req.NamespacedName, expiration); err != nil {
    return ctrl.Result{}, client.IgnoreNotFound(err)
  }

  fmt.Printf("%#v\n", expiration)

  fmt.Printf("%#v\n", expiration.Spec.Lifetime)

  return ctrl.Result{}, nil
}

For this to work we'll have to import fmt, on the commit 0fbb9b7 you can see all the details.

By running the controller again with make run, we will be able to see how it is printing this information to stdout:

$ make run
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
1.6556605118827746e+09  INFO  controller-runtime.metrics  Metrics server is starting to listen  {"addr": ":8080"}
1.6556605118832474e+09  INFO  setup starting manager
1.655660511883873e+09 INFO  Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.6556605118856435e+09  INFO  Starting EventSource  {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo", "source": "kind source: *v1.Demo"}
1.6556605118860493e+09  INFO  Starting Controller {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo"}
1.6556605118906796e+09  INFO  Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6556605119876258e+09  INFO  Starting workers  {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo", "worker count": 1}
&v1.Demo{TypeMeta:v1.TypeMeta{Kind:"Demo", APIVersion:"example.pet2cattle.com/v1"}, ObjectMeta:v1.ObjectMeta{Name:"demo-sample", GenerateName:"", Namespace:"default", SelfLink:"", UID:"67d2f0ed-d1d0-46a5-930a-70186717cfc6", ResourceVersion:"552", Generation:1, CreationTimestamp:time.Date(2022, time.June, 19, 19, 37, 51, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"example.pet2cattle.com/v1\",\"kind\":\"Demo\",\"metadata\":{\"annotations\":{},\"name\":\"demo-sample\",\"namespace\":\"default\"},\"spec\":{\"example\":\"this-is-an-example\"}}\n"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubectl-client-side-apply", Operation:"Update", APIVersion:"example.pet2cattle.com/v1", Time:time.Date(2022, time.June, 19, 19, 37, 51, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00043a348), Subresource:""}}}, Spec:v1.DemoSpec{Example:"this-is-an-example"}, Status:v1.DemoStatus{}}
"this-is-an-example"

Posted on 21/06/2022