Building a Kubernetes operator using operator-sdk (golang)
7 min read | by Jordi Prats
With the operator pattern we are going to define the state we want it to be an let the controller do changes on it so the current state matches the desired state
Using the operator-sdk we can focus on the code that makes this happen rather than on all the boiler-plate that is going to be involved on it
To start a operator-sdk project we are going to need some prerequisites:
Prerequisites
We can check the operator-sdk documentation on it's installation for go, but in a nutshell we'll need:
- Go version 1.18 or later:
- A test cluster, such as minikube
Install operator-sdk
To be able to create all the boiler-plate we'll need to install the operator-sdk cli:
export ARCH=$(case $(uname -m) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; *) echo -n $(uname -m) ;; esac)
export OS=$(uname | awk '{print tolower($0)}')
export OPERATOR_SDK_DL_URL=https://github.com/operator-framework/operator-sdk/releases/download/v1.22.0
curl -LO ${OPERATOR_SDK_DL_URL}/operator-sdk_${OS}_${ARCH}
chmod +x operator-sdk_${OS}_${ARCH} && sudo mv operator-sdk_${OS}_${ARCH} /usr/local/bin/operator-sdk
Init project
We are going to use the pet2cattle/operatorsdk-example repository as an example:
$ git clone git@github.com:pet2cattle/operatorsdk-example.git
Cloning into 'operatorsdk-example'...
warning: You appear to have cloned an empty repository.
$ cd operatorsdk-example
Then we'll be able to initialize it using operator-sdk init:
$ operator-sdk init --domain pet2cattle.com --repo github.com/pet2cattle/operatorsdk-example
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.12.1
go: downloading golang.org/x/time v0.0.0-20220210224613-90d013bbcef8
go: downloading golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
go: downloading golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
Update dependencies:
$ go mod tidy
go: downloading github.com/Azure/go-autorest/autorest v0.11.18
go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.13
go: downloading cloud.google.com/go v0.81.0
go: downloading github.com/benbjohnson/clock v1.1.0
go: downloading github.com/Azure/go-autorest v14.2.0+incompatible
go: downloading golang.org/x/crypto v0.0.0-20220214200702-86341886e292
go: downloading github.com/Azure/go-autorest/logger v0.2.1
go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.1
go: downloading github.com/Azure/go-autorest/tracing v0.6.0
go: downloading github.com/form3tech-oss/jwt-go v3.2.3+incompatible
go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0
Next: define a resource with:
$ operator-sdk create api
We can check commit 2822dd5 to check all the files that it creates
Create API
Then, we can use operator-sdk create api to create the CRD resources that we are going to use to define the desired state of out application. For a namespaced object we can use:
$ operator-sdk create api --group example --version v1 --kind Demo --resource --controller
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/demo_types.go
controllers/demo_controller.go
Update dependencies:
$ go mod tidy
go: downloading github.com/onsi/ginkgo/v2 v2.0.0
Running make:
$ make generate
mkdir -p /home/pet2cattle/git/operatorsdk-example/bin
GOBIN=/home/pet2cattle/git/operatorsdk-example/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.9.0
go: downloading sigs.k8s.io/controller-tools v0.9.0
go: downloading github.com/spf13/cobra v1.4.0
go: downloading golang.org/x/tools v0.1.10-0.20220218145154-897bd77cd717
go: downloading github.com/fatih/color v1.12.0
go: downloading github.com/gobuffalo/flect v0.2.5
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/mattn/go-colorable v0.1.8
go: downloading golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
Is we want to create a non-namespaced resource we'll need to add the --namespaced=false flag:
$ operator-sdk create api \
--group example \
--version v1 \
--kind Demo \
--resource \
--controller \
--namespaced=false
We can also check commit b7eeb09 to see what this is going under the hood for a namespaced object.
Define schema
At this point we'll need to start modifying the files that the operator-sdk cli have created.
First, we'll need to update the api/v1/demo_types.go file with whatever we want to to contain. For this example we are going to use the attribute name as an example:
// DemoSpec defines the desired state of Demo
type DemoSpec struct {
Name string `json:"name"`
}
Then, we can update the sample instance using this file: config/samples/example_v1_expiration.yaml:
apiVersion: example.pet2cattle.com/v1
kind: Demo
metadata:
name: demo-sample
spec:
example: 'this-is-an-example'
See the change on github: commit 0cc82f7
Installing the CRD
Using make install we can install the CRD to the active cluster (we are using minikube for this example)
$ make install
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -s -- 3.8.7 /home/pet2cattle/git/operatorsdk-example/bin
{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:14:14Z GoOs:linux GoArch:amd64}
kustomize installed to /home/pet2cattle/git/operatorsdk-example/bin/kustomize
/home/pet2cattle/git/operatorsdk-example/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/demoes.example.pet2cattle.com created
Deploying a CRD instance
We can use kustomize to apply it for more complex projects:
$ kustomize build config/samples | kubectl apply -f -
demo.example.pet2cattle.com/demo-sample created
But for now, kubectl apply would work just as well:
$ kubectl apply -f config/samples/example_v1_expiration.yaml
demo.example.pet2cattle.com/demo-sample created
Spin up controller
Using make run we can spin up the controller, but at this stage it's just a placeholder: It's not going to do much:
$ make run
GOBIN=/home/pet2cattle/git/operatorsdk-example/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.9.0
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
1.6556603319415438e+09 INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
1.6556603319434564e+09 INFO setup starting manager
1.6556603319438052e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.655660331943896e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6556603319438577e+09 INFO Starting EventSource {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo", "source": "kind source: *v1.Demo"}
1.6556603319439232e+09 INFO Starting Controller {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo"}
1.6556603320442946e+09 INFO Starting workers {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo", "worker count": 1}
(...)
Check resource
We can check the CRD using the usual kubectl get demo
$ kubectl get demo
NAME AGE
demo-sample 88s
Or even describe it:
$ kubectl describe demo demo-sample
Name: demo-sample
Namespace: default
Labels: <none>
Annotations: <none>
API Version: example.pet2cattle.com/v1
Kind: Demo
Metadata:
Creation Timestamp: 2022-06-19T19:37:51Z
Generation: 1
Managed Fields:
API Version: example.pet2cattle.com/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:example:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-06-19T17:37:51Z
Resource Version: 552
UID: 67d2f0ed-d1d0-46a5-930a-70186717cfc6
Spec:
Example: this-is-an-example
Events: <none>
Implement reconcile loop
If we want to actually implement something we'll have to update the Reconcile function on the controllers/expiration_controller.go file.
For this example we are just going to print some data to stdout:
func (r *ExpirationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
// Fetch the Expiration instance.
expiration := &bestbyv1.Expiration{}
if err := r.Get(ctx, req.NamespacedName, expiration); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
fmt.Printf("%#v\n", expiration)
fmt.Printf("%#v\n", expiration.Spec.Lifetime)
return ctrl.Result{}, nil
}
For this to work we'll have to import fmt, on the commit 0fbb9b7 you can see all the details.
By running the controller again with make run, we will be able to see how it is printing this information to stdout:
$ make run
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/pet2cattle/git/operatorsdk-example/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
1.6556605118827746e+09 INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
1.6556605118832474e+09 INFO setup starting manager
1.655660511883873e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.6556605118856435e+09 INFO Starting EventSource {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo", "source": "kind source: *v1.Demo"}
1.6556605118860493e+09 INFO Starting Controller {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo"}
1.6556605118906796e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6556605119876258e+09 INFO Starting workers {"controller": "demo", "controllerGroup": "example.pet2cattle.com", "controllerKind": "Demo", "worker count": 1}
&v1.Demo{TypeMeta:v1.TypeMeta{Kind:"Demo", APIVersion:"example.pet2cattle.com/v1"}, ObjectMeta:v1.ObjectMeta{Name:"demo-sample", GenerateName:"", Namespace:"default", SelfLink:"", UID:"67d2f0ed-d1d0-46a5-930a-70186717cfc6", ResourceVersion:"552", Generation:1, CreationTimestamp:time.Date(2022, time.June, 19, 19, 37, 51, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"example.pet2cattle.com/v1\",\"kind\":\"Demo\",\"metadata\":{\"annotations\":{},\"name\":\"demo-sample\",\"namespace\":\"default\"},\"spec\":{\"example\":\"this-is-an-example\"}}\n"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubectl-client-side-apply", Operation:"Update", APIVersion:"example.pet2cattle.com/v1", Time:time.Date(2022, time.June, 19, 19, 37, 51, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00043a348), Subresource:""}}}, Spec:v1.DemoSpec{Example:"this-is-an-example"}, Status:v1.DemoStatus{}}
"this-is-an-example"
Posted on 21/06/2022