Autobucket Operator

Building a Cloud Storage Kubernetes Operator with Go and Operator SDK

In the last article, we looked at Mutating Admission Webhooks as a way to extend Kubernetes. In this article we’ll explore another concept: Kubernetes Operators.

Kubernetes Operators

The Kubernetes docs defines operators as:

Autobucket Operator

In this article we’ll build “Autobucket Operator”, a Kubernetes operator that automatically manages Cloud Object Storage (like GCP Cloud Storage Buckets or S3 Buckets) for a Kubernetes Deployment. Here is a schema that represents the general idea:

Autobucket Operator Summary

Let’s code

The companion repo for this article is available on github, so you can follow along.

Sample Deployment

The Reconcile Loop

The operator controller watches the Deployments and whenever it finds a Deployment with the special annotation “ab.leclouddev.com/cloud”, it will create (if missing) a matching Bucket CR. Luckily kubebuilder and controller-runtime do the heavy lifting for us here and we basically just have to define our deployment’s “Reconcile Loop”, which checks deployments and reconciles the Bucket resources:

How about testing ?

As a big fan of automated testing, it’s comforting to learn that we can easily write tests for our controllers using envtest (which runs a local k8s control plane so we can run our tests against it), the Gingko testing framework and the Gomega matching/assertion library:

Does it work ?

Let’s try our operator in a little demo. For this, I have created another github repository bucket-text-api which is a simple Go REST API that takes a JSON input and saves text to a Cloud Storage bucket.

# install the k8s resources
$ make install
# deploy the controller manager
$ GCP_PROJECT=autobucket-demo make deploy
$ kubectl apply -f config/samples/deployment.yaml
$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
bucket-text-api 2/2 2 2 1m
$ kubectl get bucket
NAME CLOUD FULLNAME CREATEDAT
bucket-text-api gcp ab-default-bucket-text-api 2020-11-19T15:31:34Z
$ gsutil ls -L -p autobucket-demo
gs://ab-default-bucket-text-api/ :
Storage class: STANDARD
Location type: multi-region
Location constraint: US
Versioning enabled: None
Logging configuration: None
Website configuration: None
CORS configuration: None
Lifecycle configuration: None
Requester Pays enabled: None
Labels: None
Default KMS key: None
Time created: Thu, 19 Nov 2020 15:31:34 GMT
Time updated: Thu, 19 Nov 2020 15:31:34 GMT
$ curl --request POST \
--url http://<kubernetes-node-ip>:30008/save \
--header 'Content-Type: application/json' \
--data '{
"name": "test.txt",
"content": "hello operator !"
}'
$ gsutil cat gs://ab-default-bucket-text-api/test.txt
hello operator !

CR Deletion and Finalizers

One last bit I didn’t mention yet is what happens when a Deployment or Bucket CR is deleted.

$ kubectl delete deployment bucket-text-api
deployment.apps "bucket-text-api" deleted
$ kubectl get bucket
No resources found in default namespace.
$ gsutil ls -p autobucket-demo gs://ab-default-bucket-text-apiBucketNotFoundException: 404 gs://ab-default-bucket-text-api bucket does not exist.

Conclusion

We have seen in this example how Kubernetes Operators can allow us to automate cloud infrastructure logic. But there are of course a lot more uses to operators, and you can check this list of operators in the wild.

Backend & Cloud Engineer (Golang)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store