[email protected]

Build & Test a Kubernetes Operator with Go and CircleCI

A hands-on SRE guide. Learn how to scaffold with Kubebuilder, master Server-Side Apply (SSA), automate tests in CircleCI, and maximize IOPS on Bare Metal.

The Cloud Trap & The Operator Revolution

When companies migrate to Kubernetes on public clouds (AWS EKS, GKE), they often fall into a massive financial trap: relying entirely on expensive "Managed Services" (like AWS RDS for PostgreSQL or MSK for Kafka) because stateful applications are notoriously difficult to manage manually.

To escape this cloud tax, elite SRE teams build Kubernetes Operators. By writing a custom Operator in Golang, you are essentially encoding human operational knowledge (how to backup, restore, scale, and heal an application) directly into the cluster. This allows you to host complex stateful workloads on highly cost-effective Dedicated Bare Metal Servers, effectively building your own "Private Managed Cloud."

In this tutorial, we will decode the golang kubernetes controller vs operator paradigm, show you how to scaffold a project with Kubebuilder, securely test it in a test kubernetes operator circleci pipeline, and explain why Bare Metal is the ultimate hardware destination for these workloads.

Step 1: CRDs and Scaffolding with Kubebuilder

To understand how to create custom resource definition kubernetes, we use Kubebuilder, the official framework built by Google to bootstrap Kubernetes APIs.

Operator vs Controller

Every Operator is a Controller, but not every Controller is an Operator. A standard Controller watches built-in resources (like scaling ReplicaSets). An Operator introduces a Custom Resource Definition (CRD)—such as a DatabaseCluster—and injects domain-specific intelligence to manage its entire lifecycle.

Let's initialize our Operator project in Go:

# Initialize the Go module
mkdir my-first-operator && cd my-first-operator
go mod init github.com/yourname/my-first-operator
# Scaffold the Kubebuilder project
kubebuilder init --domain irexta.com --repo github.com/yourname/my-first-operator
# Create the Custom Resource Definition (CRD) and Controller
kubebuilder create api --group apps --version v1alpha1 --kind WebApp

This generates the schema in api/v1alpha1/webapp_types.go. You must modify the Spec to define your desired state, and then run make generate manifests to compile the actual YAML CRDs.

Step 2: Writing the Brain - The Reconciliation Loop

The heart of any Operator is the Reconciliation Loop. It constantly compares the Actual State of the cluster against your Desired State (defined in your CRD), and takes action.

The SRE Masterclass: API Spam & TypeMeta SSA Crashes

1. The API Spam Loop: A Reconciliation loop runs continuously. If you execute a r.Create() blindly, your operator will bombard the API server with requests. You MUST check if the resource already exists using r.Get() first.

2. The Missing TypeMeta SSA Crash: Modern SREs use Server-Side Apply (SSA) to bypass HTTP 409 Concurrency errors. However, there is a massive hidden trap: controller-runtime's r.Get() strips the APIVersion and Kind (TypeMeta) from objects when loading them into memory. If you execute an SSA patch without these, your Operator will crash immediately with a "missing apiVersion or kind" error. You must manually re-inject them!

Here is the production-ready Go logic inside internal/controller/webapp_controller.go, fortified with RBAC markers, Garbage Collection, and the TypeMeta SSA fix:

// +kubebuilder:rbac:groups=apps.irexta.com,resources=webapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps.irexta.com,resources=webapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;create;update;patch;delete
func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { logger := log.FromContext(ctx) var webApp appsv1alpha1.WebApp if err := r.Get(ctx, req.NamespacedName, &webApp); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) } // [Production Best Practice]: Ensure proper Garbage Collection pod := buildPodForWebApp(&webApp) if err := ctrl.SetControllerReference(&webApp, pod, r.Scheme); err != nil { return ctrl.Result{}, err } // [Crucial Step]: Prevent API Server Spam! // Check if Pod exists before attempting creation. foundPod := &corev1.Pod{} err := r.Get(ctx, types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, foundPod) if err != nil && errors.IsNotFound(err) { logger.Info("Creating a new Pod", "Pod.Namespace", pod.Namespace, "Pod.Name", pod.Name) if err := r.Create(ctx, pod); err != nil { logger.Error(err, "Failed to create new Pod") return ctrl.Result{}, err } } else if err != nil { logger.Error(err, "Failed to get Pod") return ctrl.Result{}, err } // [Modern SRE Fix]: True Server-Side Apply (SSA) to bypass HTTP 409 webApp.Status.Phase = "Running" // [CRITICAL FIX]: r.Get() strips TypeMeta, but SSA requires it! // If we don't re-inject it, the Operator will crash with "missing apiVersion or kind". webApp.APIVersion = "apps.irexta.com/v1alpha1" webApp.Kind = "WebApp" // Using client.Apply forces ownership and lets the API server handle conflicts if err := r.Status().Patch(ctx, &webApp, client.Apply, client.FieldOwner("webapp-operator"), client.ForceOwnership); err != nil { logger.Error(err, "Failed to patch status via SSA") return ctrl.Result{RequeueAfter: time.Second * 5}, err } return ctrl.Result{}, nil
}

Step 3: Test Kubernetes Operator in CircleCI

Because Operators manipulate the core Kubernetes API, a bad code commit can accidentally delete critical cluster infrastructure. We must test it automatically using a test kubernetes operator circleci pipeline.

We don't need to spin up a full cloud cluster just to run unit tests. Kubebuilder provides envtest, a tool that simulates a local control plane (etcd and kube-apiserver) directly inside the CircleCI Docker executor.

Here is the ultimate, bulletproof .circleci/config.yml (avoiding fragile hardcoded paths):

version: 2.1
executors: go-executor: docker: - image: cimg/go:1.24.0
jobs: lint-and-envtest: executor: go-executor steps: - checkout - run: name: Run Go Vet & Static Analysis command: go vet ./... - run: name: Install EnvTest & Control Plane Binaries command: | go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest setup-envtest use 1.30.0 --os linux --arch amd64 --bin-dir ./testbin - run: name: Execute Reconciliation Integration Tests command: | # [Bulletproof SRE]: Never hardcode paths. Use dynamic output from setup-envtest. export KUBEBUILDER_ASSETS=$(setup-envtest use 1.30.0 --os linux --arch amd64 --bin-dir ./testbin -p path) make test
workflows: operator-pipeline: jobs: - lint-and-envtest

Step 4: Security & Production Hardening

Deploying an Operator into a production environment requires shifting your focus from application code to system-level security.

Do Not Run Operators as Root

Operators possess extremely high RBAC privileges (often Cluster-wide watch/update capabilities). If an attacker breaches the Operator container and it is running as root, the entire cluster is compromised. Always configure the Operator pod's securityContext with runAsNonRoot: true and proactively drop all unnecessary Linux capabilities.

The SRE Solution: Why Operators Demand Bare Metal

We've built the Operator and tested it in CircleCI. Now comes the architectural decision: deploy kubernetes operator bare metal vs Public Cloud VMs.

The API Server & etcd Bottleneck

Kubebuilder Operators are highly optimized. For read operations, they do not constantly hit the API server; instead, they query an In-Memory Informer Cache. However, every time your Operator commits a state change (Writes/Updates), it hits the API server, which strictly demands sub-millisecond fsyncs to the underlying etcd data store.

On public clouds like AWS or GCP, your cluster's etcd is backed by network-attached block storage (like EBS), which is heavily throttled for IOPS. Furthermore, the hypervisor virtualization layer adds critical microsecond delays to every API call. When an Operator manages thousands of Custom Resources, this disk I/O bottleneck causes the Reconciliation Loop to severely lag, leading to delayed scaling and failing automated backups.

To unlock the true power of infrastructure automation, SREs deploy their control planes and Operators on iRexta Dedicated Bare Metal Servers. By eliminating the hypervisor tax and utilizing direct, local PCIe NVMe drives for etcd, your Kubernetes Operators can execute thousands of state changes per second with zero latency. You stop renting bloated cloud Managed Services, and start owning your automated, high-performance infrastructure.

Kubernetes Operator Implementation: FAQ

What is the difference between a Kubernetes Controller and an Operator?
Every Operator is a Controller, but not every Controller is an Operator. A Controller watches standard Kubernetes resources (like Pods). An Operator introduces application-specific domain knowledge to manage Custom Resource Definitions (CRDs) automatically.
How do you handle HTTP 409 errors in an Operator?
HTTP 409 (StatusConflict) occurs when multiple processes attempt to update a resource simultaneously. Elite SREs bypass this entirely by using Server-Side Apply (SSA) via client.Apply and client.FieldOwner. However, you must manually re-inject TypeMeta (APIVersion/Kind) as r.Get() strips it out.
How do you test a Kubernetes Operator in a CircleCI pipeline?
In CircleCI, you can use the envtest tool provided by Kubebuilder. It spins up a localized, simulated control plane (etcd and kube-apiserver) inside your CI job, allowing you to run integration tests rapidly without needing a full, live Kubernetes cluster.
Why is Bare Metal better for running Kubernetes Operators?
While Operators read from a fast in-memory cache, every state change (Write/Update) hits the API server and demands sub-millisecond fsyncs to the underlying etcd store. Public cloud VMs throttle this disk I/O. Dedicated Bare Metal servers provide unrestricted NVMe performance, executing state changes instantly.