Kubernetes engineering: operators and fleets

An operator is a program that keeps saying "here is what I want" until the cluster agrees. The value isn't the automation of a one-time task; a script can do that. It's the loop: the cluster that was unreachable at 3am converges when it comes back, and the role binding somebody deleted by hand reappears on the next pass.

I build these with Kubebuilder, in Go. The part that actually determines whether an operator is good is the API design: if the custom resources describe the desired state honestly, the controller is mostly mechanical. If they describe actions instead of state, you end up with a job runner wearing a CRD costume.

Case study: Cluster Auth Manager, an OCM-based auth operator

At AppsCode I work on ACE's multi-cluster layer, and nearly every hard problem there comes from the same handful of facts about fleets:

Building on Open Cluster Management means reading its code, and that's where most of my open-source work comes from: patches to OCM itself, its API types, clusteradm, cluster-proxy, managed-serviceaccount and multicluster-controlplane, plus contributions to K3s, vCluster, Kyverno, KubeVirt and OpenFGA.

There's a practical reason to do this beyond good citizenship: a fix carried as a local patch is a fix you maintain forever.

All open-source contributions

Alongside the controller work I package and operate what I build: Helm charts as the unit of deployment, FluxCD and ArgoCD for GitOps reconciliation, Ansible for the layer beneath the cluster API, and OpenTelemetry, Prometheus, Thanos, Grafana and Perses for telemetry. I've also packaged a platform for self-hosting, which is a good exercise in humility: you give up assumptions about storage classes, ingress controllers and network policy, and everything has to degrade gracefully.

The voice agent platform runs on K3s, which I also contribute to upstream, so the distribution under the product is one I've read the source of. It's split into deployments with genuinely different load shapes, on their own node roles: the API and its queue workers, a WebRTC SFU for the media plane, the voice agent consumers, the autoscaler as its own process, and Qdrant as a stateful service to back up and monitor like any other.

A few habits from platform work carried straight over. Schema migrations run in an init container on every deploy, so a rollout can't start against an unmigrated database. The autoscaler is a separate process rather than a thread in the API, because the thing that decides capacity should not die with the thing consuming it. And admission is explicit: a saturated pool refuses new calls with a clear error instead of accepting them and degrading every call already in progress.

The one thing that doesn't transfer is elasticity. GPU capacity isn't something you paper over with more replicas, so placement and concurrency have to be right rather than recoverable.

LLM infrastructure and serving