DevOps and platform engineering

The cluster's state belongs in a repository, and a controller's job is to make reality agree with it. Everything good about this follows from one property: because the desired state is a file, a bad change is a revert instead of an incident where somebody reconstructs last week's configuration from memory.

I've used ArgoCD and FluxCD for that reconciliation, with Helm as the packaging unit so environments differ by values rather than by having their own copies of the manifests. The moment an environment gets special-cased manifests, it stops being a useful test of anything.

Case study: GitOps automation for enterprise OpenShift

Not everything is declarative, and pretending otherwise creates worse systems than admitting it. My rule is a hard line at the cluster API:

At AppsCode I helped design and build a multi-tenant monitoring system spanning many clusters: OpenTelemetry for collection, Thanos for metrics with object storage behind it, ClickHouse for high-volume high-cardinality data, and Perses for dashboards. The requirements that shaped it are the ones I'd start from anywhere:

GitHub Actions and Makefiles do the building, testing and image publishing; the deploy step is a change to a manifest, not a script with credentials. Keeping CI out of the business of mutating clusters is what makes the pipeline safe to hand to someone else.

Client work has taken me across AWS, GCP and Azure, plus Akamai, Hetzner and Rancher-managed environments, with Terraform and Kubernetes doing the provisioning. Portability is less about avoiding lock-in in principle and more about the fact that different clients simply run in different places.

On the GPU side of a product I ran, "how much do we need" was answered with a closed-loop load test rather than an opinion: one pod class at ~11.5 requests/sec and ~25 concurrent users at p95 ≤ 6s, scaling linearly per pod, with response time falling out of users / throughput. A p95 quoted without the offered load beside it doesn't mean anything, and that applies well beyond GPUs.

Running models yourself also turns each one into a service with a lifecycle: images to build, weights to place, capacity to budget between processes that all want it at once, warm-up before traffic rather than during it, and a vector store to back up like any other stateful component. It's the most infrastructure-shaped AI work I've done.

Serving and scaling models

A lot of "the app is slow" turns out to be a query. I've tuned PostgreSQL and ClickHouse for low-latency retrieval and real-time processing, and Redis usually shows up as the queue or the cache that keeps the slow path off the request.