Skip to content
Case study

Managing many Kubernetes clusters from one control plane

My work at AppsCode on ACE: a dashboard and API server that manage public and private Kubernetes clusters from a single place, the operators that automate access to them, and a multi-tenant monitoring system that keeps every tenant's metrics, logs and traces separate.
  • Go
  • Kubernetes
  • OCM
  • OpenTelemetry
  • Thanos
  • ClickHouse
  • Perses

The problem

One Kubernetes cluster is a solved problem. An organisation with thirty of them (some on EKS, some on-premises, some sitting behind a firewall with no inbound route) is not. Every operational question that was trivial in one cluster turns into a distributed systems question: who has access to which cluster, how does a developer get a kubeconfig without someone hand-editing YAML, and where do you look when you don't know which cluster the problem is in?

ACE (AppsCode Container Engine) is the product that answers those questions, and the multi-cluster layer is where I've spent most of my time since January 2023.

What I built

  • Centralized multi-cluster management. A single ACE Dashboard view over both public (vendor-managed) and private clusters, so operators stop context-switching between provider consoles and kubeconfigs.
  • Operators and APIs for access. Kubernetes operators that automate authentication and authorization, generating scoped kubeconfigs and role bindings instead of leaving them to manual RBAC edits.
  • Multi-tenant monitoring. A metrics, logging and tracing pipeline built on OpenTelemetry, Thanos, ClickHouse and Perses that spans clusters while keeping tenants isolated from each other's data.
  • Self-hosted packaging. ACE packaged so it can be installed into a customer's own cluster, vendor-managed or self-managed, not just run as a service.
  • API server features. A steady stream of features integrated into the ACE API Server itself, which is where the platform's behaviour actually lives.

Architecture

Hub and spoke

The model that makes multi-cluster tractable is hub-and-spoke: one cluster holds the inventory and the desired state, and an agent inside each managed cluster pulls that state down and applies it locally. It inverts the connection, which is what makes private clusters reachable at all: the agent dials out, so the hub never needs a route in.

Everything else follows from that. Access requests are recorded as resources on the hub and reconciled into real identities and role bindings on the managed cluster. Credentials are issued per cluster rather than shared. Removing a user is a delete on the hub, not a scavenger hunt.

Observability that has to be multi-tenant

The monitoring side has a different constraint: the data is enormous, arrives from everywhere, and must never leak across tenant boundaries. The pipeline splits along the strengths of each component.

  • OpenTelemetry as the collection layer, so instrumentation isn't tied to any one backend and the same collectors handle metrics, logs and traces.
  • Thanos for metrics: Prometheus semantics, but with object storage behind it and a global query view across clusters instead of one Prometheus per cluster that nobody can join.
  • ClickHouse for the high-volume, high- cardinality data, where a columnar engine is the difference between a query that returns and a query that times out.
  • Perses for dashboards, which keeps visualisation declarative and reviewable rather than clicked together by hand.

The parts that took the most thought

  • Reachability, not just authentication. A private cluster can't be treated as a slightly harder public one. The whole design has to assume the hub can never initiate a connection.
  • Tenancy is a data problem before it's a UI problem. Isolation enforced only at the dashboard is not isolation. It has to hold at the query layer, which shapes how the metrics pipeline is partitioned.
  • Reconciliation has to be cheap. A controller that lists every resource in every managed cluster on every loop works fine in a demo and falls over on a real fleet.
  • Self-hosting changes the assumptions. Packaging the platform for someone else's cluster means giving up control over versions, storage classes, ingress and network policy: everything has to degrade gracefully.

Notes

ACE is a commercial product, so this write-up stays at the level of architecture and reasoning. The open-source counterpart to the access-management piece is Cluster Auth Manager, and much of what I learned here went back upstream as contributions to Open Cluster Management and related projects.

Related