GitOps automation for enterprise OpenShift
- OpenShift
- ArgoCD
- Ansible
- Kubernetes
- Helm
- GitOps
The problem
Enterprise clusters are rarely built once. They're built, adjusted by hand during an incident, adjusted again by someone else six months later, and then a second environment is created that is supposed to match the first but doesn't. The cost shows up as an outage that only happens in staging, or only in production, and nobody can say why.
The goal was to make the clusters' configuration a thing you read in a repository rather than discover by running oc get.
What I built
Two complementary layers, with a deliberate line between them.
- Ansible for everything up to the cluster API. Provisioning and configuration steps that are inherently procedural: the work that has to happen before there's a Kubernetes API to submit manifests to.
- ArgoCD for everything inside it. In-cluster resources come from Git and are continuously reconciled, so manual changes are either reverted or visible as drift instead of quietly becoming the new normal.
- Helm as the unit of packaging, so the same chart with different values produces each environment rather than each environment having its own copy of the manifests.
Architecture
Git as the source of truth
The repository holds the desired state; ArgoCD's job is to make the cluster agree with it. That single decision gives you change review, history, and rollback for free: a bad release becomes a revert instead of an incident where someone reconstructs the previous configuration from memory.
Where the imperative/declarative line sits
The most useful design decision in this kind of suite is being strict about which tool owns what. Ansible is good at ordered steps against hosts and APIs; ArgoCD is good at holding a cluster at a known state indefinitely. Blurring them, with Ansible applying manifests that ArgoCD also manages, produces two systems fighting over the same object, which is a genuinely unpleasant thing to debug.
Environments as values, not branches of logic
Promotion works when staging and production run the same chart with different values. As soon as an environment gets its own special-cased manifests, it stops being a test of anything.
Notes
This was client work on private infrastructure, so there's no public repository. The approach generalises. I use the same split, procedural tooling below the cluster API and GitOps above it, on the platforms I work on now.