As the number of services on a Kubernetes cluster grows, so does the question of what is actually running and how it got there. Manual deployments, ad-hoc fixes and slightly different configuration in each environment lead to drift and to incidents that are hard to explain. GitOps addresses this by making Git the single source of truth for the desired state of the system.

Key takeaways

  • Declare everything in Git and let an in-cluster operator pull and apply it.
  • Share definitions in a base, and express each environment's differences as small overlays.
  • Keep secrets out of Git by referencing a managed secret store.
  • Use pull-request rules to add change control where it matters, especially production.

How it works

In a GitOps model, a controller such as Flux or Argo CD runs in the cluster, watches a Git repository and continuously reconciles the cluster to match what the repository declares. A deployment is a merged pull request. If someone changes a resource by hand, the controller detects the difference and restores the declared state. Rolling back is reverting a commit.

Because the cluster pulls changes, you do not need to give your CI system broad credentials to the cluster, which is a security advantage over push-based deployment.

Structure the repository: base and overlays

A pattern that scales is a base directory holding shared definitions, often Helm chart releases, and an overlay per environment or cluster holding patches. Kustomize applies those patches, so a production overlay can change image versions, resource limits, replica counts and environment variables without copying the whole definition. This keeps environments recognizably similar and makes differences explicit and reviewable.

Promote through environments

Move a change from development to staging to production by promoting the same artifact, usually a container image tag or chart version, through overlays, instead of rebuilding for each environment. Automate lower environments, and require explicit approval for production.

Handle secrets properly

Secrets do not belong in Git, even encrypted ones, unless you have chosen a well-understood encrypted-secret approach and manage its keys carefully. A common alternative is to store secrets in a cloud secret manager, such as Azure Key Vault or AWS Secrets Manager, and mount them into pods through the Kubernetes Secrets Store CSI driver or an external-secrets operator, using workload or managed identity for access, not static credentials. Git then contains only references.

Add change control where it counts

  • Protect the main branch and require reviews.
  • Apply stricter rules to production paths: more approvers, required checks and, where your process needs it, a reference to a change record in the pull request title that a check can enforce.
  • Run validation in CI: lint manifests, render Helm and Kustomize output and check against policies before merge.

The pull request history doubles as an audit trail: who changed what, when, who approved it and why.

Know when it is healthy

Monitor the reconciliation itself: failed syncs, drift detections and the time between merge and applied. Send alerts to the team that owns the change, and make it easy to see the cluster state next to the repository.

Pitfalls to avoid

  • Hidden manual changes that the controller keeps reverting; investigate rather than disable the controller.
  • Over-clever templating that nobody can read; prefer plain overlays.
  • One giant repository with no ownership; define who owns each directory.
  • Neglecting rollback tests; practice reverting.

Where to start

Pick one non-critical service, put its deployment fully under GitOps and live with it for a few weeks. The lessons about repository layout, secrets and review rules will shape the rollout for everything else.

This article offers general engineering perspective. It does not describe a specific client engagement.