GitOps

GitOps Principles

GitOps uses git as the single source of truth for infrastructure and application configuration. A controller continuously compares desired state (git) with actual state (cluster) and reconciles differences.

Core principles: declarative configuration, version controlled, automatically applied, and continuously reconciled. Changes happen via git commits and merges, not kubectl apply from laptops.

• Git provides audit trail, rollback, and approval workflows
• Pull-based: cluster pulls changes (more secure than push-based CI/CD)
• Drift detection: manual kubectl changes are reverted automatically

# GitOps workflow
# 1. Developer commits manifest changes to git
# 2. CI runs tests and validation
# 3. Merge to main branch
# 4. GitOps controller detects change
# 5. Controller applies to cluster
# 6. Controller monitors and self-heals drift

Argo CD

Argo CD is the most popular GitOps tool for Kubernetes. It watches git repositories and syncs applications to the cluster. Provides a UI for visualizing application health, sync status, and diffs.

Define an Application CR that points to a git repo, path, and target cluster/namespace. Argo CD supports Helm, Kustomize, and plain YAML.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/k8s-manifests.git
    targetRevision: main
    path: apps/web/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Flux CD

Flux is a CNCF GitOps toolkit — modular components for source management, kustomize/helm application, and image automation. It integrates natively with Kubernetes RBAC and supports multi-tenancy.

Flux automatically updates image tags when new container images are pushed to a registry. Combined with image policy, it promotes images through environments.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: web-app
spec:
  interval: 1m
  url: https://github.com/org/k8s-manifests
  ref:
    branch: main

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: web-app
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: web-app
  path: ./apps/web
  prune: true

Repository Structure

Organize git repositories with Kustomize overlays for environment promotion. Base manifests define common configuration. Overlays for dev, staging, and production patch differences.

Separate application repos from infrastructure repos. Application repos contain Deployment, Service, and Ingress. Infrastructure repos contain cluster config, operators, and shared resources.

k8s-manifests/
├── apps/
│   └── web/
│       ├── base/
│       │   ├── deployment.yaml
│       │   ├── service.yaml
│       │   └── kustomization.yaml
│       └── overlays/
│           ├── staging/
│           │   ├── kustomization.yaml
│           │   └── patch-replicas.yaml
│           └── production/
│               ├── kustomization.yaml
│               └── patch-resources.yaml
└── infrastructure/
    ├── monitoring/
    └── ingress/

Secrets in GitOps

Never store plaintext secrets in git. Use Sealed Secrets (encrypt secrets for git storage), External Secrets Operator (sync from Vault/AWS SM), or SOPS (Mozilla encrypted files).

Sealed Secrets encrypt with a cluster public key — only the cluster can decrypt. External Secrets keep secrets in external stores and sync to Kubernetes at runtime.

• Sealed Secrets are encrypted and safe in public repos
• External Secrets never store secrets in git at all
• Rotate sealing keys periodically for security

# Sealed Secret workflow
kubectl create secret generic db-pass \
  --dry-run=client -o yaml | \
  kubeseal --controller-name=sealed-secrets \
  --controller-namespace=kube-system -o yaml > sealed-secret.yaml
# Commit sealed-secret.yaml to git safely

GitOps Best Practices

Use pull requests for all changes with required reviews. Run validation in CI: kubeval, conftest (OPA), and kustomize build. Implement progressive delivery with Argo Rollouts or Flagger integrated with GitOps.

Monitor sync status and alert on OutOfSync applications. Test disaster recovery by restoring from git to a fresh cluster.

# CI validation pipeline
# 1. kustomize build overlays/production
# 2. kubeconform -strict -summary manifest.yaml
# 3. conftest test manifest.yaml -p policies/
# 4. Merge if all checks pass → Argo CD auto-syncs