Skip to content

The adventures of GitOps with a Monorepo

Ah, GitOps. The magical buzzword that everyone pretends they've been using for years. Because, obviously, if you're not managing your infrastructure like it's just another codebase, are you even DevOps-ing?

Gone are the days of logging into servers, running ad-hoc scripts, and praying nothing breaks. Now, we push YAML to Git and let automation take the wheel. Simple, right? Sure—until you realize your cluster is on fire because someone merged a "quick fix" at 5 PM on a Friday.

But hey, at least with GitOps, everything is version-controlled, auditable, and (in theory) rollback-able. So, let's dive into why GitOps is the best thing since sliced CI/CD pipelines.

Why ArgoCD? UI Simplicity and CRD Integration

When selecting a GitOps tool, ArgoCD stood out due to its user-friendly UI and seamless integration with Kubernetes Custom Resource Definitions (CRDs). The UI provides clear visibility into application states, making it easier to track deployments and identify issues without relying solely on CLI commands. Additionally, its CRD-based approach allows for declarative management of applications, aligning well with Kubernetes-native workflows.

The Plan: Refactoring the Deployment Repository for ArgoCD

Currently, I manage a single deployment repository for all my applications. While this centralized approach works, it needs to be restructured to take full advantage of ArgoCD. The goal is to improve automation, maintainability, and visibility while keeping everything in one place.

Steps for the Transition:

  1. Restructure the repository to use ArgoCD’s Application and AppProject CRDs, ensuring clear separation between different applications.
  2. Define ArgoCD applications declaratively so that each application has its own dedicated configuration within the repository.
  3. Enable automated synchronization, allowing ArgoCD to continuously monitor the repository and apply changes as they are merged.
  4. Establish a structured deployment workflow, making it easier to track, audit, and roll back changes if necessary.

By refining the deployment repository, I aim to streamline operations, reduce manual intervention, and ensure a more scalable and reliable deployment process.

Defining the Structure for Application Values

The first step in the transition was to establish a clear and scalable structure for defining application deployment values. To maintain organization and simplify management, the repository is structured as follows: 

repo/
│── apps/
│   ├── <namespace>/
│   │   ├── <application>/ # Kustomize application
│   │   ├── <another-app>/
│   │   ├── _helm/ # Holds all Helm values files for this namespace
│   │   │   ├── <helm-chart-values>.yaml
│   │   │   ├── <another-chart-values>.yaml

Structure Breakdown:

  • apps/: The top-level directory containing all deployed applications, organized by namespace.
  • <namespace>/: Each Kubernetes namespace has its own subfolder, ensuring clear separation of applications.
  • <application>/: Each deployed application has its own folder containing its Kustomize base and overlays.
  • _helm/: A dedicated folder inside each namespace to store values files for Helm charts deployed within that namespace.

This structure ensures that application configurations are modular, easy to maintain, and compatible with both Kustomize and Helm-based deployments. With this foundation in place, the next step is integrating it into ArgoCD for automated synchronization and deployment.

Telling ArgoCD How to Load Applications

With the repository structure in place, the next step is to instruct ArgoCD on how to discover and manage applications. Since all applications belong to a single deployment repository, we start by defining an AppProject to manage them under a unified scope.

Defining the AppProject

To ensure proper access control and organization, we define an AppProject that includes all our applications. This project acts as a logical boundary for our deployments. 

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: home-cluster
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  description: Main Home cluster configuration

  sourceRepos:
  - '*'

  destinations:
  - namespace: '*'
    server: https://kubernetes.default.svc

  clusterResourceWhitelist:
  - group: '*'
    kind: '*'

  orphanedResources:
    warn: false

Automating Application Creation with an ApplicationSet

Rather than manually creating ArgoCD Application resources for each deployed app, we leverage ApplicationSets to automate the process. This allows ArgoCD to dynamically generate application definitions based on the repository structure.

Bootstrapping ArgoCD with a Root Application

To enable the automation, we define a root application that is applied manually via CI/CD. This application is responsible for syncing the entire ArgoCD configuration, including the ApplicationSets that define how individual applications are created.

The root application is placed in:

repo/  
│── argocd/  
│   ├── apps/        # Contains ApplicationSet definitions  
│   ├── root-app.yaml  # Root ArgoCD application applied manually

Root Application YAML Definition

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd-apps
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: home-cluster

  # Source of the application manifests
  source:
    repoURL: ssh://git@gitlab.xxx.xx/xxx/repo.git
    targetRevision: HEAD 
    path: argocd/apps

    # directory
    directory:
      recurse: true
      jsonnet: {}

  destination:
    server: https://kubernetes.default.svc
    namespace: argocd

  # Sync policy
  syncPolicy:
    automated:
      prune: false
      selfHeal: true
      allowEmpty: false
    syncOptions:
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    - RespectIgnoreDifferences=true

    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  revisionHistoryLimit: 5

ApplicationSet for Automatic Application Management

Inside the argocd/apps/ directory, we place ApplicationSet definitions that dynamically generate ArgoCD applications based on the repository structure. This allows all applications to be automatically discovered and managed by ArgoCD.

With this setup, managing deployments becomes seamless:

  1. Apply the root application manually via CI/CD.
  2. The root app syncs all configurations inside argocd/apps/.
  3. The ApplicationSet discovers and registers all application deployments automatically.

This approach ensures a scalable and maintainable GitOps workflow with minimal manual intervention.

Automating Application Management with ApplicationSets

To dynamically create ArgoCD applications, we define two ApplicationSets:

  1. Kustomize ApplicationSet – Handles Kustomize-based deployments.
  2. Helm ApplicationSet – Manages Helm chart releases based on a predefined configuration.

Kustomize ApplicationSet

The Kustomize ApplicationSet is straightforward. It uses the Git generator to scan the repository and iterate over the apps/ directory, excluding the _helm/ folder. Each subfolder inside a namespace becomes an ArgoCD Application with a Kustomize source. 

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: kustomize-releases
spec:
  goTemplate: true
  goTemplateOptions: [ "missingkey=error" ]
  generators:
    - git:
        repoURL: ssh://git@gitlab.xxx.xx/xxx/repo.git
        revision: HEAD
        directories:
        - path: apps/**/**
        - path: apps/**/_helm
          exclude: true
  template:
    metadata:
      name: "{{ .path.basename }}"
    spec:
      project: home-cluster
      source:
        repoURL: ssh://git@gitlab.xxx.xx/xxx/repo.git
        targetRevision: HEAD
        path: "{{ .path.path }}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{ index .path.segments 1 }}"

      syncPolicy:
        automated:
          prune: true
          selfHeal: true
          allowEmpty: false
        syncOptions:
        - CreateNamespace=true
        - PrunePropagationPolicy=foreground
        - PruneLast=true
        - RespectIgnoreDifferences=true
        - ApplyOutOfSyncOnly=true
        - ServerSideApply=true

Helm ApplicationSet

The Helm ApplicationSet is more complex since it needs to determine which Helm charts to deploy and with what values. To achieve this, we use a Matrix generator that combines:

  • A Git generator to retrieve the charts.yml file inside argocd/apps/, which defines the Helm charts to deploy.
  • A List generator to parse the charts.yml and iterate over it.

Example charts.yml File

charts:
  - repoURL: https://kubernetes-sigs.github.io/external-dns/
    chartName: external-dns
    targetRevision: 1.15.1
    releaseName: external-dns
    namespace: networking
  - repoURL: https://argoproj.github.io/argo-helm
    chartName: argo-cd
    targetRevision: 7.8.2
    releaseName: argocd
    namespace: argocd

Using the data from charts.yml, the ApplicationSet combines the Helm chart information with the corresponding _helm/ values files to define Helm-based applications.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: helm-releases
spec:
  goTemplate: true
  goTemplateOptions: [ "missingkey=error" ]
  generators:
  - matrix:
      generators:
      - git:
          repoURL: ssh://git@gitlab.xxx.xx/xxx/repo.git
          revision: HEAD
          files:
          - path: argocd/apps/charts.yml
      - list:
          elements: []
          elementsYaml: "{{ .charts | toJson }}"
  template:
    metadata:
      name: "{{ .releaseName }}"
    spec:
      project: home-cluster
      sources:
      - repoURL: "{{ .repoURL }}"
        targetRevision: "{{ .targetRevision }}"
        chart: "{{ .chartName }}"
        helm:
          releaseName: "{{ .releaseName }}"
          valueFiles:
          - "$values/apps/{{ .namespace }}/_helm/{{ .releaseName }}.yml" # (1)
          ignoreMissingValueFiles: false

      - repoURL: ssh://git@gitlab.xxx.xx/xxx/repo.git # (2)
        targetRevision: HEAD 
        ref: values

      destination:
        server: https://kubernetes.default.svc
        namespace: "{{ .namespace }}"

      syncPolicy:
        automated:
          prune: true
          selfHeal: true
          allowEmpty: false
        syncOptions:
        - CreateNamespace=true
        - PrunePropagationPolicy=foreground
        - PruneLast=true
        - RespectIgnoreDifferences=true
        - ApplyOutOfSyncOnly=true
        - ServerSideApply=true
  1. Here you can see how we utilize the folder structure from above to dynamiclly build the values files path from the namespace and the releaseName.
  2. I am using here the ability to define a remote values files location since the git generator above only loads the charts.yml

Summary

  • The Kustomize ApplicationSet discovers and deploys Kustomize applications automatically.
  • The Helm ApplicationSet pulls Helm chart definitions from charts.yml and combines them with the values stored in _helm/.
  • Both ApplicationSets ensure that all applications in the repository are automatically managed by ArgoCD.

With these ApplicationSets in place, applications are deployed dynamically without requiring manual ArgoCD configuration for each one.

Automating Dependency Updates with Renovate

Keeping dependencies up to date is critical for security, stability, and feature improvements. Rather than manually tracking updates or relying on latest tags (which can introduce unexpected issues), I use Renovate to automate dependency updates in a controlled and auditable way.

How Renovate Works

Renovate scans the repository, detects outdated dependencies, and creates pull requests with updated versions. It follows a set of predefined rules to manage updates efficiently, reducing the manual effort required to keep everything current.

Renovate Configuration (renovate.json)

The repository is configured with the following key settings:

  • Automated PRs with dependency grouping: Renovate creates pull requests for dependency updates, grouping them intelligently.
  • Preserve semantic versioning ranges: Ensures updates respect defined constraints.
  • Security alerts: Automatically labels PRs with security fixes.
  • Automerge for patches: Patch updates are automatically merged to reduce maintenance overhead.
  • Rate limits disabled: Allows efficient batch processing of updates. 
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:recommended",
    ":preserveSemverRanges",
    ":dependencyDashboard",
    ":rebaseStalePrs",
    ":enableVulnerabilityAlertsWithLabel('security')",
    "group:recommended",
    ":automergePatch",
    ":combinePatchMinorReleases",
    ":disableRateLimiting"
  ],
  "enabledManagers": [
    "kustomize",
    "argocd",
    "helm-values",
    "custom.regex"
  ],
  "prHourlyLimit": 25,
  "branchConcurrentLimit": 100,
  "argocd": {
    "fileMatch": [
      "\\.yml$"
    ]
  },
  "helm-values": {
    "commitMessageTopic": "helm values {{depName}}",
    "fileMatch": [
      "(^|/)_helm/.*\\.ya?ml$"
    ],
    "pinDigests": false
  }
}

Custom Managers for Kustomize, ArgoCD, and Helm

To fully support our GitOps workflow, I use custom managers to detect and update dependencies across different deployment formats:

  1. Kustomize & ArgoCD: Renovate scans and updates version references inside YAML files related to ArgoCD and Kustomize configurations.
  2. Helm Values: The helm-values manager updates Helm values files stored in _helm/, ensuring our Helm deployments use the latest versions.
  3. Custom Regex for Helm Charts (charts.yml): Since Helm charts are defined in a charts.yml file, a regex-based manager extracts version details and updates them automatically.

Custom Regex Managers for charts.yml

{
  "customManagers": [
    {
      "customType": "regex",
      "fileMatch": [
        "(^|/)charts.ya?ml$"
      ],
      "matchStrings": [
        "repoURL:\\s*(?<repoURL>https:\\/\\/.*)\\n\\s*chartName:\\s*(?<chartName>.*)\\n\\s*targetRevision:\\s*(?<targetRevision>.*)"
      ],
      "depNameTemplate": "{{chartName}}",
      "currentValueTemplate": "{{targetRevision}}",
      "datasourceTemplate": "helm",
      "registryUrlTemplate": "{{repoURL}}"
    },
    {
      "customType": "regex",
      "fileMatch": [
        "(^|/)charts.ya?ml$"
      ],
      "matchStrings": [
        "repoURL:\\s*(?<repoURL>[a-zA-Z.]+\\/[a-zA-Z\\/]+)\\n\\s*chartName:\\s*(?<chartName>.*)\\n\\s*targetRevision:\\s*(?<targetRevision>.*)"
      ],
      "datasourceTemplate": "docker",
      "packageNameTemplate": "{{repoURL}}/{{chartName}}",
      "currentValueTemplate": "{{targetRevision}}"
    }
  ]
}

Benefits of Using Renovate

By leveraging Renovate, I ensure that:

  • All Kustomize overlays, Helm charts, and ArgoCD configurations stay up to date.
  • Updates are handled automatically without manual intervention.
  • Changes are tracked through pull requests, allowing for validation before merging.
  • The repository remains secure and stable by promptly applying security fixes.

This fully automated approach eliminates the need to manually monitor dependencies, making GitOps maintenance more efficient and scalable. 🚀