Kubernetes GitOps with FluxCD - Part 2 - Secret Management using SOPS

In Kubernetes-based GitOps workflows, securely managing sensitive information presents a unique challenge. While GitOps principles encourage storing all cluster configurations in Git repositories, committing plaintext secrets creates significant security risks. SOPS (Secrets OPerationS) offers an elegant solution to this problem when integrated with FluxCD.

In our previous post, we explored how to perform the initial setup of FluxCD in a Kubernetes cluster. Building on that foundation, we’ll now address one of the most critical aspects of GitOps implementation: Secret Management.

SOPS, developed by Mozilla, enables encrypting specific values within YAML, JSON, and other configuration formats while keeping the file structure intact. When combined with FluxCD’s native support for decryption, this allows teams to safely store encrypted secrets directly in their Git repositories alongside other infrastructure definitions. The secrets are only decrypted at runtime within the Kubernetes cluster, maintaining security while preserving the GitOps workflow.

This article explores how to implement a robust secret management strategy using SOPS and FluxCD. We’ll cover the setup process, encryption workflows, integration with different key management systems, and best practices for maintaining secure GitOps operations in production environments.

1. Install SOPS and GPG

Your installation method may differ depending on your operating system. Below is the command for openSUSE Tumbleweed:

sudo zypper in sops gpg2

For other distributions:

Ubuntu/Debian: sudo apt install sops gnupg2
macOS: brew install sops gnupg
Arch Linux: sudo pacman -S sops gnupg

2. Generate Keypair

Next, we’ll generate a GPG keypair that will be used for encryption and decryption:

export KEY_NAME="cluster.local"
export KEY_COMMENT="For Flux Secrets"

gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF

This creates a 4096-bit RSA key with no expiration date. The output will look similar to:

gpg: directory '/home/apurv/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/apurv/.gnupg/openpgp-revocs.d/A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4.rev'

Now retrieve the key fingerprint, which we’ll need for subsequent steps:

gpg --list-secret-keys "${KEY_NAME}"

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   2  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 2u
sec   rsa4096 2025-02-25 [SCEAR]
      A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4
uid           [ultimate] cluster.local (For Flux Secrets)
ssb   rsa4096 2025-02-25 [SEA]
      8B514FAAC98DF16F61EE487A45E294E6CACB03D4

Take note of the fingerprint (in this example: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4).

3. Store Private Key as Kubernetes Secret

Now we’ll export the private key and store it as a Kubernetes secret that FluxCD can access:

gpg --export-secret-keys --armor "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4" | kubectl create secret generic sops-gpg 
--namespace=flux-system 
--from-file=sops.asc=/dev/stdin

secret/sops-gpg created

Let’s verify the secret was created properly:

apurv@oxygen:~> kubectl -n flux-system get secrets

NAME          TYPE     DATA   AGE
flux-system   Opaque   3      25h
sops-gpg      Opaque   1      38s

Since the local key is unprotected (we used %no-protection for demonstration purposes), we should delete it from our local machine:

gpg --delete-secret-keys "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4"

4. Configure In-Cluster Secrets Decryption

For production environments, the recommended approach is to store secrets in a separate repository with restricted access. However, for this tutorial, we’ll patch the existing cluster manifest to add support for the decryption provider.

Edit cluster/default/flux-system/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
+patches:
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: flux-system
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-gpg

This patch configures FluxCD to use SOPS for decryption, referencing our previously created GPG key secret.
Let’s commit and push our changes, then verify the deployment:

git commit -m "Configured SOPS decryption for main cluster repo" && git push origin

flux get kustomizations flux-system --watch

NAME            REVISION                SUSPENDED       READY   MESSAGE                              
flux-system     main@sha1:c35e4e45      False           True    Applied revision: main@sha1:c35e4e45

5. Export Public Key and Configure SOPS

We’ll export the public key to the cluster directory:

gpg --export --armor "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4" > ./cluster/default/.sops.pub.asc

Next, create a SOPS configuration file to specify which parts of the YAML files should be encrypted:

cat <<EOF > ./cluster/default/.sops.yaml
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    pgp: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4
EOF

This configuration tells SOPS to only encrypt the data and stringData sections of YAML files, leaving metadata intact for Kubernetes to process.
Commit these configuration files:

git add . && git commit -m "Added SOPS public key and config"

6. Validate the Setup

Let’s verify our setup by creating a sample secret and ensuring it works properly:

Create ./cluster/default/samplesecret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: samplesecret
  namespace: default
type: Opaque
stringData:
  message: This is a secret message

Now encrypt this file with SOPS:

sops --encrypt --in-place samplesecret.yaml

After encryption the yaml file looks like below.

apiVersion: v1
kind: Secret
metadata:
    name: samplesecret
    namespace: default
type: Opaque
stringData:
    message: ENC[AES256_GCM,data:H0iMCWDGpqRSQZtjSXTbMlXaRvXQRxqs,iv:/r5ylcoqWd1wnOp1p9ksDUEs+kkPkdQz31I66LXrpxo=,tag:r2m9+dyo9tnBz+ty42cfrA==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2025-02-25T12:00:11Z"
    mac: ENC[AES256_GCM,data:AuJ9hHq2+Yars67Rw8QhkpjCu8iwqGVsWDJ2/1oEx430yaGSCNXXcZnntek9rsI3xuEzyVwTJcXIE/1yW1bCvPPdWUOel+dY+mLkLSUxio238FbDoGI8mv+7FLpDe4Tn9GZdBbaMyGF6FwXXfQJ8021tLSqK6IJQN8nW2XbT0L0=,iv:KkmRa+HzZ0lI0vsEdqX+ZY6HIpudA/0ABny6OruwuQ8=,tag:6gaSJssqOEQk53DvXCZT0Q==,type:str]
    pgp:
        - created_at: "2025-02-25T12:00:11Z"
          enc: |-
            -----BEGIN PGP MESSAGE-----

            hQIMA0XilObKywPUARAArblXLVn7VaJ+Sw2jNqptBPoPaDq7CA4V2LqUoGQUdIiX
            6pHZoZmfWCpvXAUTqLQ1xEnZ32XHl+lwEVxLzV6OckkYcZKxMGz1lHZGmK7QghtV
            Gm32lFvXVOnMfI9uZ96WH9WW+8Dng3VgyBAK/putNG+N3NLkeXa3vrWaaNaHY/Dj
            aBPK1FqzFAYLbFlGjadA1+1xpFfA4JnE5dLX5HTDIo9DKldJWxxqY6cGo6jTg1lO
            j+vmLHQicdKrApbMqdq1KyRLWTU21B0cRlfIIK37rX9xESXjUlwPmF27sew2KPsx
            iuLyofDu/fVD9kFUC/Zkrmog0IgBWaKuUniOBT1KkcPJoJ5AX9m3qyTGjTGNwvVU
            DWATppNCqnO2NhN2D13sOxL5/BSkJT4HZgdj8oIiQQjv0QwLW6nwlEHcmveiX6aK
            CRUOR51gp+ja+fAlZnevYUpJMJZFz4TX1LTVXqUd8dJeXePaAsCPzO38Ywrpe83D
            Z9n9ABYawCohwSb+Nd1/eoU0RBZwRfcD0PumTbWj8mbSMn6cPHJuAY1j1WZKH1/y
            PM583Vv1zA4NlUJwhAiqI/X0kl22+Qh4tq+tdrhPVLw7P+m7diHG2pyd6jKDZ/K9
            D3tY+eeQWcM3EV0hRBN7yA87Rs07lCy0m72PqAtD07qoUbO6jLXipIuE7TgPuZ3U
            aAEJAhBexKkWWprEJjk+jpt4h8aXyG7wUAeafCWr2kIWz0/kOSnG0STCjuL1kDbu
            +Ysah6EqMijU1sQBJv9Jn5oQ9eTAiHwN2Brh8F1nCPT+E6Ih6lbiJSLAD8duEa7V
            Nsgo8cYB0ebx
            =R3G0
            -----END PGP MESSAGE-----
          fp: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4

Lets push these changes.

git add samplesecret.yaml && git commit -m "Added sample secret"
git push origin

Let’s verify that the secret is created in the cluster:

kubectl get secrets
NAME           TYPE     DATA   AGE
samplesecret   Opaque   1      52s

Verify the content of the secret:

kubectl get secret samplesecret -o jsonpath='{.data.message}' | base64 --decode

This is a secret message

What next ?

Future posts will explore advanced GitOps patterns with FluxCD, including:

Helm chart automation
Image update automation
Notification and alerting configuration

Stay tuned for each of these topics.

References

Official FluxCD Documentation Manage Kubernetes secrets with SOPS
Mozilla SOPS – https://getsops.io/
GitOps Working Group – https://opengitops.dev
Kubernetes Documentation – https://kubernetes.io/docs/

Kubernetes GitOps with FluxCD – Part 2 – Secret Management using SOPS