Kookyhaus Labs

Secrets Management

3 min read

This project uses SOPS (Secrets OPerationS) to securely store secrets committed to the publicly available GitHub repository. Secrets are encrypted using age keys and stored within YAML files. Decryption is handled automatically by Flux during deployment.

Prerequisites

macOS:

brew install sops age

Ubuntu/Debian:

apt install sops age

Fedora:

dnf install sops age

Generating Encrypted Secrets

Follow these steps if you want to add a new secret to the repository.

  1. Create a standard, base64 encoded Kubernetes secret YAML manifest.
# Save base64 encoded secret to secret.yaml
kubectl create secret generic test-secret \
  --from-literal=user=admin \
  --from-literal=password=adminpassword \
  --dry-run=client \
  -o yaml > secret.yaml
# secret.yaml
apiVersion: v1
data:
  password: YWRtaW5wYXNzd29yZA==
  user: YWRtaW4=
kind: Secret
metadata:
  creationTimestamp: null
  name: test-secret
  1. Encrypt the data or stringData fields using the sops CLI with the appropriate Age public key:
# Add AGE_PUBLIC_KEY to environment variables
export AGE_PUBLIC_KEY=age1...
# Encrypt YAML manifest with age key
sops --encrypt --age $AGE_PUBLIC_KEY --encrypted-regex '^(data|stringData)$' --in-place secret.yaml
# secret.yaml
apiVersion: v1
data:
    password: ENC[AES256_GCM,data:dEXYcoiwzbiZtCgLXEzg4k6QDWk=,iv:JZISBhKRAVWTxinKCK/O61t0d9UuXbEicOSqdJXpXWY=,tag:Yj9DnYW5ngZ87OpsCxiXrQ==,type:str]
    user: ENC[AES256_GCM,data:4RP6lp3x4P0=,iv:gb+NrOJOi6j7ezczzCRPu/aLMsOzye+3U88IXU4L67k=,tag:gmEQj3kZ+7d/Z5VJ2hD6lQ==,type:str]
kind: Secret
metadata:
    creationTimestamp: null
    name: test-secret
sops:
    age:
        - recipient: age19ktsh93hekpjk5cwjex3njgl9fyxtsfd3lz49khjwxtnlk8ucc6qeqad4a
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRL1VWNUpteWs5MG5Kbjdn
            eGNodHJCS1FaNjZjeTA2K1g0OTNXVjZ0bFhVCjNzblNUL0w0dXNwbGZESzBtcHla
            VUtFaXpSOGN1WGZYbG9nYVI5ZEEwY3MKLS0tIHhrTVJ1d3NaV0JLNGU4S3dNaFhU
            ekJvYUZtQXJ5YkswWjRxZ0Z2eklaR2MKTisB8Y4bi+rSmWriLbj2LfdZJS89BOyY
            r9ZcMGgJE4gZBEucBjfZlBoz6W2yx4KGr4o+NmI5LvxKqYMMXCJEJg==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2025-05-12T17:07:43Z"
    mac: ENC[AES256_GCM,data:ogYW7wjclZXqftghHAv5hVE9iELzo2vcsv7vq7QAskV3aOOLFmu6qy20WAPXGGZdIkDGjZCJkhsAGd36P4eKiGJV9ifRiY7RQWJBFlG3rnaOQlr9WNWka4hnuvesgcw+/uyCTfkOkCsEakdalnrCOHJmSntn1QqShyBKRqyM9Xk=,iv:+A6mQ4a4V4E5HcNTm9U7g5wZA19UouBoAMN9QsSxZtM=,tag://UuOeOAZyEKnVn6OqSQtQ==,type:str]
    encrypted_regex: ^(data|stringData)$
    version: 3.10.2
  1. Commit the encrypted secret.yaml file to the Git repository.

Cluster Cold Start / Disaster Recovery

In a cold start scenario where the cluster state is lost, the Age private key must be reintroduced to the cluster for FluxCD to decrypt secrets.

  1. Retrieve the Age private key from its secure secret store (1Password). Save it to the system with kubectl access as age.agekey (Cluster Access Control)
  2. Create the sops-age Kubernetes secret in the flux-system namespace containing the private key:
# Create sops-age secret with age.agekey
cat age.agekey |
kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdin

FluxCD will then use this key to decrypt SOPS-encrypted secrets during synchronization.

Initial Setup

Follow these procedures if you are setting up secret management on a cluster for the first time.

  1. Create an age key
# Generate age key
age-keygen -o age.agekey
 
# view age key
cat age.agekey
  1. Create the sops-age Kubernetes secret in the flux-system namespace containing the private key
# Create sops-age secret with age.agekey
cat age.agekey |
kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdin
  1. Add .sops.yaml file. Note the .*.yaml regex. It’s important to be consistent with .yaml file extensions throughout the repository.
# .sops.yaml
creation_rules:
  - path_regex: .*.yaml
	encrypted_regex: ^(data|stringData)$
	age: <age_public_key> #age1xd6...
  1. Add the decryption configs to Kustomization files within the clusters/<cluster_name>/ folder.
    Example:
# clusters/staging/apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 3m
  retryInterval: 1m
  timeout: 1m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/staging
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Graph View

Backlinks