Managing secrets with SOPS in your homelab

Sealed Secrets, Ansible Vault, 1Password or SOPS - there are multiple ways how and where to store your secrets. I went with SOPS and age with my ArgoCD GitOps environment.

Managing secrets in your homelab, be it within a Kubernetes cluster or while deploying systems and tooling with Ansible, is a topic that arises with almost 100% certainty. In general you need to decide, whether you want secrets to be held and managed externally or internally.

• Externally managed: this includes either a self-hosted and externally hosted secrets solution like AWS KMS, password manager like 1Password or similar
• Internally managed: solutions where your secrets live next to your code, no external service is need

One important advantage I see with internally managed solutions is, that I do not need an extra service. No extra costs and connections, no chicken-egg-problem when hosting your passwords inside your own Kubernetes cluster, but cannot reach it when the cluster is down.

Therefore I went with SOPS for both, secrets for my Ansible scripts and secrets I need to set for my K8s cluster. While SOPS can be used with PGP, GnuPG and more, I settled with age as encryption. With SOPS your secrets live, encrypted, inside your repository and can be en-/decrypted on-the-fly whenever needed.

The private key for encryption should, of course, never be committed into your git repository or made available to untrusted sources.

Setup SOPS + age

First, we need to install SOPS, age and generate an age key. SOPS is available for all common operating systems via the package manager. I either use Mac or Arch:

• Mac: brew install sops age
• Arch Linux: sudo pacman -S sops age

Now we need to generate an age key and link it to SOPS as the default key to encrypt with.

Generate an age key

Our age key will live in ~/.config/sops/age/default.agekey.

1

mkdir -p ~/.config/sops/age && age-keygen > ~/.config/sops/age/default.agekey

Now we tell SOPS where to find our age key. I put the next line in my .zshrc.

1

export SOPS_AGE_KEY_FILE=~/.config/sops/age/default.agekey

The last thing to do is to put a .sops.yaml in your folder from where you want to encrypt your files. This file acts as a configuration regarding the age recipient (key) and how the data should be encrypted.

My config file looks like this:

1
2
3
4
5
6
7
8

creation_rules:
    - path_regex: k8s/*\.sops\.yaml$
      encrypted_regex: '^(data|stringData)$'
      age: >-
        age15xzkvh5csaxu2w4x280mhwur5m8svynl3l0gdgvax2r3tt4psqpqr623ag
    - path_regexp: ansible/*\.sops\.yaml$
      age: >-
        age15xzkvh5csaxu2w4x280mhwur5m8svynl3l0gdgvax2r3tt4psqpqr623ag

• Two separate rules depending on the folder, where the encrypted files are located
• Files ending with *.sops.yaml are targeted
• The age key, that should be used for en-/decryption is specified

You might wonder yourself about the first rule with data|stringData. I will just quote the [KSOPS docs](To make encrypted secrets more readable, we suggest using the following encryption regex to only encrypt data and stringData values. This leaves non-sensitive fields, like the secret’s name, unencrypted and human readable.) here:

To make encrypted secrets more readable, we suggest using the following encryption regex to only encrypt data and stringData values. This leaves non-sensitive fields, like the secret’s name, unencrypted and human readable.

All the configuration can be found in the SOPS docs. Let’s now look into the specifics using our new setup with either Ansible or Kubernetes.

SOPS for Ansible

Ansible can automatically process (decrypt) SOPS-encrypted files with the [Community SOPS Collection](Community SOPS Collection).

1

ansible-galaxy collection install community.sops

Additionally in my ansible.cfg I enabled this plugin (see docs) via

1

vars_plugins_enabled=host_group_vars,community.sops.sops

Now, taken from the official Ansible docs:

After the plugin is enabled, correctly named group and host vars files will be transparently decrypted with SOPS.

The files must end with one of these extensions:

.sops.yaml
.sops.yml
.sops.json

That’s it. You can now encrypt your group or host vars files and Ansible can automatically decrypt them.

SOPS for Kubernetes

SOPS can be used with Kubernetes with the KSOPS Kustomize Plugin. The configuration we already prepared, we only need to apply KSOPS to our cluster.

I use the following manifest - see more examples in my homelab repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - service_a.sops.yaml
  - service_b.sops.yaml