notesassorted ramblings on computer

Secret Management for Ephemeral Guix VMs

Last Updated: • Tags:

When using configuration management tooling (such as Guix, Nix, or even Ansible), we often end up storing secrets within that configuration (e.g., passwords). Ansible has a built-in solution for this problem: ansible-vault, which encrypts whole YAML files and decrypts them before loading. However, with purely functional deployment, this is a bit more tricky. With both Nix and Guix, the store is world-readable, therefore—depending on the use case—the secrets must not only be encrypted in the Git repository but also in the store of the configured system.

Nowadays, in both the Nix and the Guix world wrappers around SOPS seem to be the established solution. SOPS itself is not specific to purely functional deployment; it’s likely most commonly used with Ansible. However, when using SOPS with Ansible, we would still do encryption/decryption of secrets locally on the system from which we deploy our hosts. With purely functional deployment, we also invoke SOPS on the configured system. This allows us to keep our secrets as world-readable files in the store.

For Guix, this is implemented by sops-guix, which performs decryption of these files through a one-shot service. Decrypted files are stored with configured permissions in /run/secrets. Usage of sops-guix is already described in a blog post by the original author. In the following, I want to focus on a particular detail: utilization of SOPS in conjunction with guix system vm.

Ephemeral Guix VMs

The guix system vm command is one of my favorite features of Guix, especially in comparison to more established configuration management approaches like Ansible. Given an operating system declaration, it makes some on-the-fly adjustments of the declaration and then generates a shell script that boots the system in an ephemeral VM. This makes it extremely easy to test configuration changes locally before deploying them.

However, when using sops-guix, there is a slight problem with this workflow. We need to decrypt secrets using SOPS within the VM, and since SOPS uses asymmetric cryptography, this requires the VM to have a keypair and its public key to be included in the SOPS configuration. We cannot generate this keypair in the VM after we provision it, as we need to know the public key prior to provisioning for the SOPS configuration. To workaround this problem, we can generate the keypair on the host and then mount it into the VM.

To this end, we can use everyone’s favorite network file system: 9P! Fortunately, QEMU supports exposing host directories in the VM via a 9p file server using -virtfs. Further, guix system vm has some glue code to pass -virtfs through the generated shell script and add the required changes to the file-system section of the operating system declaration (causing the directory to be mounted on boot). Consequentially, we can generate an ephemeral keypair on the host, add it to the SOPS configuration, provision the VM, and mount the keypair via 9P.

Generating a Keypair

Within my Guix configuration repository, I store this dedicated keypair in a subdirectory and later mount it from within the VM. I use GnuPG with SOPS as I need it for Guix authorizations anyhow, but using age is also possible. In order to generate a GnuPG keypair in a subdirectory called gnupg/ run:

gpg --homedir gnupg/ --quick-generate-key root@guix-vm

import the generated key into the keychain on your host:

gpg --homedir gnupg/ --export | gpg --import

Afterwards, extract the fingerprint using:

gpg --homedir gnupg/ --list-secret-keys

and add its fingerprint to the .sops.yaml configuration file. Finally, update the encryption keys of all existing files:

find . \( -name "*.yaml" -a \! -name '.sops.yaml' \) -exec sops updatekeys {} \;

VM Setup

First, we need to configure SOPS in the operating system declaration with which we provision our VM. This is also explained in greater detail in the aforementioned blog post. An exemplary configuration might look as follows:

(service sops-secrets-service-type
 (sops-service-configuration
  (config sops.yaml)
  (gnupg-home "/var/lib/sops")
  (secrets
   (list
    (sops-secret
     (key '("radicale" "users"))
     (file (mycfg-file "myhost.yaml"))
     (user "radicale")
     (group "radicale")
     (permissions #o400))))))

For this configuration to work, we need to mount the previously created gnupg/ subdirectory to /var/lib/sops in the VM. As discussed earlier, gnu system vm can do that for us using 9P; we just need to pass appropriate --expose and --share options:

guix system vm \
	--expose="$(pwd)" \
	--share="$(pwd)/gnupg"=/var/lib/sops \
	my-operating-system.scm

Invoking the shell script emitted by guix system vm should boot the image via QEMU. In the VM, the secret-service-type should run on boot and decrypt the configured secrets. Afterward, they should be available in /run/secrets with the configured permissions.

Future Work

The deployment of the key material is still a bit annoying as it requires fiddling with the SOPS configuration. Ideally, I would want the key material for the VM to also be ephemeral (i.e., created and disposed of on-the-fly).

See Also