macOS: Configuring a Nix remote cache
The Nix Multi-User installation is the only available setup on macOS. In Multi-User Mode, Nix sets up some OS users and delegates building to a daemon that distributes load across these users. From the documentation:
When a unprivileged user runs a Nix command, actions that operate on the Nix store (such as builds) are forwarded to a Nix daemon running under the owner of the Nix store/database that performs the operation.
One of the major Nix selling points is the ability to upload artifacts to a distributed cache – Nix calls these substituters. Given a locked flake and system architecture, it’s possible to build its outputs once and upload the build artifacts. Consumers can download cached artifacts without building.
Shipping Nix overlays
As part of my OSS work, I maintain a set of OCaml-focused Nix overlays with support for static and cross-compilation. This project packages quite a few libraries that I use on a day-to-day basis and is the first input I add to any flake. We provide one such distributed cache. On every run, CI builds the relevant OCaml package set across multiple architectures and configurations1. Once done, it uploads to our own custom substituter at anmonteiro.nix-cache.workers.dev
.
We recently started building the overlays on aarch64-darwin
too, right after GitHub announced support for arm64 Macs in GitHub Actions.
Nothing happens?!
Imagine my surprise when running nix build
with a pre-propulated cache started building everything that CI had just uploaded to storage?!
You see, the OCaml Nix overlays ship with a nixConfig that suggests using our substituter, but Nix doesn’t seem to use it on macOS arm64 out of the box. Even after asking whether we should trust this provided substituter:
do you want to allow configuration setting 'extra-trusted-public-keys' to be set to 'ocaml.nix-cache.com-1:/xI2h2+56rwFfKyyFVbkJSeGqSIYMC/Je+7XXqGKDIY=' (y/N)?
Making substituters work
The issue above stems from the fact that the nix
command itself runs in the logged in user; but building is delegated to the daemon. When we accept the overlays configuration, the settings are saved for the user who invoked the command, but not for the build users.
To make it work, we must track substituter information in /etc/nix/nix.conf
:
trusted-substituters = https://anmonteiro.nix-cache.workers.dev
trusted-public-keys = ocaml.nix-cache.com-1:/xI2h2+56rwFfKyyFVbkJSeGqSIYMC/Je+7XXqGKDIY=
Then, you need to add the substituter itself to ~/.config/nix/nix.conf
:
substituters = https://cache.nixos.org https://anmonteiro.nix-cache.workers.dev
The explanation is as follows: in our user-local config, we specify that we want to use the substituter. In /etc/nix/nix.conf
, we tell the daemon that we trust these substituters.
Alternative Method
Alternatively — and I only found out about this one after writing the first version of this post — you can add your username to trusted users in /etc/nix/nix.conf
:
$ echo "trusted-users = $(whoami)" | sudo tee -a /etc/nix/nix.conf
Then, you can rely on accepting the nixConfig
provided by the overlays flake. This is the method I ended up adopting, since it only requires adding a single line to nix.conf
.
Caveat: it looks like this only works if you also track nixConfig in your own flake. In my testing, it looks like nixConfig
in flakes isn’t transitive.
Restart the Daemon
Once you’ve edited those files, don’t forget to restart the Nix daemon:
$ sudo launchctl stop org.nixos.nix-daemon
$ sudo launchctl start org.nixos.nix-daemon
Once the Nix daemon has been restarted, new builds should use the configured substituters. Happy building!
At the time of this writing, we provide artifacts for x86_64-linux
, x86_64-darwin
and aarch64-darwin
, static compilation on musl64
and cross-compilation on aarch64_multiplatform_musl
.