Air-Gapped Build Farm for Supply Chain Proof

The Goal

Our customer org delivers software to downstream consumers in a security-critical, regulated environment, and the core challenge was proving the software supply chain was intact. For all software they provide, they are legally required to prove that the software supply chain is intact:

Costs of the accepted, established solution

In Germany, Germany’s BSI (Federal Office for Information Security) regulates security-critical organizations. The traditional answer to software supply chain proof hurdles is: Take everything offline.

This means developer machines, CI, Git repos, and package mirrors (including open-source) move into a secured environment. Every new patch or repo with code, as well as every person entering this environment, must be security-accredited.

The result is a guarantee that software packages originating from this environment have an untampered software supply chain. This is, however, only needed from release to release and imposes hurdles on daily work:

• External contractors need security clearance (Sicherheitsüberprüfung) before they can start.
  • This is especially annoying because most software security relies on secure design, not obscurity.
  • What’s really needed is an audit of code produced by specialists, not of all specialists.
• All updates must go through a security gate. As a result, toolchains often freeze, and internal forks pile up and age poorly.
• Security fixes arrive late; smaller orgs rarely learn of vulnerabilities before publication.
• Getting source code out is nearly impossible, even to contribute fixes upstream.

The proof is sometimes weaker than it looks: Approved toolchains and deps are often prebuilt, such as Docker images and distro packages. Distros are trusted when they have good standing in public. Approval = chain of human decisions. “Only compilers of Debian release X are fine” at first, then at some point hand-waving, aka “this npm package list is OK” From the outside, it doesn’t always seem like a consistent framework of rules was applied equally to all software inputs.

The new faster, cheaper, and better solution

To keep that proof while removing the daily costs, we designed a new infrastructure: Coding, review, testing, and CI happen in a normal environment where non-security-accredited employees and contractors can participate. Release builds then run on an air-gapped, Nix-based offline build farm.

Because the Nix sandbox generally separates all build processes in an offline environment, source downloads become specialized derivations: Only fixed-output derivations (FODs) may access the network. Their content hash must be known beforehand, which cryptographically guarantees their content. Nix calculates the full dependency graph of each build result. FODs are the leaves of those dependency graphs.

This way, all FODs in a release graph can be isolated and archived into a tarball, which is then signed again. A security-accredited employee can import this tarball into the air-gapped build environment. The build environment can then build everything from scratch: the product and all its dependencies, including the compiler and the compiler’s compiler. While this happens, we can audit the hashed FOD tarball, since it resembles the complete supply chain.

The proof of concept can be demonstrated with two laptops, one online and the other offline. For production, we installed a build cluster orchestrated by a Hydra instance. The final setup looks like this:

Because of the cryptographic hashing, software builds and audits can be performed independently of each other: The infrastructure proves that the full set of input sources leads to the given binary output. The approval process is now objective because official gatekeepers can assess the original commit, patches, and build configuration for every package and all its dependencies.

Trusted binary caches for offline operation

All development of new products and projects can happen online with the new way to prove release integrity. For infrastructure maintained completely offline, an offline NixOS binary cache that contains only trusted binary packages also proved valuable. This way, NixOS-based infrastructure could be reconfigured in offline environments with even higher guarantees than infrastructure based on other distributions.

The build infrastructure has enough compute power to build fresh pins of nixpkgs in a few days for further redistribution in secure environments. Questions about the feasibility and rough cost of such infrastructure could be answered with new clarity:

• 2 servers, 64 cores + 256 GB RAM each
• Full nixpkgs rebuild in ~1 week.
• FOD seed creation: 1 day (excluding download time)
• 114,467 FODs, signed and exported
• FOD size: 1.6 TB uncompressed / 900 GB compressed

Result

Release builds now happen behind the air gap, and the core proof remains intact. The expensive secured environment can now shrink to a handful of single-purpose machines. The quality of the software supply chain proof has improved as it covers all dependencies. Software development, release builds, and audits can now happen independently. The offline build infrastructure is relatively cheap and can be maintained by reduced security-accredited personnel. The architecture and flow of signatures and proofs map directly to the client’s regulatory documentation. An internal binary cache hosts all nixpkgs packages as trusted binaries for environments configured and maintained offline.

Initially, there were concerns about NixOS capabilities and the costs of setting up and maintaining such new infrastructure. To address them, we trained trusted internal key architects and software developers and provided smaller-scale proof-of-concept setups before ordering the final setup. Together with the customer, we resolved all objections by training trusted internal key architects and software developers and by providing smaller-scale proof-of-concept setups before ordering the final setup.

Admission of updates is now decided on a reviewable dependency tree with automatically generated software bills of material (SBOM).

Open source

This proof is only possible because the all the code is open: Nix, Hydra, and nixpkgs are open source, so every package and its compiler can be rebuilt from source and inspected, instead of being trusted because of who shipped it. There is no vendor/distributor between the customer and the supply chain.

It also works in the other direction: Fixes can go upstream, because code is not caught in the air-gapped environment any longer. The client no longer has to maintain internal forks for everything.

The approach itself is public, so other organizations can evaluate it before making any commitment:

• Our demo repo shows the process: offline-build-site
• The hydra-dashboard provides the observability
• Our background article explains secure offline rebuilds with FODs: secure supply chains with Nix.

Value Proposition

• Non-accredited employees and contractors can work on the code from day one, without waiting for security clearance.
• Development and release don’t necessarily have to use exactly the same toolchains. Developers can improve code quality with the latest and greatest tooling, while release builds might still have to use older versions of the tooling that already went through the security gate.
• The proof covers all dependencies, down to the compiler, and the compiler’s compiler, etc.
• The expensive secured environment shrinks to a handful of machines, operated by a much smaller accredited team.
• The client’s supply chain is literally one graph structure that serves builds and documentation as one source.

What began as a rule to take everything offline ended with a small build site for releases, stronger proof than before, and daily development back online.