• Monzo's Workloads and Sensitive Systems: Monzo runs various workloads, some of which are very sensitive like handling secret passwords or authenticating employees. An example is the multi-party authorisation system (MPA) for AWS run on AWS Nitro Enclaves, which is effective in disallowing single employees from having too much privilege. Another sensitive workload is Key Ceremonies, which are also run in AWS Nitro Enclaves for easier automation.
• Reproducible Builds and Their Benefits: Reproducible builds mean bit-by-bit identical builds with the same inputs yielding the same outputs. They are useful for verifying artifacts and detecting malicious code. The Reproducible Builds project has a great summary.
• State of the Art in Monzo's Reproducible Builds: Monzo achieved reproducible builds by defining the build environment as a Dockerfile with hashes in base images and versions in package installs. The build steps in the Dockerfile produced reproducible artifacts and consistent hashes.
• Shortcomings of the Current Method: This method had issues like not being able to control what commands inside the Docker build downloaded from the internet, difficulty in finding sources used in building an image, and dependency issues due to specific system requirements like Docker itself.
• Requirements from Reproducible Builds: To ensure the integrity of deployments, Monzo needed to be able to consistently reproduce artifacts, trace sources used in builds (dependency enumeration), pin source versions (dependency pinning), and prevent downloading without knowledge (offline sandboxing).
• Using Nix for Builds: Docker couldn't meet these requirements, so Monzo switched to Nix. Nix allows defining builds as derivations with clear build inputs and outputs and can build inside an offline sandbox. Pinning dependencies is done by assigning SHA256 hashes.
• Re-thinking Artefact Assembly: The problematic tool in Monzo's build process was AWS' nitro-cli build. Monzo's solution was to use AWS' libraries to build EIFs directly and assemble the necessary files like the Linux kernel and initial filesystem as Nix derivations.
• Putting It All Together: Monzo decided to use Nix and the AWS library to build EIFs and open-sourced the build recipes at github.com/monzo/aws-nitro-util. An example shows that building the same EIF twice yields the same hash.
• Conclusion and Next Steps: Monzo introduced Nix to achieve identical, deterministic builds. Next steps include building more enclave components from source and improving continuous deployment with a binary cache.
• Opportunities at Monzo: Monzo offers a dynamic work environment, competitive salaries, and career growth opportunities. Visit their careers page for job openings.