Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Nerding out about Nix and NixOS with Jon Seager
Play On
29 JAN • 2024
57 mins
Share:
In this episode, I’m joined by Jon Seager, VP of Enterprise Engineering at Canonical, to explore the fascinating world of Nix. Jon’s experience with automation tools like JuJu and charms offers unique insights into how Nix is transforming software development and system management.
We start with the dual nature of Nix as both a functional programming language and a package manager. What particularly caught my attention was Nix’s ability to create truly reproducible systems - a feature that sets it apart in the software development landscape. Jon’s explanation of how NixOS combines the Nix package manager with its module system reveals the elegant simplicity of immutable system configuration.
The conversation gets especially interesting when Jon shares his home setup. His integration of NixOS with TailScale for a zero-trust environment demonstrates the practical power of Nix’s declarative approach. The way he maintains his system configuration shows how Nix can make complex setups both secure and manageable.
I was particularly intrigued by the resources Jon has curated for the Nix community. For those wanting to dive deeper into the Nix ecosystem, here are some invaluable resources:
- Zero to Nix - The perfect starting point
- Determinate Systems’ Nix Installer - Streamlined Nix setup
- Jon’s NixOS Config - Real-world configuration example
- Wil T’s Nix Guides - Comprehensive learning resources
- Jon’s Libations - Creative Nix applications
If you’re interested in system configuration, reproducible builds, or the future of package management, you’ll find plenty of practical insights here. Jon brings both deep technical knowledge and hands-on experience to the discussion, making complex Nix concepts accessible while maintaining their technical depth.
Transcript
Show/Hide Transcript
Hi, welcome to this episode of Nerding up with Victor.
Today I got a special guest with me, a friend of mine.
His name is John and he's from Canonical.
Hi, John.
Do you want to say a few words to introduce yourself to the listeners?
Yeah, indeed.
So thanks for having me.
I'm VP of Engineering at Canonical for what we call Enterprise Engineering.
So in my day job I look after a good portion of Canonical's automation tooling, so juju and the World of Charms, which kind of provide operators for popular open source software.
So in my kind of sphere of influence is things like Postgres and MySQL and Prometheus and Loki and Grafana.
We have a 5G core kind of suite of automation that we can deploy some machine learning tools, one named kubeflow.
And so I spent most of my day kind of in the Canonical and Ubuntu ecosystem, working on cloud automation for enterprises, essentially.
But my history with open source is relatively long.
I was a early days Arch Linux user and kind of even before that, lots of kind of early versions of Ubuntu and Gentoo and various things like that.
Yeah, amazing.
Yeah.
There's a lot of interesting stuff that you've been talking about, particularly about juju, which I think we should dive into in a future episode because there's some really cool stuff brewing in the juju space, I think.
Cool.
Interesting enough.
In today's episode we will not speak about anything Canonical, but rather about something completely different, which is Nix.
And you really got me excited about Nix a few months ago.
Actually it was last year at the Buddha summit, I think you started to really pitching the idea of Nix.
And in today's episode we're going to cover Nix and we're going to cover a Nix OS Nixon and kind of tie it all together with how you do a dev environment and how you run your, well, local machines really.
So maybe we can start with Nix and what is Nix and why should we care?
Yeah, indeed.
So while I kind of professionally work for Canonical and Ubuntu, I took on the challenge last year of learning Nix and nixos.
I'd been kind of hearing a lot about it on podcasts and reading lobsters and such and decided to kind of dive into it.
And I'm quite captivated by it.
So the one thing I would note is that it's probably one of the like most dense technical ecosystems I've ever tried to get into.
The documentation situation is getting better, but there's quite a lot of movement and so Just trying to get a footing to get started can be quite tough.
So firstly, Nix is actually two things.
So Nix firstly is a functional programming language.
I've heard it described as a bit like JSON, but with pure functions.
It has some similarities to something like Haskell.
And Nix is essentially the language that is used to describe packages and system configurations and CI checks and developer environments and kind of anything in the world of Nix.
Nix is also a package manager.
So probably the vast majority of Nix users are consuming the package manager rather than the operating system necessarily.
So it's been around for a good long time now, I want to say 12, 13 years that I could go wrong.
And Nix packages, which is the kind of GitHub repository which stores all of the definitions for the packages, is I believe, the largest Linux package repository in the world.
It's actually bigger than the Arch Repositories and the AUR combined.
So if you're looking for a piece of software, the chances of it being available with Nix is pretty high.
And similar to some of the things I do in my professional life, it's kind of operating system cross platform.
So you can install Nix on macOS, you can install it on any Linux, and that package collection sort of becomes automatically available to you.
So the way it kind of like loosely hangs together is that using the Nix language, you write derivations which output packages, right?
So derivations you can think of as like a blueprint for a package.
And one of the interesting things about Nix is it has this kind of feature of reproducibility by default.
So it will do that trickery, you see, in the file system where everything was, you know, starting at the UNIX epoch and such.
And the idea is that a derivation should build exactly the same on any given machine, right?
So it's a very portable way to build packages for different systems.
Are they fully reproducible or are they.
That's a great question.
I guess it depends on how far down the rabbit hole you go.
I mean, they're pretty reproducible in the sense that.
I don't know, I don't know.
In fact, they must be to an extent.
So part of the package building process is you have to define a hash for the sources of the package.
And in cases for things like Node JS packages or Rust packages or Go packages, you not only have to define a hash for the source code that you're fetching to build the package with, you then define a sort of intermediate hash of all of the vendor dependencies, whether that's Node Modules or the Cargo Packages or go modules, whatever it is.
So I'm pretty sure, yes, I, I, I, I'm sure I have read somewhere on a NIX forum an effort to make it even more reproducible.
I can't claim to be an expert on exactly what that nuance is, but to all intents and purposes, yes, packages are reproducible.
Fair enough.
And I, that's, I mean, I don't want to derail the introduction too much to nix, but it definitely sound like it solves some of the SBOM issues and supply chain issues by doing that level of pinning as well.
It does.
And it also solves the kind of dependency hell issues that we've seen for kind of years with different Linux distributions.
So Canonical tried to solve this or are solving this to an extent with snapd and snaps, right, where you might have 10 different pieces of software that depend on 10 different versions of Glibc.
And in the times of old that was difficult, right?
Like you had to do lots of trickery in your file system and lots of different build environments and compilation.
Nix kind of makes that a first class problem that it will solve.
So the way it works is every single package, every dependency, can exist in multiple forms on your machine at a given path, right?
So all of the packages and libraries and things that you download with nix going to a place in the file system called the Nix store, and they have these kind of unique names.
So each revision, each version of a package, whether that contains a library or an application, has kind of a long hash and then the name and version of the thing, which means you could have 10 versions of glibc 2.38, all with slightly different hashes because they've perhaps been built slightly differently, perhaps with a different set of compiler flags or whatever it is.
And then the applications that depend on those kind of are patched automatically to link to the right versions that were specified in its derivation.
And the kind of magic of nix is in taking care of that automatic patching of applications and making sure that libraries are all resolved from the right places and that kind of thing.
So, so it's very different than.
It just brings me down memory lane of Gen 2 in the early days where they solved that kind of by compiling everything from source, which is, I guess they still do, I guess it's still active and people are still using it, but they solve it, they solve that by compiling everything.
So you compile from scratch and then here you start link versus link against specific hashes and yes and no, right?
So with Nix you can build all of the packages from source, so you can go get yourself a Nixos machine and build all of the package.
So the derivations that exist in Nix packages, you can elect to build them yourself.
They also run a thing called the binary cache, which means that if your machine, if you allow it to, which most people probably do by default.
Right.
If you, if your machine looks for a given dependency, it will first check the binary cache and say, okay, do I have something that looks like that Nix store path in the binary cache?
And if so, it'll download the right version for your kind of processor architecture and operating system and just unpack it.
If it can't find something in the binary cache, then it will build it from source by default.
So, similar to how, and I was going to ask you like, about comparing it to Homebrew or Brew, it's similar in that sense as well.
I believe Homebrew has a very similar logic.
It tries to pull binaries and if it doesn't have one, it then.
Yeah, indeed.
Back to compare.
Compiling.
Yeah.
Okay, perfect.
So let's go through some of the core concepts in Nix.
I think there are Flakes and then there are some other things you want to go through some of the core building blocks.
Yeah.
So I think flakes are kind of a new and they are defined as an experimental feature at the moment.
They've been around for years as far as I can tell, but there's no.
The community has yet to reach consensus on exactly what that experience should be and the features it should have.
Safe to say I've been using flakes to run or all of my machines and build my software for the last year and nothing has changed to the extent where I've had to make any changes to my code.
And I would say if you're going to get into Nix today, my opinion is that you should invest in that ecosystem.
There are a couple of kind of small corporate entities around Nix and nixos who are peddling it quite hard and I think for good reason.
So flakes, I mean at a very fundamental level, a flake is really just a directory that has a flake mix file and a flake lock file.
And Flakes essentially outputs Nix expressions that others can use for building packages, building modules, accessing developer shells, all sorts of things.
So it's basically a mechanism to share code, share Nix code.
Right.
And so if you're the developer of some open source application, you could include a flake nix file and anyone would be able to point their nix Package Manager at your GitHub repo with the flake nix in it.
And it would then learn how to build the package, get a developer shell, perhaps there's a nixos module that you've included.
It just makes it very easy to kind of use things from around the Nix ecosystem.
And the other thing it brings is this concept of a lock file, which is very similar to your kind of package lock, cargo, lock file, go, some file.
Right.
It's a way of saying this set of NIX expressions expects, you know, NIX packages at this SHA hash as an input.
Right?
And so that's, you know, towards kind of reproducibility.
Right.
You're not going to get, okay, well know IAPPED installed on this day and the version changed and now my build doesn't work.
You can kind of say, no.
Like this flake expects exactly this slice of the NIX packages history to build, and thus it should be reproducible.
Right?
Okay.
And one thing that I guess I spoke to some people about this a little bit about is obviously there are some shared features that with Docker in the sense of how you can run it in some ways, but it's also very different.
So maybe we can zoom in a little bit on how it compares to Docker.
And then perhaps a more interesting one is how we could use NIX inside of Docker to build Docker images to make them more reproducible and more tightly controlled, I guess.
Yeah.
So there are some parallels.
I mean, Docker is primarily used these days as a means for packaging applications with their dependencies.
And certainly you can do that with a Flake or with a NIX package.
What you don't get by default with a NIX package is containerization.
Right.
Which clearly you get with Docker.
Certainly the way I run a little home server with Nix, and the way I that used to be Docker Compose, and then it was microk8s for a while, and now it is just Nix running a bunch of nixos modules that are either available publicly or that I maintain as part of my flake.
So you don't get the containerization by default, though there are ways to kind of declaratively define containers that you want to run as part of a NIX system.
The other thing you can do is you can use NIX derivations to build other artifacts from NIX packages.
Right?
So an example of that is you can use a NIX derivation to build a Docker image.
Right?
So literally the output of that derivation is an OCI image which is built, which has only the paths in it needed to Run the things that you want to put in it.
Right.
So a little bit like the distroless type containers, right?
You could have just the, you know, inside that image, it would have just the bits of the NIX store that are needed to run your package, right.
And then your package itself.
Or you can use a Docker file as normal and you could start from a Ubuntu based image and install the NIX package manager in that image and then do your work in there and do multi stage builders.
Right.
So you can kind of docker images a couple of different ways.
With Nix, you can do it kind of from the outside or from the inside, if that makes sense.
Yeah.
So I guess if you were to do that in, if you thinking about it develop environment, you would probably use the Nix file to run things locally outside Docker.
But when you're pushing it in CI cd, you would output that to a Docker file, which then you can consume in your cluster.
I mean, one of the really nice things that I like, I personally enjoy about the Nix ecosystem is the Nix package manager has this concept of running a package without ever installing it.
So I did this just a couple of days ago.
I was listening to a podcast that was extolling the virtues of a process manager thing called bottom.
Like top, right, but bottom.
And I wanted to try it out.
I didn't necessarily want to install it.
And I literally typed nix space, run NIX packages bottom.
And it just went and fetched that thing into my Nix store and ran the program.
And when I press Ctrl C it's gone, right?
Like the files are still in the next store, but they'll get garbage collected on some interval or the next time I clean it.
And if I want to install it, I can then go and put that package in my system configuration if I never want to see it again.
No harm done, right?
And the same can be done with development shells.
So I set up as part of my, the flicks that I put in my projects, I'll have a development shell set up which says, actually I want a Z shell with, I don't know, the go toolchain built into it plus golang CI plus GO releaser.
And so when I CV into that directory, I type mix develop and it drops me into a shell that has all of those things good to go.
And when I'm done, I just exit that shell and those tools are no longer in my path.
Right.
And so you can have these nice kind of isolated developer environments.
You can have a rustic developer environment for your rust Project a GO developer environment for your GO project.
And that can include packages, dot files, environment variables, like whatever you can think of.
You can kind of package up these like portable developer environments for all of your projects.
That's, that's very beautiful in particular for Python, I guess it's a bit of notorious for versions and packages and you might, with even within your project, you might use different version of Python and definitely different sub module packages.
Right?
Yeah.
So having that package that way sounds like a very clean of doing it so you don't have to pollute your entire system.
And also sounds more manageable than VMs or virtual environments that Python is heavily leveraging.
Normally, yeah, the thing that you get.
And we'll come on to nixos a little bit later, but the thing that, the way that the Nix package manager manages things is you don't.
You end up with much less of this feeling of like craft accumulating on your machine over time.
Like if you ever managed a Debian system or Uvanti system or Arch system, even if you're really, you know, competent and careful and meticulous, if you install and uninstall a bunch of packages over a period of five years, there will inevitably be stuff lying around on the machine.
Right.
That you don't necessarily want.
You can kind of escape that with Nixos.
You don't really have that situation because of the way the next store is on the file system and the way it can be kind of garbage collected and cleaned out.
You, you kind of have this feeling of a very fresh machine which only has what you've asked for at any given moment.
Yeah, I love the concept of that because I can definitely relate to the problem.
But let's.
So you helped me out with getting the screenly CLI ported to Nix.
Do you want to do a bit of a show and tell of how it works and basically bootstrap a Nix setup from zero, really, to the screenly CLI installed and I'm going to perhaps unpack what that actually does.
Yeah, certainly.
So if you look at my screen now, I've started a fresh Ubuntu VM with a tool called Multipass, which is something that I look after in my day job.
So Multipass is just a nice quick way to get Ubuntu instance on any machine.
So if I just jump into that machine, I'm now on a fresh.
A fresh Ubuntu machine.
Right.
The first thing you'll want to do in order to use next is obviously install the package manager Itself.
I mentioned a couple of sort of commercial entities that work around Nix.
One of them is determinate systems who are relatively new player, and they've put out a Nix installer which is written in Rust, which is kind of opinionated and comes with a whole bunch of the defaults that I appreciate.
Like it enables flakes by default.
It, you know, does all the things that I would probably do myself.
So that's a single kind of cult of bash as ever.
But that will get you up and running with the nix package manager on a Mac, on a Steam Deck, on a Linux machine, in CI.
Doesn't matter what it is.
Like if nix can be installed on, this is probably the way that you want to get it.
So once it's installed, we're just going to source the nix daemon file, which just places the nix package manager in the path.
And now we have access to the nix command, right?
So this is just a plain, at this point, plain Ubuntu machine with the Nix package manager installed.
One of the things that you can do with Nix I mentioned about running packages that you don't have installed on your machine is I could just run the screenly clique.
So I happen to know the package name is screeny cli.
So nix run and then you kind of say like repository hash name of package.
The default is the nix packages repository.
So if I run that command, it's now going to go off and fetch the latest versions of nix packages, figure out what to do, go and get the latest version of that package and run it, right?
It'll run the default command.
So there you go, that's the screenly cli.
And I can do whatever that might be, right?
So I can pass help to it.
Like I can just use the screenly clique as if it was installed on my machine.
The other option is I could do nix shell, right?
So nix packages screenly cli.
So this is going to drop me into a shell where the screenly CLI is now in my path.
So now I can just be.
If I could remember what the name of the command is, it's just screen there.
Thanks.
There you go.
So now it's in my path, right?
So I can use screenly as normal.
And then when I'm done with this shell and I don't need access anymore, I can pop out the shell and you can see that screenly is gone.
The interesting thing is it does stick around in the file system, right?
So if I take a look at the NIX store, you can see that we still have the output of these derivations in the file system.
So this is essentially what was downloaded from the nix packages repository.
And if you were looking to there, you kind of see like a very minimal root file system that has like bin slash, user, whatever.
Like imagine like just a snap, a slice of the file system that contains the things that package needs, right?
So under the hood, what happens when you pop in to that environment?
Is it a chroot environment or actually what's technical mechanism that should happen?
It is basically a whole bunch of path, variable mangling and ldpreload and these sorts of things, right?
So these binaries, the screenly binary will have been built and patched at build time.
So it will already be pointing at the right versions of libc.
So if we look at the rest of the next store, just to give you an idea, you can see there's all kinds of stuff in here, right?
So you've got Python setup tools, you've got cargo, you've got all kinds of things in here, right?
Like this is essentially all of the packages you would need to run a very tiny little Linux distribution.
And the way when you build a nix derivation, it will take care of making sure that those packages know which other packages they depend on and point to them in the right places, whether that be through path.
And it also makes extensive uses of wrappers, right?
So they are some applications that you kind of have to wrap and kind of explicitly export the path to say like, okay, one of the things you depend on is over here and over there.
But the nix sort of package ecosystem provides tools that can build those wrappers for you.
I can show you an example of this in a little while.
Yeah, so because it's very different from LXC and Docker.
Why particularly Docker?
So it doesn't leverage C groups or anything like that or not at this level.
No, I mean those constructs will available to you in nixos.
Right?
But this is just about essentially going, getting, running the package.
The other thing we could do is we could talk a little bit about the development environment stuff I talked about.
Yes, so if we do, if we clone the screenly repository CLI and we jump into that directory.
So this is just the upstream repository and you can see in there's a flake lix and I can now just run if I just for the sake of argument, just show you that I don't have cargo installed on this machine, right?
I have none of the rust dependencies.
If I wanted to work on your project, I can now type mix, develop and that's going to read the Flake definition, which I'll show in a moment.
It's going to go figure out what all the packages are that are needed for the development shell for this project, download and or build them, and then drop me into a shell where they're all present.
So this is currently downloading Cargo and the Rust compiler and all the various other things which should be done relatively quick.
And now you can see I can run cargo build and it's going to go build your project for me.
And I've got all the stuff I need in my path.
Good to go.
That's very elegant.
And so you can imagine if you're a team of people who work on a project, you no longer have to have these kind of like protracted developer setup guides.
You just say install the Nix package manager, clone the thing and type next developer and you're good to go.
Right?
Like that's right.
That's the kind of deal.
So I won't let that build all the way through.
There's a couple of things that probably take a little while.
If I just take a quick look at what that actually looks like, I'll jump out because the VIM on my actual machine is a little bit nicer configured than that fresh Ubuntu.
So we look in here.
So this is that repository that we talked about.
And if we take a look at the Flake Lix, this is essentially how I package screenly, right?
So I chose to use Flake straight away.
So this is the kind of input specification I was talking about.
So this basically says this Flake is expecting to build against the Nixos 2311 release set of packages.
23.11 was just released a couple of weeks ago.
It has the kind of inbuilt notion of building for different packages in architectures that will handle cross compilation and all that for you.
You don't have to try and figure out what that means.
You just tell it the architect as you want and it'll make it happen.
And then this is the actual package derivation.
So I talked about derivations.
The next language can feel a little bit obtuse to start with, I will concede.
But you can see here the Nix community have provided this kind of wrapper for building Rust packages, which takes care of going and fetching cargo dependencies and putting them in the right place.
So essentially you just give it a package name.
I've told it the source is in this current directory.
I've told it where the cargo Lock file is.
So as part of the build process, it will go and download the exact version of all of the Rust dependencies it needs and vendor them into the next store so that they are good to go.
Your build happens to need package config and believe it or not, Perl, that's part of the open SSL cargo package, it turns out.
And then there's a whole bunch of extra stuff that's needed for the Apple variant, and that's essentially it.
You can run like all of the rest of it in terms of how to run cargo, build and patch the binaries to look for the right libraries in the next store is all done for you.
Right, that's it.
This is all just metadata, essentially.
And this is pretty much exactly word for word what it looks like in the upstream Nix packages repository, with the exception that the source block tells it to fetch from GitHub in that case and not from the current directory.
Right, very cool.
And if were hypothetically were to add Docker as a target rather than just a binary, how much work would.
That be in terms of probably two or three lines.
You essentially have another output for the flake, and there's some Docker tools that are part of the kind of Nix packages repository.
Basically say, like, build me a Docker image with that package and it will output you a Docker image.
Oh, wow.
That is really that simple?
Okay, I don't have an example of that to show you, but it really is like two or three lines of code tops.
The development shell.
This is the development shell I was talking about.
So this basically says the default development shell for this flake is called screenly cli.
Set some environment config to ensure that flakes are always enabled, and it basically will derive the set of packages it needs from the set of inputs that the package derivation takes.
So I've said give me a shell that has all of the packages that are needed to build that Nix package that we just defined up there.
And I can add other things in here, right?
Like I could say, I also want H top vim and I don't know, Proto Buff and all of the other stuff, whatever it is you might want.
And then anyone who ever type Nix developed would get those things automatically available in their path.
Right.
That's super clean and I presume dev shell and the default navshell, those are standard building blocks in Nix.
It's not something that is custom here, but it's actually why.
No.
So there's a.
There's essentially a.
There's a Schema for what a flake NIX can output.
So it can output, among other things, it can output packages, overlays, nixos modules, nixos machine configurations, development shells, checks, and so in each of those categories it can have a default.
So when you just say Nix develop and nothing else, you'll get the default shell.
If you type nix build and just point at the flake, it will build the default package.
Right?
But you can have other things, other packages.
And I have an example I can show in a moment, which has lots and lots of different packages in it.
Right?
Because there's only one application in here, it just makes sense to make the default screen cli.
Right?
Sure, sure.
No, that's really cool.
You, you mentioned Nix OS a few times.
Should we switch gear and talk a bit about Nix os?
Yeah, let's do that.
So Nix OS is fundamentally just a Linux distribution that is based upon the Nix package manager and the kind of principles that it enforces.
So it is what you would describe as an immutable operating system in that large portions of the file system are mounted, read only when it boots.
And you can't just go hacking around in etc or whatever, you have to.
The way you mutate the system configuration is by mutating a set of NIX files and then rebuilding the kind of system from that, like generating the system from that.
And Nix OS has these, has this concept of generation.
So every time you make a material change to the machine configuration, it will build a new generation and the old one will stick around, which means if you break something, you can revert back to it will just boot the last generation because it's all based on essentially a pile of symlinks.
So one of the weird things about nixos is it throws all of the rules of the standard kind of Unix file hierarchy system out of the window.
And so one of the challenges, and one of the reasons the Nix package manager has all of this magic for automatically patching binaries and injecting, you know, creating wrappers that set the path variable, is because on an XOS machine, like for example, User Bin Bash doesn't exist.
Like LIBC is not where you would expect it to be, it is in the next store and it is symlinked into place.
But as a result of that, you get really fine grained control.
So I can give you an example if I take a look at how my machines are defined, which we'll jump into in just a sec.
But essentially you have this declarative by nature operating system and that declarative kind of nature goes all the way down in my case to file systems bootloaders.
I have everything on all of my machines completely defined, literally how the disks are partitioned, how the bootloader is laid down, how the firmware is updated on the machine, right through to what fancy configuration do I want in Neo Fetch so I can post it on Reddit.
Right.
Like the dot files for Vim, the.
The extensions I have installed in Chrome, the packages I have available in various different environments.
Right.
All defined in code checked into Git.
That's cool.
Yeah, it's super cool.
There's no going back.
I mean, to give you an idea of what that looks like.
I mean, so this flake, one of the interesting things about flakes for system configuration is you can have multiple machines defined in a single flake and reuse code kind of modules like you would in any other programming language, right?
So this is a set of inputs that I use.
You can see it as kind of repositories that provide packages or modules I might want.
So I have NIX packages itself.
Agency is a package which provides kind of secrets management using the age package Lanza Booty is a community project that provides secure boot support for nix.
So the machine I'm on uses secure boot with automated full disk encryption unlock from the tpm, which is super nice.
And then these are actually projects that I maintain.
So Crafts is a repackaging of Carnival's craft tools.
There's snapcraft, charmcraft, Rock Craft, but available for Nixos because I have to build these artifacts every day and I prefer to do it natively on my machine that are vm.
Ember is a small project I wrote that uses Firecracker to launch Ubuntu VMs, really fast for 10 testing and libations is a very recent addition which is a little cocktail recipe app for my phone that I host on a server and browse to over tailscale.
And I have kind of multiple machines here, right?
So that's my laptop, that's my desktop, it's a little server in my in laws house and this is my home server.
And these all use a kind of common set of modules, if you will, that allow me to kind of reuse all of the complicated bits.
So for example, in services you'll see I have one set of configuration for example for pipewire, so services enablepipewire and I reuse this module in all my machines to make sure that I never have to worry about how to configure pipewire Again, I have various modules for bits of hardware.
I have these nice audio engine monitors on my desk, and pipewire likes to suspend inactive syncs after a given set of seconds.
And because they have their own DAC in them that makes them pop a bit like you're unplugging a 3.5mm jack and plugging it right in.
And so this piece of config I write, once I include it on all the machines that have audio NG monitors attached to them, and they stop popping on all those machines, it just solves the problem for me forever, basically.
So, yeah, it's interesting.
This is what you're seeing here is the nixos module system in action.
So you can think of nixos as the combination of the NIX package manager and the nixos module system.
And nixos modules are the thing that turns a pile of packages into a distribution.
So it says, okay, well, this package needs to be started with systemd before that package on this target, right?
This package requires a state directory in varlib Fu.
This package requires, you know, some configuration in Etsy, blah, blah, right?
And so that's kind of how they work if we look at the hardware side, Kara, this is the machine I'm on right now.
So I enable this new convention for kind of bootloaders and a declarative boot called Boot Spec.
And that's what enables me to use Secure Boot.
I then say which kernel modules are available in the INIT RD and which kernel modules I like enabled on the machine.
And I tell it I like to use the latest Linux kernel in this case.
This is my disk specification.
So this is with another community project called Disko.
So you can see I essentially specify two devices, NVMe Zero, which is my root file system, which has an ESP partition and a bunch of butterfs sub volumes.
And then I have a second disk which is just for data, and I specify kind of an encryption key for it and a whole bunch of other stuff.
And so when I point the next OS installer at this flake, it basically, the installation process for one of my machines is I boot the nixos LIVE ISO.
I clone.
I literally type essentially one or two commands, right?
I basically clone my repo and say nixos install and point it at this flake and give it the host name I want it to build, and it just drops the configuration on the machine.
I reboot and the machine is exactly as it was before, right?
So like the everything down to extensions installed in Chrome, plugins installed in Vim, the whole thing, theme that I want to set on my desktop.
Everything is entirely declarative, right?
So it's a bit like, what, cloud init on steroids in some ways, I guess.
Yeah.
And you can just kind of go.
So I use another project called Home Manager.
And Home Manager is.
You can think of it as kind of like a, I don't know, almost like a nix virtual environment for your home directory.
And so I use Home Manager to, for example, to configure vim.
So we could look them in, we could get meta here and look at like the.
The vim config of the vim that we're looking at.
So this is my vim config.
I use a Home Manager module and I basically say enable vim with this package.
Set up some aliases for me.
Here's a bunch of LUA config that I've written and here are the VIM plugins that I want.
And that means that on any machine I go to, vim is set up identically.
I never have to work out how to do it again.
Right?
Like I just.
And I actually use this Home Manager config to provision Ubuntu VMs.
So I not only does this manage my nix machines, but on Ubuntu now I download the nix package Manager and I use Home Manager to provision all my dot files.
Right?
So these, you know, each of these kind of nix statements will be translated in the background to whatever the format is required for this app.
So in this case tmux, it will write a tmux.com file based on the contents of this.
Right?
That's super cool.
So.
So you don't have to manage dot files in the traditional simulink git repo.
One example is git, right.
If you look here, this is essentially my what the sorts of thing you'd expect to find in your git config email name.
There's some rules about which of my identities I use, depending on which directory I'm in, my file system aliases.
And this will get rendered out into like a git config in your home directory, essentially.
That's super clever.
That's super clever.
So the way you bootstrap this machine is so you do a live cd essentially, and then you invoke by git repo.
You clone the git repo.
Yeah, I can actually show you.
I've scripted it to an extent, so let me just quickly show you what that looks like.
So I have this install with disco script, which I sort of collaborated on A bit with a chap called Martin Wimpress, who's quite well known in the Ubuntu community.
So this is basically a wrapper.
This is how I installed this machine.
I'm talking to you right now.
So I re.
I recently kind of redid it because I went down the whole secure boot and TPM backed FD root.
But the crux of this is basically I run disco and pass it the disk config for the host.
If you imagine target host would say Cara Freya Huggin.
Right.
The names of the machines I have.
And so I say nix run disco from the live ISO, it will go and essentially partition the disk from underneath me.
And then I say nixos install flake and give it the name of the host to build.
So in this case it would be Kara.
And then all this is just making sure the nixos config gets like asking to cross into the file system just so I don't have to re clone it when I boot up.
But essentially the install process is those two commands.
I run disco and then I run NIX install nixos install and point it to my flake and that's it.
And so I, like, it's really nice to not have to worry.
When I wanted to go through this process of.
I didn't used to use disco.
I had kind of did like a manual partitioning and install process the first time.
And I'd been using the machine for maybe three or four months.
I got it all set up exactly how I liked it.
And the entire process of like new compaving the machine and getting it back to exactly how it was before was like five or six minutes in total.
Okay.
It just becomes a total non issue.
Right, right.
Is this something that you would, I mean, you obviously use for your home lab.
Is it something that you would consider production ready for the cloud or is it still kind of far off from there?
Or how do you.
I think, yeah, like, I mean, depends on your use case.
Right.
Like, I mean, good luck finding big company.
You know, one of the reasons that people choose Canonical, for example, in my professional life is because we're a company that can provide professional support and security, maintenance and things like that.
And you know, there are certainly companies around that do consulting around Nix and will help you with that, but it's nowhere near as, you know, prolific as something like Canonical or Red Hat or Seuss or whatever.
Right.
But I know of companies who run entirely on Nix.
Right.
Like, and there's something nice from a security posture standpoint, you build these VMs that have exactly what you asked for and nothing more in them.
Right.
Attack Surface is very small.
I can give an example actually of a VM spec that I use just for testing some code.
So if I go to Flake.
So I mentioned earlier on the crafts.
Flake.
So this is essentially packaging snapcraft.
So for building Snap applications and interacting with the Snap store as a developer.
But for on Nix, it's a bit of a strange situation, but this repository provides all of the Python packages that they depend on and then the apps themselves, charmcraft, Rockcraft and snapcraft.
And I wanted a way of testing that these things functioned correctly on nixos in CI.
So before I merge a change, I want to make sure that I've bootstrapped an actual nixos machine, installed charmcraft and tried to build a real charm with it.
Right.
And so I did that by including a nixos configuration for the build VM in the repository.
So, which I can show you here.
So if we just take a quick look at VM Nix.
So this is the entire configuration for a NixOS machine.
There's a few shortcuts here because it's a QMU vm, so it's a little bit simpler in terms of, you know, as ever, it's a vm, but essentially you tell it what platform packages you want to get from Nix, it uses an overlay, which is provided by the Flake.
So the overlay is a way of essentially customizing your machine's view of Nix packages.
And this is super interesting because, for example, imagine that Nix packages provides version 1.2 of A thing foo, like some package on the Internet, but you are like Captain Bleeding Edge and you want to have 1.3.
It's just been released and it hasn't landed in Nix packages upstream yet, but you want it on your machine.
You can literally say, right, take all of Nix packages, except patch the attribute to say, 1.3 rather than 1.2, and rebuild the package with that attribute changed.
Nice.
So here this is basically saying make Nix packages look like it already has my packages in it.
Charmcraft, snapcraft, Rockcraft, you know, enable.
In this case, it's just a CI testing vm.
So I disable the firewall, enable open ssh, allow login with password and allow lexd, and that's it.
If I want to run that VM from any machine, I can do nix, run the test vm, and I'm going to tell it to demonize and go in the background.
So there you go.
It's pre built.
So all of the dependencies downloaded.
This would sometimes take a couple of minutes because we'd have to build an entire rootfs and I can now ssh into that vm.
Right.
So it's set up with a password that I've already set in the config.
And there we go.
We're now in a Mixos vm and inside this Nexus VM is not snapcraft, apparently.
So this is essentially a brand new.
A brand new Nexus vm.
So I should be able to do, for example, okay, it hasn't gone through the whole setup, I need to try again on this demo.
But essentially this is a brand new Nixos VM and when I come out of it's gone again.
And anyone could git clone the crafts flake and run this command and that VM would begin running on their machine with all the exact same set dependencies that we just spoke about.
Right, nice.
So one thing you mentioned security before about improving security because it's basically reducing or removing a lot of dependencies I.e.
potential attack vector, essentially.
But one thing that stood out to me is if you are keeping around so many different versions of various like let's say libc, for instance, would not the fact that you're not pinning into a system LIBC potentially increase the risk of you having a compromised version for that particular package?
You're maintaining like five different versions of libc, for instance.
Yeah, I mean, LIBC maybe wasn't a great example in the sense that I don't think there's a.
I could be wrong.
Maybe there are different versions for a given release of nixos.
I don't know off the top of my head.
But I mean, yes, it's like anything else.
Like you have to have a bit of good hygiene about it.
Right.
If you never update your machine, you're going to have out of date stuff.
Right?
Yeah, the.
There's a couple of different kind of channels.
They're known as in Nix.
So you can run.
They do six monthly releases, so normally in May and November.
So we just had 2305 in May and there's 2311 just happened and they'll be at 2405.
And so actually quite similar to Ubuntu in some ways.
Or you can find Unstable.
Right.
So I follow Unstable because I'm a lunatic and I like my machines breaking on me all the time, which is actually unfair.
They don't break often, but you know, it comes with a little bit more instability than the kind of, you know, six Months he releases.
So it's.
It's kind of up to you.
I have some machines that run on a stable release for the reason that, you know, I.
My little home server, for example, I don't want to have to fiddle with it all the time.
I want it to perform the.
I just want it to work.
Yeah.
My laptop and my desktop, I run a bit more bleeding edge because I want to play with all the new shiny pipewire and new shiny Hyperland and new shiny whatever else.
Right.
Okay, that makes sense.
So one thing I wanted to kind of wrap up this conversation with is I think you have probably some of the most.
Complicated might not be the best word, but sophisticated, perhaps home setup and developer environment setup with.
I know some of it is tailscale and there is a bunch of Envoy, I believe it is, and just want to walk me through a little bit of like your pet project, I guess, which is your.
Your home lab.
Yeah, certainly.
So, I mean, it's not actually.
I mean, in reality, not as complicated as it probably sounds.
So let me.
I can sort of demonstrate some of it.
So I use.
I use Tailscale really heavily.
I probably beginning to sound like a paid shill for Tailscale, but I'm not.
I just think it's absolutely great tech.
I just think they do a wonderful job.
I maintained my own WireGuard server on DigitalOcean for years and for some reason resisted going to Tailscale.
And the first day I tried it, I was like, oh my God, why have I been so.
I mean, it's not that complicated.
I have all my machines on tailscale, so my phone, my laptop, my desktop, my home server, my Apple TV, my iPad, like everything is on tailscale.
And that takes the problem of connectivity out of the picture.
All of my machines can talk to all of my machines all of the time, irrespective of what network I'm on, which makes life pretty easy.
I have a honking great desktop under my desk with a big 32 core Ryzen chip in it.
And then I have a relatively tiny kind of ultrabook laptop.
And the way I.
When I travel for work, I take my little tiny laptop and it runs a web browser and a terminal.
And I basically connect over my tailscale network to my big desktop.
So if I'm ever doing any technical work, it's all happening on the desktop.
And I use things like Visual Studio code Remote.
Right.
Just a remote VS code.
Which is, by the way, like you showed me that.
And I was.
I've been never been A big fan of VS code but that is a game changer.
Like the developer environment, you just have.
The local window, the machine.
It looks like it's running locally on your machine, but actually when you open a terminal, it's a terminal on the remote machine.
You open a file, it's a file on the remote machine.
Right?
It's a super nice way to do your work.
In particular with tailscale.
It's such a clean developer experience.
Obviously if you're into I use Vim.
So if I'm in the terminal, I use vim.
If I'm not, you know, forced to be in the terminal because I'm not an SSH or whatever.
Like my editor of choice for day to day work is versus Code.
And it just to me that just is a super nice benefit is I can carry this super lightweight laptop with a nice long battery life and you know, doesn't break my back carrying it around, but I can get access to my ridiculous desktop and my server at home without thinking about it.
And with VIM binding on versus code it makes it more sensible as well.
Yeah, told.
I mean but I mean the server setup is not.
Not super crazy, right?
So the server is called Thor because back when I bought it I thought it was quite powerful.
These days, not so much.
It's just an old NUC.
It's literally an i7 NUC.
And it's a pretty simple config, right?
It has the boot stuff set up as per my other machine.
I specify the disks with disko and in extra.
I also back this thing up with Borg Backup.
So this is an example of how you can back up like you can declaratively set up Borg backup.
So here I'm using the Nixos module system to say services Borg backup and I create a job called Borg Base.
I throw it some paths that I want to back up and some paths I want to exclude.
I give it the repo and I am using agency here to kind of bring the secrets in.
So the secrets are all in the git repository, encrypted, using nice modern cryptography and that goes away every evening and backs my stuff up to board base.
And I don't really think about it in terms of services.
Again, this is all so the way I like to run it is it runs a whole bunch of media server stuff, Jellyfin and that sort of stuff.
And that is all run kind of bare Nixos modules, right?
They're just like installed as Nixos modules services.
Jellyfin Enable equals true and it appears and I have traffic in front of that, but I've got traffic configured to pull TLS certificates from tailscale.
So tailscale has this neat feature where it can issue you internal HTTPs certificates.
So it uses the DNS challenge, I believe, to basically, when you use tailscale, each of your machines gets a DNS name with a thing they called Magic DNS.
And they can basically do all of the plumbing for you to make sure that you get a cert from let's encrypt that's valid inside your tailnet.
Right.
And so there's this nice feature in Traefik 3, and in fact, this is an interesting kind of illustration of overlays.
Right.
So Traefik 3 is in beta, so it's not in the Nix packages repository at the moment, but I want it.
Right.
And so what I say is, on my systems there is a package called traffic3 which takes the existing traffic definition from nix packages and just overrides the version number and the hash of the source code and builds it otherwise exactly the same.
So it keeps all the same.
All the knowledge about how to build traffic is all in Nix packages.
And I'm just saying do that, but do it with that version of the source code instead.
Very nice.
So that just means that traffic on my machine happens to be, in this case 300 beta 5.
I'll do this kind of periodically.
Like maybe there's a bug in a package and there was a bug in LEXD for nixos a few weeks ago, and I use LEXD every day for work.
And so I patched it in an overlay to make it work on my machine, submitted the patch to nixos and then when it landed in nixos, I removed the overlay and pulled it down from the Nix package repository.
Right.
So it's a really nice.
Like you can just kind of patch things on your system there and then if you want to, either indefinitely or until it gets fixed.
So the beautiful structure of this setup is essentially you got a zero trust environment almost that, largely thanks to tailscale and traffic, you have a beautiful secure environment that.
Well, it is vpn, I guess, but it's zero trust in the sense of like using proper sign certs, using proper set of setups between routing between devices.
Yep.
So, I mean, if you.
For example.
So my latest project is a little cocktail recipe app.
It's called Libations, it's on my GitHub and that is really interesting.
Tailscale provide a library called tsnet which allows you to essentially embed the idea of joining a tail scale network into an application.
This application knows how to join a telnet.
Essentially I set an environment variable which has a key and I run libations and it will join my telnet as if it's like a machine.
It'll just appear on my tailnet as an application that exposes a service and that's deployed on my server.
If we look at, for example, so if I just break down this a little bit.
So each person's tailscale network, Tailnet gets like a, an ID, right?
Mine is Tailnet D5DA.
You can choose it and change it and mess around.
I've just left it as the default.
And so my kind of tailscale network is tailnet d5da ts.net and then each of my machines are, you know, hostname.
So Cara, Thor, Plugin, whatever.
And if I look, if I try to grab this thing now, you'll see it's HTTP.
And I was hoping to get the certificate there as well.
You can see here it's done a TLS 1.3 handshake with a certificate that was issued by.
Let's encrypt based on the tailscale certificate name.
Right.
And this app is only ever available on my telnet.
It doesn't exist on the public Internet, so to speak.
But all of my machines, or critically my phone can see it.
Right.
and that's it.
I use tailscale acls so I limit which machines can talk to which other machines.
but yeah, it's looks really complicated but in reality is super low maintenance because all the machines are declaratively configured and all the networking is taken care of by tailscale.
It really like is.
It's the lowest maintenance kind of home lap situation I've had in a long time.
And you could treat everything like a DMZ essentially on your network.
Yeah, with the caveat that the learning curve for Nix and Nixos is steep.
Like, it took me, I ran what the machine that is now running as a little home server.
I ran on my desk alongside my desktop for maybe three months.
I watched a whole bunch of videos and I just kind of iteratively built this machine up and blew it up a bunch of times and broke it and kind of slowly iterated towards a point where I thought, actually now I think I could probably install this.
I think I went to my laptop next and then after I'd used it on my laptop for a bit, I was then like, all right, the next Way to level up is just to put this on my desktop and suffer through it every day.
And now I think I'm pretty converted.
Yeah, I mean I.
I was initially starting to working with porting our CLI to.
To nix myself an evening and I figured well, how hard can it be?
Well, it turns out that it's slightly steeper loading curve than waking up a docker file or something.
Yeah, it's one of those things like it's a lot of investment up front, but it is super powerful.
Like I now for all the, for all of the things that I work on, I do a flight Linux.
I mean I can show you that libations thing that I was just talking about, right.
This is so this is the flake nix for that and it has a package, I mean that's literally all of the code to package that go project as you know, as a NIX package for any system.
Right.
So I literally say the source is this directory.
Here's the hash for all of the dependencies.
It needs Hugo because it serves a static site built with Hugo and that's it, that's the whole thing.
That is the build recipe for that package on any Nix machine.
And then I also provide a Nixos module.
So if anyone caught onto the modules thing, this is basically saying on a Nixos machine if you've got this kind of set up in your environment, you can do, you know, services libations enable equals true and it will set up this systemd unit for you with this environment and like these settings.
Right.
So it's a way of providing kind of very similar to snaps.
Right.
You work in the Snap where snaps have SNAP services.
Right.
So you've got the actual package and then the ability to run something as a daemon.
This is very similar to that in some respects.
Interesting.
So do you need to do that for.
I guess you do that for your traffic setup as well where you need to renew certificates, you define them in a similar sense.
Wow.
I can show you the traffic setup.
It's not soup.
Interesting because a lot of it is taken care of by traffic.
Ultimately that is not what I wanted.
So if we look at.
So so this is set up in a slightly more modular fashion.
So this kind of just basically says enable traffic, enable the traffic module.
But instead of using the package in upstream, use my Traffic 3.
Right.
So this is saying use exactly what the Nixos folks, the find Nixos folks have defined but don't install their Traefik package.
Install mine and then on Thor I have this config where I essentially say, you can see again, like this is defined in the NIX language, but will get rendered out to a config YAML or config toml.
Right.
So this will become a config that Thor, that Traefik knows how to build.
Right.
So you can see here, if you're familiar with Traefik, I don't know.
But here are the routers.
Right.
That I have.
Here are the services that back those and they point to all the different ports of all the different things that are running on the machine.
And I have a couple of, you know, where is it?
Here's the let's encrypt config that tells it how to use.
What's this?
So yes, this machine has some stuff on the external Internet and uses distillations.
Let's encrypt provider.
And some stuff is in tailscale.
And that's it.
Like, that's all the.
So there's 108 lines of nix, including a couple of helper functions.
Right, nice.
And behind the scenes on NIX os, I presume it relies on systemd to manage services.
Yeah, exactly, it is.
I mean, to the extent that, you know, if I just take a look at.
If you look in here, you can kind of see some of the nix magic at play.
Right.
So I don't know if you.
This is Etsy systemd system where you'd expect to find system units.
And you can see each of them is kind of a symlink to somewhere in the next store.
Right?
Right.
We could take a look at traffic on my server if you wanted.
Right.
So if I jump onto my server.
So you can see Traefik service is symlinked to this thing in the NIX store.
So if we take a look at that file, that's the systemd unit that was written by the NIX OS module.
And you can see that it is running Traefik 3.00 beta 5 as I declared in my overlay.
And it's running from this config file.
So if I grab that and cut that out.
You can see this is a valid traffic config but generated from the.
The kind of nix language stuff.
Right.
And then it kind of goes, you know, it's got a providers thing here which again, you can kind of follow all the.
Down there's all the routers that have been translated from my NIX code into something that traffic can understand with all the different kind of host names and paths and various things.
It needs.
Right.
Wow.
But obviously the nice thing about NIX is while it's a little bit terse and not everyone loves the language, once you've learned it, the machinery behind the scenes then takes care of figuring out how to render a traffic file or a TOML file for traffic or a VIMRC or whatever it is.
Right.
Yeah, no, this is super interesting.
I wish I had enough time to play with this, but I definitely will try to carve out some time in the future and maybe we do another in person no doubt session and yeah, I'll.
I'll send over some links that you can chuck into your.
There's a couple of really good resources.
There's the people who make the NIX install I spoke about have a thing called Zero to Nix, which is this nice kind of getting started guide and I watched some videos from an Australian chap called Will Taylor, I think his name was.
I'll send you the link.
And that's like a nice little YouTube series that goes from kind of installing Nixos the kind of most primitive way right through to managing your system with a flake.
And that's how I started and then I just slowly kind of iterated.
And the nice thing is most like a huge number of people are sharing their configs on GitHub.
Mine is completely public.
I'll put a link to that too.
You can see it, right?
So you can kind of.
I spent a long time just kind of looking around at other people's flights going, oh, that's interesting.
Haven't seen that before.
Like oh, I'm going to steal that.
You know I'm very compelled into throwing up my proxmox Home Lab server in favor of.
Of this.
But yeah, it, yeah, I don't think it's something that should be taken lightly.
Like built in modules for managing systemd unspawn containers and managing VMs and like you can have all of that stuff just in a nix config checked into git somewhere and it'll just happen.
Yeah, we should also talk one day about LEXD UI as a Proxmox alternative.
That's super interesting.
Absolutely.
I think we need to do a follow up session about some other fun things we've been noting about in person over the last year or two and then we can yeah, pick that up in a future show.
But this been super helpful John, I much appreciate you coming on the show and I hope everybody on the.
On listening to this I found this as interesting as I have found.
So thank you so much John and have a day.
Cheers.
Bye.
Found an error or typo? File PR against this file or the transcript.