Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
All things ZFS and FreeBSD with Allan Jude
Play On
01 DEC • 2024
1 hour 18 mins
Share:
In this technically rich episode of Nerding Out with Viktor, host Viktor sits down with Allan Jude, a distinguished FreeBSD developer and ZFS expert, for an in-depth exploration of advanced storage systems and operating system architecture. Allan, known for his extensive contributions to FreeBSD and deep expertise in ZFS, breaks down complex concepts into digestible insights while maintaining the technical depth that advanced users crave.
The conversation begins with a comprehensive look at ZFS’s architectural foundations, where Allan explains how the copy-on-write mechanism fundamentally transforms data integrity and storage management. He illuminates the practical implications of ZFS’s design choices, from its self-healing capabilities to its sophisticated approach to data verification, making a compelling case for why ZFS remains a cornerstone for enterprise storage solutions.
Viktor and Allan dive deep into FreeBSD’s networking stack, examining how its architectural decisions enable exceptional performance and reliability in production environments. The discussion reveals why major technology companies continue to trust FreeBSD for their mission-critical operations, with Allan sharing real-world examples and best practices drawn from his extensive experience in the field.
Throughout the episode, listeners are treated to practical deployment strategies that bridge theoretical knowledge with real-world applications. Allan offers invaluable insights into optimizing ZFS configurations, managing storage pools effectively, and leveraging FreeBSD’s security features to their full potential. The conversation also touches on the future of storage systems and operating system development, with both hosts sharing their perspectives on emerging trends and technologies.
Whether you’re a systems administrator looking to enhance your storage infrastructure, a developer interested in operating system internals, or a technology enthusiast curious about advanced filesystem architectures, this episode delivers actionable insights and deep technical knowledge. Allan’s ability to explain complex technical concepts while maintaining practical relevance makes this discussion an essential listen for anyone working with enterprise storage solutions or Unix-like operating systems.
Transcript
Show/Hide Transcript
Welcome back to another episode of Nerding out with Victor.
Today we're going to go deep into the world of ZFS and FreeBSD with Alan Jude.
Welcome to the show, Alan.
Hello.
Thank you.
So you've been around the FreeBSD world for a long time, and I feel like I mentioned before you hit the record button, FreeBSD is not really getting enough attention for their deserves, really.
So I guess for the people in the audience who are not Even familiar with FreeBSD and the BSD family, maybe can you do a quick intro to, like, why it sets it apart and just some big backstory?
Yeah, I guess we can start with the initial part of the backstory.
So UNIX was originally developed at Bell Labs, which was the research arm of AT&T, the phone company, the only phone company in the US Back then.
And so they developed this operating system basically to be able to build things and also to convince management to buy them a big enough computer, also to make printed manuals and so on to do actual work for the phone company.
But that really was just a side effect of getting a computer they could use to actually write computer programs, a more comfortable environment.
And so they built this, and as it started going out, they found that they wanted people to know how to use it and to find new things to do with it.
So they licensed copies to a bunch of universities.
And back then, computer programs weren't really the same, and there was no standard architecture.
Right.
Every computer was a completely different computer that basically spoke a different language.
And so you had to write the operating system and the programs in, like, the compiled to this assembly dialect of that machine, which is completely different from some other machine.
And that's how or what led to the invention of the C programming language, as we can write the code once and compile it for multiple different machines.
Anyway, one of the universities that got a copy of original research, Unix, as it was called, was the University of California at Berkeley.
And they started adding things to it and changing it to be useful for them.
And eventually what they did was they would send copies of it on a tape because floppy disk hadn't been invented yet.
So they'd mail this tape to other universities and other people who would take it and they would change it and they would send back some of their changes.
Of course, you know, we didn't have Git or tools for managing patches.
So it was a little crazy back then.
But eventually some of those changes made it back to Berkeley and they got incorporated and went out to other people on the next Tape.
And that was kind of the precursor to open source.
No one had thought up licensing or text to put on it yet.
It was just, you know, this is the program and you use it.
And of course every program comes with all the source code and you can do whatever you want with it.
But then it turns out, you know, AT&T had ideas about this.
But that went on for a while and really got interesting.
And eventually they added things to it, including the TCP IP stack, which was the invention of the Internet.
And there's some great stories about that.
Kirk McKusick, one of the people who wrote the file system for that original Unix or the original BSD Unix has a history series on DVD that talks about some of the really interesting things around the invention of TCP ip.
And there was like a proprietary version and then the BSD version.
And the BSD version was faster at some parts, but worse at other parts.
And the way the story went was the proprietary one was faster, but it would crash.
And in the time while it was rebooting, after it crashed, the BSD one would catch up.
So if you transferred a big enough file, you would still be faster on the BSD1 and things like that.
I didn't even know that.
Okay, Yeah, I didn't know that.
I didn't even know that backstory of the TPIP stacks actually.
Yeah, that was, I think BBN it was called.
It was like way back at the very beginning, like when the Internet was only for the government.
It wasn't the open Internet yet.
Right, right.
It's probably still the arpanet.
Yes, and things like that.
Anyway, that's a great lecture that Kirk gives that you can catch from other conferences or from history dvd.
But eventually they kind of finished up the last version of that and a company started called BSDI that was going to make a version of this and actually sell it.
And so they did the port to what was the 386 and made this BSDOS and they were going to sell it to people.
And it was, they were doing a pretty brisk business at the time because, you know, your only other option was Windows or probably wasn't even Windows yet.
Like, yeah, early versions of Windows or like sunos, which was expensive and came with special hardware, whereas the 386 was a cheap machine you could just buy.
Right.
But they made the probably ill advised decision to have their phone number to order this software be 100 its Unix.
Whereas Unix was a trademark of AT&T.
And AT&T is like, yeah, no, Also, your code probably contains, you know, our code, which, you know, we charge $1,000 a license for at least, or if not, a lot more.
And you know, you're selling your software to other people for less than that, so you're not paying us.
How does that work?
And this led to the famous ATT USL lawsuit.
And it turned out there were maybe seven files that needed to be rewritten, but not really.
I don't think the final details ever really got fully disclosed.
Right.
But it slowed BSD down, especially the adoption of it, at just the wrong time, as it turns out.
And so in Finland this student named Linus was like, you know, I want a Unix like operating system and oh, I can't use BSD because it's tied up in this lawsuit, so I'll have to build my own.
And that was the start of Linux.
And like you said, if it hadn't been for the AT and T lawsuit, I would have never bothered making Linux and we would just have BSD instead.
It's a crazy parallel universe, right?
Yeah, yeah, that's a really interesting parallel universe.
Like how that would have affected the evolution of the UNIX wars and what would have happened where and how different life might be just because of the licensing.
So BSD and similar stuff like the MIT license, ISC license, basically the whole license is a couple of sentences that say, you can use this code for whatever you want, but don't take our name off of it and don't claim you were the one that wrote it.
And so the Sony PlayStation 4 and 5 are based on FreeBSD.
And so at the back of the manual there's a bunch of pages of copyright notices that saying, this contains code from this person and this person and so on.
And that's the extent of Sony's obligation, right?
Well, many of the companies that use this BSD software do give back because there's an advantage to doing so.
You know, if you're taking FreeBSD and building something on top of it, you're making some changes.
If some of those changes will be your intellectual property or is it the unique selling feature of your product and you want to keep those secret, but there are other parts and smaller features and so on that aren't secret but will be work for you.
Every time you upgrade to a newer version, you will have to reintegrate those, and that's a lot of work.
If you can contribute those back, they become part of the main line and everybody collectively maintains them, then it's that much easier for you to update in the future.
And really I think the thing that sells it for me the most is the thought of if I buy a washing machine that has the ability to send a push notification to my cell phone when the laundry is done, I don't want it to be using some proprietary or hand built network stack because the people who wrote the software on it didn't want to give back to the gpl which has this viral clause that means any code that touches this has to become open.
So if they're going to use some closed source thing that they bought or write their own thing to avoid that's probably going to lead to a bad time.
Whereas if they can have this standard reference implementation that is under a liberal license, then we're probably all in better shape.
Yeah.
So that takes us to the modern BSD landscape, I guess, where you have, well, three, well, I guess three major flavors of bsd Open, Free, nat.
I guess they're all.
They are the main ones.
There is.
There are some derivatives of those I think as well.
But those are the main ones today.
Right.
Can you speak a bit?
Just compare and contrast like real quickly, like for those who.
Not those who are new to BSD and just understand what their strengths are between the three of them.
Yeah.
So back in the early 90s when, after the BSD USL lawsuit was settled and it became possible, these different versions of BSD came out and the first two were NetBSD and FreeBSD.
And NetBSD wanted to focus on portability running on all those different types of computer.
Especially at the time there were, you know, in the early 90s there were still a lot of these older big iron machines around and companies maybe weren't still producing new software for them.
And there were these ideas of other CPU architectures other than the X86.
And NetBSD really wanted to focus on, you know, when a new thing comes out, we can port to it and be running on it really quickly because they saw that as being the future of computers, not this 386, 486 type thing.
So their focus is always on this portability and being able to run on all the different types of CPUs at the same time.
With FreeBSD the focus was we have this one main architecture, the 386 and eventually the x86.
And we really want to be able to use that as a desktop and a serp.
And so that was their book.
After a couple of years, One of the NetBSD developers had a falling out with the other developers and wanted to do something quite a bit different.
And so they forked off from NetBSD and created OpenBSD.
The main goals there was that the repository, not just the source code, but the repository and the history, would be open to the public and then have a really big focus on security.
And it's come to the point where OpenBSD is mostly kind of like the original Unix, an environment for the developers to do things, whether that's their everyday things or building and kind of this incubator for all these ideas of different ways to do security.
So on top of their other enhancements, they've done the idea of privilege separation.
So one of the things that's kind of maintained inside the OpenBSD project is OpenSSH, the SSH server that's used in every operating system.
And originally that had this one bigger binary that ran as root and then could spawn, you know, every time you logged in.
But that's now been split up into separate tools so that if one gets compromised, it's running without the same level of privileges that, you know, any risky processing is done as like a less important user that can only communicate over interprocess communication with the main process that has all the privileges, so that less chances of it being exploited.
Right.
And then they've taken that idea further and further as they've gone.
But one of the other really interesting ones they do is they actually relink.
So not compiling, but assembling the bits together, the kernel every time it reboots.
So as it's booting up, it makes a new kernel for next time, basically.
And it puts all the pieces together in a random order so that no machine will be exactly the same for an exploit to be able to know where it's going to find like the.
Certain functions for memory injections or for memory attacks, essentially.
Yeah.
For basically any kind of memory attack.
Where you're going to try to find these gadgets to string together, do an exploit.
Every time this machine reboots, it'll be different.
And that means every machine will be different and you won't be able to have the same kind of.
It'll make the attacker's life that much harder.
Right.
And they do a lot of interesting work like that.
Yeah.
I love the slogan for OpenBSD was no default vulnerabilities for like 10 years or whatever it was, or 15 years, whatever the number is today.
Right.
And then it became kind of a heck of a long time when there was one.
The one time.
And.
Right.
They've Had a really good track record on.
Yeah, it's pretty cool in terms of just to compare and contrast, do they share kernels or they completely.
They just.
Toolkit.
Yeah, so the kernels, I suppose technically were close to the same in the very early 90s, but they are very diverged.
Now one of the things that sets each of the three BSDs apart from something like a Linux is that each one is a complete operating system.
In that, you know, in one repository you have the kernel, the basic utilities, like ls, cat, grep, et cetera, and a bunch of the other pieces.
And all the drivers all live in one source code repository and can be built and run with just that.
And in fact, I pretty sure in all three of them that a bunch of the stuff comes with the operating system and then the packages are off to the side and by default there won't be any packages installed.
And then you can maybe you install whatever additions you need on top of the operating system.
Whereas almost every Linux distribution is all right, we take the a version of the kernel, that who knows which one, and we combine it with this bunch of GNU Core utils and other things, and all of those are packages that get installed.
And so the concept of having an operating system with no packages in Linux doesn't really make any sense because all of the components of the operating system, including the kernel, are packages.
Whereas in the bsds you have the operating system and then separately you have all the third party package, right?
And the package managers are rather sophisticated, like the ports from bsd.
I know that was overhauled a decade ago, right?
It used to be just tarballs and now that, yeah, they redesigned that, I think, what, 10 years ago.
I think.
In the very beginning FreeBSD added this concept of ports, which is basically a separate repository with a directory structure sorted into categories and then each program has a directory and inside it there's a little make file, a little.
Oh, yes, sorry, yes, make.
And in there when you run the right command, it knows, hey, I need to go to this website, download the source code, extract it, check the checksum to make sure nobody's feeding me the wrong source code, extract it, apply the set of patches, maybe run the configure script, compile it, install it, and so on, and make sure it gets installed to the right place and that you have all the information you need to be able to uninstall it and all that.
And so it did that for a long time and there were binary packages, but back in the early days, the way it worked Was all of those packages were built once at the day of the release and included on a second cd, but they were never updated.
So you had the version of the packages that shipped with the operating system when it came out and if it was six months later, sorry, all you have is this old packages and you'd have to build your own from the ports tree and that was it.
And it was not ideal.
So yeah, I think around FreeBSD8 for reference, we're on version 14 right now.
With FreeBSD8 we released a new package manager that meant somebody would a tool called Poodra would take that ports tree and recompile it every three or four days because it takes that long to do the build and post it.
And so you could just do package upgrade and it would download stuff kind of the same as an APT or a Yum or whatever.
Yeah, because I remember the big difference is that gets rebuilt every three or four days.
So it's very fresh.
Yeah, because I remember one of the issues.
I used to run BSD, FreeBSD specifically many years ago and we ran it at somewhat small scale but like I think, I don't know, 50 servers maybe so smallish scale but I remember do those installations across a fleet of servers where you have to compile everything on every single server and then you have to.
There were some mechanisms for distributing packages but it's very different from the Linux world, which is apt get installed and voila, that's all you have to care about.
Yeah.
And so eventually we added the concept of a quarterly branch.
So on top of those packages built every three or four days, there's a second repo also built every three or four days.
But is the major versions of those packages only change once every three months.
So that you have the choice of basically a rolling release or a quarterly release that's just security fixes every day, but doesn't randomly change from version three to version four of a program in the middle of your cycle so that you can safely just run automated packages.
And the big thing is with that system there's also the available the program called pudrare which allows you to compile those packages yourself.
So if you have that fleet of 50 servers, the big thing that the ports tree let you do is there are configuration options for all these programs.
So you know, you can decide whether you want this program.
Like if you're installing WordPress, do you want it to use MySQL or Postgres as the database or something like that.
And there's all These knobs you can tweak, but the packages that come from upstream only have the defaults.
So if you want different options, then you can use podrare and build your own package repository that's going to have those options and be able to use it for all 50 of your servers.
Right.
Are those reproducible builds now?
Pretty close to it.
The operating system is fully reproducible and I think the packages are reproducible as well.
You have to, not necessarily by default.
You have to set a couple of options to enable it because there's some downsides to faking the date you set for everything.
But yeah, there's a lot of work that's gone into making that more reproducible so that you can take the same source code and the same tool set and build the identical CD for FreeBSD 13.3 to prove that what the FreeBSD release engineering team gave you came from this exact source code.
Yeah, I mean that's super important.
So let's talk a bit about what actually BSD has been used for FreeBSD in particular, Netflix massive FreeBSD shop, at least was.
I'm not sure they still are.
I think they are still using the cdn.
Are they?
Are, are they still running their entire CDN on Netflix?
On FreeBSD when you are browsing Netflix and picking a video that all runs in Amazon on Linux, but as soon as the video starts playing, that's all coming from FreeBSD.
And so yeah, they built this thing called the Open Connect appliance, which was customized server that they could send to Internet exchanges, but specifically to your ISP.
So when they first started they were using commercial CDNs, but that got too expensive and it got to be that Netflix individually was such a high percentage of the Internet traffic that there weren't any more bandwidth providers available that they weren't already busy with Netflix traffic.
And so to help ameliorate that, they came up with this concept of the Open Connect appliance, which they would basically send to your ISP and your ISP would install in their data center, so that the most popular videos and TV shows that you're watching would be coming from the box inside their network and wouldn't use any of your ISP's Internet traffic, saving your ISP a bunch of money and saving Netflix a bunch of money.
So it worked out pretty well, and Netflix didn't really invent that concept.
Akamai, one of the bigger CDNs had come up with that before, but not to the same degree that what Netflix did.
And so Netflix chose to use FreeBSD for that because of a couple of reasons.
A the license, although it turns out they don't have any proprietary code that's actually in the operating system, so that doesn't matter so much for them.
But the big thing for them was the speed with which they could upstream stuff.
So when they made a change, they were able to contribute it back to FreeBSD and then have it included in the mainline within a couple of weeks.
And so they run the development version of FreeBSD, so not the releases, but actually the development version some number of weeks behind live.
I think it's like six weeks or something.
And so with that they're able to find problems, engineer a solution and get it upstream and then have it ship.
Whereas if they worked with Linux, it would be a lot harder to get a change into the kernel in general.
And if they did, by time a distribution shipped it could be like three years.
They could do their own thing, but that'd be a lot more work.
And just the fact that FreeBSD has all the infrastructure to basically make your own distribution kind of included made it really easy for them, especially because they ended up making quite a few changes.
They worked with NIC vendors including Mellanox, which is now Nvidia and Chelsea to design this concept of encryption offload.
So one thing that set BSD apart back in the early days of like FreeBSD4 and like the early Internet, like very early 2000s or in late 90s, was it was really good at being an FTP server and a web server.
Part of that was this system special system call it had called Send File.
So normally, you know, we have a web server, back then it was mostly Apache running in user space and you know, a request comes in over the network and then we have to wake up the web server and say, hey, there's a new request, do you want to accept it?
Web server says, I'm not too busy, I'll accept it.
And it would go back to the kernel and then eventually a request comes in and it processes it and okay, I need to open this file.
And now I want to read some data from this file.
And now I'm going to ask the kernel, hey, could you send this to the user?
And then when that's done, tell me and I'll read another chunk of the file and then I'll send it to the kernel.
And it involves a lot of copying back and forth because you know, when the web server asks to read a file, it goes to the kernel, hey, could you read this file?
So the kernel reads it up and then has to copy it from the kernel to Apache.
And Apache doesn't look at it just immediately says, hey, kernel, this buffer you just gave me, could you write it back to this socket instead?
And it was all this back and forth.
So with send file, the web server or the FTP server can just say, hey, I have this socket and this file descriptor of a file I've opened.
Can you send this range of bytes in it to the socket for me and just tell me when you're done?
And so now, instead of having to cross that boundary back and forth all the time, you can just say, hey, kernel, just read from this file and write to the socket and do it until either there's an error or you finished the amount I asked you to do.
And that sped things up greatly.
But then HTTPs took off and now we need to encrypt all of that.
And so that meant we couldn't just have the kernel do it.
You had to still copy it out of the kernel to user space, then feed it through OpenSSL to encrypt it and then send it to the user, which involves copying it back into the kernel.
And this took away a lot of the performance that was available.
So they invented this concept of kernel tls, where basically your web server, now, modern days, usually nginx, will accept that connection, negotiate the encryption with the user, and then once it's done the public key part, right, and proven the SSL certificate and all that, it will have a symmetric key, basically a bulk encryption key.
And it can now set that as a socket option on the socket and then do the send file system call.
And now the kernel has the key it needs to encrypt the data, and it will actually, through a kernel module of OpenSSL, encrypt the data and send it whenever you write it to the socket.
So then the send file system call works again and it allows all this offload so you don't have to copy back and forth.
Which became especially a big deal after Spectre and Meltdown, where crossing that kernel user space boundary required a bunch of extra steps and slowed down a bit.
And so that made a big difference there.
But you're still doing the encryption on the cpu, which is fast, but not as fast as it could be.
So Netflix took it one step further after that when they wanted even more performance out of each server where there are dedicated chips on the NICs that can do the encryption.
So instead of sending that bulk encryption key to the kernel, you send it to the kernel, but the kernel sends it to the network card driver, and then it writes the data to the network card driver unencrypted.
And the NIC itself, in its own special CPU, will do the encryption and then send the packet on the network.
And so this allows you to offload all that CPU usage to a dedicated encryption chip on the nic.
And this allowed Netflix to originally go from, you know, maybe they could keep 100 gigabit server busy by sending all this traffic.
Now they're up to doing 800 gigabits per second from a single CPU server.
Wow, that's impressive.
Yeah.
I remember seeing a talk, I think it was EuroBSDcon I went to many years ago.
Netflix did.
I think it was.
Maybe it was around the time they announced these OpenConnect boxes.
And I remember them mentioning, because of the volume of traffic they're running through these boxes, the bugs that they discover are things that nobody else basically would have discovered in 10 years because of the sheer volume.
Yeah, I remember there was one early on in the IPv6 stack, which was basically an overflow.
When you crossed 4 billion of something, it would go back to zero and it caused a problem.
And in general, so they would crash the kernel, but in general, nobody was sending so much traffic that they would do that in less than a month or two, even on a busy host, and probably even longer on another host.
And so it would happen infrequently enough that nobody really got overly bothered by it.
But when Comcast first switched to having IPv6 for all their customers, and so all those people suddenly switched from V4 to V6, going through Netflix, it would take out a Netflix box every couple of hours.
And so they quickly saw the pattern and were able to find the problem and fix it easily and upstream that into FreeBSD and have an errata node so that everybody got the fix.
And it was really interesting just to see that kind of interesting scaling issues.
Right, right.
All right.
And also there was a new announcement.
The German Savant Fund decided to invest some pretty good chunk of change into the Freebies foundation, which I'm sure was very welcome to modernize some infrastructure.
Because how big is the team, like the bsd, the active maintainers?
How ballpark.
One of the other big differences with FreeBSD is rather than on Linux, where there's Linus as the benevolent dictator for life, FreeBSD has a core team of nine people that's elected every two years by the developer base, and that entire developer base has write access to the repository, whereas in Linux it's Linus and a dozen lieutenants or so that actually have the ability to merge stuff and you just send them requests.
With FreeBSD there are 250ish people that can just directly commit stuff into the repositories.
And so anybody who's committed things in the last 365 days at the day of the election is allowed to vote.
And that decides the nine people that will kind of be in charge of a project for the next two years.
Interesting.
Okay, yeah.
Different Covenant model entirely then.
Yeah.
And so the Sovereign Tech Agency's Sovereign Tech Fund is invested money in FreeBSD to improve some of the infrastructure and try to burn down the BugPal and get things in a more sustainable setup.
And then there's also funding coming from the Alpha Omega project to implement additional two FA and a bunch of other security stuff, including an audit of FreeBSD's native hypervisor.
So rather than KVM that they have on Linux, FreeBSD has something called Beehive, which was originally developed at a vendor, and when they decided they weren't going to use it, they upstreamed it and it became part of FreeBSD.
Right, okay, very interesting.
All right, let's switch gear to the primary topic for this conversation, which is run zfs.
Obviously it's really important to give the backstory of FreeBSD, because that's very tightly coupled with zfs, or at least in the history of it.
So you can run bsd.
Sorry, you can run ZFS on Linux these days as well.
And maybe we can start there.
ZFS on Linux and ZFS on bsd.
And BSD is slightly different.
Maybe talk a moment about that.
How they differ and why they differ.
Well, we can start back at the very quick intro in the beginning.
So Sun Microsystems started developing ZFS because they needed a new file system.
They had tried a couple of times to write new file systems, but they got the teams kind of too big, too quickly, and it got complicated and they ended up never finishing any of them.
So Jeff Bonwick handpicked a fresh graduate coming out of Brown University, and basically the two of them got locked in a room for a while with a whiteboard and came up with a whole new file system from scratch.
And especially trying to solve a lot of the problems that file systems had at the time.
Because if you think back to the early 2000s, most file systems had a limit on the biggest file you could have.
So depending if you're on Windows, it was like 2 gigabytes was the biggest file you could have.
And some of them you couldn't have a partition bigger than 4 gigabytes.
And weird things like this.
Even on Linux, most of the file systems there was a limit to how big the biggest file you could have is how many files you could have, how big the volume could be.
All these things were relatively tight limits were actually running into.
Most of them are now big enough that they're harder to run into.
But ZFS was designed with the idea that everything was dynamically allocated so you can't run out of inodes.
If you throw a billion files at it, you're not ever going to get an error.
Most file systems now still have a limit to the number of inodes.
Some of them can dynamically add more after, like you can run a command to adjust the limit and add more.
But they all do still have a limit, whereas ZFS just doesn't because it allocates them dynamically.
It's like you can have as many as you need until you run out of space, right?
But the other big thing was every other file system that exists right now.
Traditional file system assumes you have one disk and it only uses that one disk.
If you have multiple disks, you have to use a volume manager like MD RAID on Linux, or a hardware RAID controller or something to turn that multiple disks into one fake disk that you can then feed to the file system.
ZFS has that functionality built in so you can give it all your disks and it actually knows that they are individual disks and can do things with them.
Anyway, after a couple of years, sun decided to open source that, because everything at sun was open source, because they found that model was better and it was their competitive advantage over something like Windows.
And So when some FreeBSD developers saw that, they're like, well, you know, Solaris originally spawned out of bsd, yeah, many of the concepts are not that different.
We'd be able to port this.
And so one developer in particular, Pavel Doadek, ported ZFS to FreeBSD.
You know, he did most of the initial work in a kind of mad sprint over just less than a month of getting it from it's not in FreeBSD to.
You can actually mount and read write files in a ridiculously short amount of time.
So you did that.
And ZedFS started coming into FreeBSD and getting a lot of attention from that.
And that continued for a long Time when Oracle bought sun, they closed off the source code.
So a project called the lumos, which is a fork of the last open source version of Solaris, started and previously switched to using that as where it would get newer ZFS features from and where it would contribute its changes back to.
But at that time, each version of ZFS was kind of unique to that operating system.
It was up to each operating system to kind of maintain their version and try to keep them in sync.
And then the ZFS on Linux project started around that time as well, having their first release a while after FreeBSD, but they've been working on it around the same time that FreeBSD started, and eventually that became the ZFS on Linux project and was also there on GitHub.
And as things changed and some of the companies that were using Solaris and Lumos switched to using other things or just, you know, their time came to an end, less and less new features were being added in the LUMOS repo, and more of them were appearing in FreeBSD and Linux.
And then getting them back into Illumos and then there and then waiting to be able to pull them back in was causing delays, but also just features were being added in different orders and it was getting kind of complicated.
So that student I mentioned who originally helped design ZFS has now been doing it for over 20 years and is the leader of the project.
He came up with this concept of open ZFS where we would have one repo that was just the agnostic code, just the core of ZFS and not the integration with each operating system.
Right, okay.
And we could compile this in user space and run all the tests, and basically have a common code base that each operating system can pull from, add the glue for their operating system and go from there.
But that didn't really manage to go anywhere because that upstream repo by itself wasn't useful because it didn't have the integration for any operating system.
It was just the common code.
And so it meant somebody would have to do a lot of work to be able to actually get that running and build the infrastructure around it to test it, but to no direct value to them or anybody.
And so that concept was there and the REPO existed and we'd push and pull code from it, but it was not really working as intended.
So especially as more development started coming in on the Linux side and things were getting kind of done out of order a bit, and it was just getting complicated to, you know, eventually they had a bunch of features that FreeBSD didn't have.
But FreeBSD wanted to pull from Illumos, but Illumos didn't have those features yet.
It got complicated.
So we looked at the idea of having a real version of the OpenZFS repo, which would be one repository that includes the OS glue for all of the operating systems and means it would be one official upstream code repository that'd be the same on all the operating systems.
Right.
And so in time for FreeBSD 13.0, that project happened.
And so the OpenZFS repo you see on GitHub now actually is Linux and FreeBSD's code mixed together basically under the modules subdirectory, there's an OS directory and there's one for Linux and one for FreeBSD, and all the common code lives in module ZFS.
And then the operating systems have their special bits in those other directories and it means the code is literally exactly the same.
On FreeBSD and Linux it all comes from exactly the same source code.
There are one set of files that are different based on which operating system you're running, mostly just to integrate with those kernels which are different and provide what we call the Solaris porting layer, which translates the common codes Solaris calls into Linux.
Or FreeBSD calls for things like allocating memory from the kernel, which is done differently.
Linux has the slab allocator and FreeBSD has the UMA, the Unified Memory architecture and so on.
Right, but it means that we have the same code and we added version numbers so you can actually see, you know, FreeBSD14 and Ubuntu 2404 have basically the same version of ZFS.
Right, but there if remind me, there was some issue that held back ZFS coming into mainline Linux for quite some time and it was a license related issue, if I'm mistaken.
Right, and so technically that's still true.
Their ZFS is not included in mainline Linux.
So when sun released zfs, they did it under a license called the cddl, the Common Development License, which is basically identical to Mozilla's MPL that Firefox has released under.
So it says the source code is under this CDDL license, which is mostly liberal, it's not viral or anything like the gpl, but in particular Kadenta clause that when you compile a binary out of it, you can license that binary however you like.
So you can take ZFS and compile a kernel module and license that module as gpl.
The problem is actually an incompatibility with the GPL where if you make this GPL license ZFS module and link it into your kernel, the GPL license says that should make the source code for ZFS GPL licensed.
And the CDDale doesn't let you do that because that would not make any sense.
Right.
And so that's where the issues come from.
And so it's not included in upstream.
Kind of like was it the bcachefs recently was getting added to Linux and it sounds like maybe it doesn't have a long life left after some arguments between the maintainers.
So ZFS is not included in mainline, but Ubuntu has started shipping it as part of their distro and it turns out nobody's suing them.
And so it has kind of become okay to basically for Ubuntu to compile it for you and you load it on your module, whereas, you know, it was always okay for you to compile it yourself and load it on whatever version of Linux you wanted.
But that was a bit of a pain with dkms is not really a pain anymore, but being able to start to get closer to the integration like you can see with FreeBSD, where our bootloader fully supports it and our installer knows all about it.
And we build features like boot environments on top of it, which is basically having multiple different root file systems, possibly based on snapshots of your root file system.
Meaning that if you install some new packages and it breaks something, you can just reboot and from your bootloader pick an older version of your root file system and be back to how your system was an hour ago and everything works again.
It's crazy.
Like one of the things, like I've been using ZFS for well over a decade, I think by now, in one capacity or another.
And the thing that always strikes me is this is the fastest, but it's very old.
But it's still probably the most sophisticated file system out there in terms of feature sets.
And it just keeps chugging along and it's just so reliable, have so many beautiful features that are not widely available across any other file system, really.
Like particularly about snapshots and all these things.
Right.
So let's talk about some ZFF fundamentals.
Like, you obviously is an authority on CFS in your day job, and maybe for those not familiar with your day job, do a quick spiel about what you actually spend in your day and why you're spending your day doing CFS work.
Yeah.
So back in 2018, I founded a company called Clara, which is clarasystems.com and we provide professional development and support services around FreeBSD and ZFS, including ZFS and Linux.
And so customers come to us when they hit a bug in ZFS and need help with it.
We sell support subscriptions, and we develop new ZFS features.
So, for example, we developed a feature to be able to delegate a ZFS data set into an LXD container on Ubuntu so that one of our customers could run unprivileged Docker inside a container, but using ZFS so that Docker would be able to use ZFS snapshots and so on.
So when they give, they take your pool in zfs, and in zfs you can have.
So it combines all those hard drives you have into one pool of storage, which you can build multiple file systems that share the free space.
So you don't have the problem of partitioning you used to have.
If you had a 20 terabyte hard drive and you decide, okay, I'm going to Split this into four 5 terabyte partitions and run the five different workloads on them, and suddenly one of them is only using 1 TB, the other one would like to use 6.
But now that's not where your partitions are.
And you have this problem of like, oh, I don't have enough space over here, but too much space over here.
By pooling your storage in zfs, it means that you can just use all the space where you need it, and so you can create one of these virtual file systems and then give it to a container.
And inside that container, the fake root user can only see that one and none of the rest.
So they use this to run a CI system for many different customers.
So each Docker workload runs inside a container and can't see the other ones, but does have access to actually create and destroy snapshots and clones and do all the stuff Docker needs to do to run really quickly, taking advantage of the copyright features of zfs.
And so, yeah, they came to us needing this.
We developed it and then we also upstreamed it, included it by default in OpenZFS.
So then when Ubuntu 22.04 came out, it included that feature and they were able to just run stock Ubuntu and have it in production.
Very cool.
So let's talk a bit about fundamentals in zfs, and you kind of alluded to already, I mentioned pools.
There are like three building blocks.
I guess they're V devs, pools and data sets.
Right.
And let's Speak a bit.
Well, maybe the fourth one that I'm not that I'm missing, but let's speak about the fundamentals there.
So he's like, get big data way.
Yeah.
So like we said, ZFS is a volume manager, so it can also basically do the RAID for you.
And so a V dev is one of your kind of RAID components.
So you take all the hard drives you have and then you can combine them in groups called V devs, and that V dev can have a transform on it that makes it do something.
So if you have no transform, you just have each disk as a separate V device, you've basically done a RAID zero, and it means if you lose any drive, all your data is gone, and that's bad.
So you can do a mirror V dev where you're going to have pairs of disks and each display every block you write to some disk.
The second disk in that mirror is going to have an exact copy of it.
And we'll come back to why ZFS does that better, using checksums in a minute.
But then it also has what it calls Raidz 1, which is when you combine any number of drives in a group and it can withstand the loss of any one drive and keep working.
That's the same as a raid 5, except for it has one slight advantage.
So with raid 5, there's this flaw called the write hole.
Whereas when you update something, it's going to write the new data to the hard drive, and then it has to write to a different hard drive the updated parity, so that if it loses that drive using the parity and the other rows on the remaining hard drives, it can calculate what that.
What data was on that missing drive.
Basically, it adds up the chunks of all of the drives together and gets a value and writes that in the parity.
And then if one drive is missing, it can take that thing, subtract what's on every other drive, and get back the value that would have been on the missing hard drive.
But because it updates the data one hard drive and then updates the parity separately, if the power goes out between those two steps, then the parity is wrong, but the RAID controller doesn't know it.
And so when it boots up, it takes that parity, subtracts the remaining drives, and gets back the wrong answer.
And so now the block you just wrote, isn't there some gibberishes, which is a combination of the new data and the old parity?
Right, Because ZFS is transactional, it doesn't have that problem.
And we'll get back to how that works in a second.
But it also has Raid Z2, which means you combine a bunch of drives and any two drives can go missing and it works.
So that's like Raid 6.
That's available on some hardware controllers.
And how many drives do you need for all of these?
Well, V devs, I guess.
How many V devs do you need for each of these to work?
I mean, let's call V dev a physical drive, I presume in most practical use cases.
So technically the V dev is generally the transform the group of a bunch of physical drives, right?
Right.
So for a raid Z1, technically you need at least two drives, one and the parity.
Although it doesn't make sense to do that with less than 3, because you might as well just do a mirror if you only have two.
Yeah, and the same with Raidz 2.
Technically you only need three drives, but it probably doesn't make sense with fewer than four.
One of the big additions that ZFS has is Raid Z3, which allows you to group together some drives and lose any three of them and keep going.
That isn't something any hardware RAID controller I know of can actually do.
And I may have limited two.
All you would need technically is four, but you'd probably want six or more for it to make sense where you're going to lose half of them and keep it.
Because the big thing is, out of that set, whatever the number is, 1, 2 or 3, that many of the drives basically not going to be used to store your data.
It's just going to store the parity to be able to reconstruct the data.
Yeah.
Okay.
So if you use four drives in a RAID Z3, you're only going to get the space of one hard drive.
Right.
Another thing that's different than most hardware RAID is it doesn't have dedicated parity drives.
Right.
It's not going to be that, you know, if you have six drives in a Raid Z3, that three of them are only going to contain parity.
Z spreads the parity out between the drives so that you get more the bandwidth of the drives when you're writing.
So that provides that.
And then.
So each V dev or group of drives is responsible for its integrity.
So it has its own RAID there.
But then we combine multiple V devs if you have enough hard drives, and those are basically just striped together.
And so if you lose any one V dev, you've broken the whole pool.
But, you know, if you've used Raidz 3 in each of your VDEVs then it'll be fine.
And so, you know, a big server I built many years ago is basically a petabyte of usable space made up of a bunch of v devs of 12 terabyte hard drives in a RAID Z3.
And so I think it was like 9 or 10 of these VDEVs.
And that way you get all this space and, you know, as long as you don't lose more than three drives from any group of 12, then everything will be fine.
And, you know, when a hard drive dies, you replace it with a new one and it rebuilds and you're okay.
Right.
So that provides that's the V dev layer, and then above that you have the pool, which is basically giving you all the usable space of all those drives put together.
And it means that out of that you can create these virtual file systems called data sets.
And they're the same thing as kind of like an ext4 file system or whatever.
And you could have multiple of them, but each one will show up as a dynamic size where basically they are the amount of data they contain plus all the remaining free space in the pool.
Right, right.
So if you have five of these file systems, each one will say that it has 20 terabytes of space left.
But if you write to any one of them, all five of them's free space will go down by the amount you changed.
So kind of over provisioning in a sentence.
Yeah, yeah, except for we're not actually lying and saying that we have more on any of them.
And as you write stuff to them, the free space on all of them will go down.
But as you delete something one of them, the free space on all of them will go back up.
Right.
Unless you have snapshots, which we'll get to in a second.
And so that allows you to have different settings on each data set as well.
So ZFS has a feature called transparent compression, where you can set compression on the data set that contains my home directory, but not the dataset that contains my music or something.
Right.
And what this is, as you write the data to ZFS, it will use a fast compressor like LZ4 or Zstandard to shrink the size of those.
And then on disk, it'll actually store the compressed version, and then when you read the file back, it'll decompress it first before it gives it to you.
So the application doesn't have to know that there's compression happening, but the file system takes care of it for you.
And then all your, you know, the Many, many copies of the ZFS source code I have on my server take a lot less space because source code is text and compresses really well.
Right.
And what's overhead?
Let's talk about compression for a second, because I think that's a super interesting one.
Like what's the.
On a modern server, let's talk about ZZ standard, like a basic compression or the basic one, I think, in ZPass.
What's the overhead, CPU wise, I presume would be the big impact.
Right?
Right.
So that's the interesting.
With LZ4, the overhead is quite minimal.
It can compress multiple gigabytes per second per core.
So as long as you have a couple of cores, you're probably going to run out of performance on your storage, even if it's NVME before you run out of CPU time.
Interesting.
And it can actually end up being faster because if you have 100 gigabytes of source code you're trying to or 100 megabytes of source code you're trying to write, and a regular spinning hard drive that can write 100 megabytes per second, then it would take you a second to save that 100 megabytes of source code.
If you can compress it at 2 gigabytes per second, so it takes a fraction of the second to compress it, and it compresses down to 50 megabytes, you write it to the hard drive and it only took half a second.
So it means technically, if you had more source code, you could write 200 megabytes a second to this hard drive if your compression ratio is 2x.
So in exchange for a little bit of CPU time, you could make your hard drive seem faster only if the data is actually compressible.
And the same thing applies when you read.
Now, when you read, you only have to read 50 megabytes in order to read all 100 megabytes of the data.
And LZ4 decompresses at over 10 gigabytes per second per core.
Right.
And so it can end up making a lot of workloads faster.
Interesting.
Are there any?
Because I believe compression is disabled by default.
So as of version I think 2.2 compression is on by default.
Okay, it's now to the point where it's so fast it doesn't have a penalty if you try to Compress things like MP3s or video files that are uncompressible or even encrypted files.
Computers are so fast now that it basically doesn't have a high enough cost.
Right.
There's also a feature in ZFS called Early Abort where it can notice.
I've been compressing this chunk for a little bit and I noticed that it's not compressing enough that it's going to be smaller enough to matter.
Basically if it's not going to save you at least 12% if it becomes obvious we stop and don't finish trying and just store it uncompressed.
Okay, so there's like.
Okay, interesting.
Yeah.
Because I mean if you are chucking compressed files on the compressed compressor, that's not going to make much.
Yeah, that's just going to happen.
It's just going to use up a bunch of CPU time.
Now LZ4 is so cheap it doesn't make a big difference, but it will also notice and give up early in order to save some of that CPU time.
Right, okay, so there's what you're saying essentially there is no reason not to use it.
Exactly.
It's now to the point where you might as well just have LZ4 on everywhere and it will help you where it can and it won't make a big difference where it doesn't help.
Right.
I would imagine back in the early days that was quite not the case.
Right.
Well, especially before LZ4 there was only basically LZMA which wasn't or not LZGB which wasn't as good.
And gzip.
And gzip can compress better, but it is slow.
Like gzip will top out at like 50 or 100 megabytes per second per core.
And yes, that will you will run out of CPU before you run out of storage bandwidth.
So gzip will really hurt you.
As far as CPU usage, Z standard supports 19 different levels in ZFS and so you can tune it from only use a bit of the CPU to use all the CPU.
But if you use up to 19, it will only compress at like a single digit megabytes per second.
So you only want to use that one when it's like I'm writing this once, I know it's compressible and I'm just going to keep it forever.
And it'll be worth it to spend the CPU time to compress it that much.
Right.
But we also added a feature I think in 2.2 or 2.3 where if you enable a really high Z standard level, it will quickly try to compress it with LZ4 first and see if LZ4 couldn't compress it, we assume Z standard won't be able to compress it and we just won't even try.
Whereas if it does compress it, then we'll let it.
And so it allows you to skip trying the heavy compression on files that are definitely not compressible, again to avoid the overhead and let you just kind of turn it on without having to worry that it's going to spend a lot of time trying to compress things that aren't important or not compressible.
Right.
Okay, so we talk about data sets.
One thing about datasets that I found important is that you can run encrypted datasets on top of a pool.
Right.
So you could have non encrypted or you could have some encrypted, Maybe speak a bit about how that works because that's kind of a relatively unique feature as well to zfs.
Yeah, so most block level systems have some kind of encryption.
So on Linux there's lux and on FreeBSD there's Gelly.
And these use an algorithm called aesxts, which is whole disk encryption, and they use a key and they encrypt the data on the disk.
With zfs, we wanted more granularity.
So yeah, we have these multiple different data sets.
And maybe I want my operating system non encrypted so it's easier to boot from and so on, but I want my home directory encrypted.
And so it uses AES gcm, which is the encryption normally used for like HTTPs, and it encrypts not the structural information.
So like the name of the data set and the ZFS specific bits are not encrypted because it needs to be able to work on those.
But basically the file names and the actual data in the files are all encrypted.
And so earlier we mentioned the checksum.
So ZFS stores a checksum of every block so they can verify that the block didn't get corrupted by your hard drive.
Or as we mentioned, when you're doing a mirror in traditional mirroring with hardware, RAID or md, RAID and so on, if the two sides of the mirror don't match, there's no way to tell which one is right and which one's wrong.
Yeah, instead of S, because we store a checksum separate from the data, we can read both pieces of data and see which one matches the checksum and know, oh, the SHA256 says copy two is the right one.
And so we'll repair copy one with the right copy.
When we encrypt files we split that checksum in half.
We keep the first half of the SHA256.
And in the second half, we keep the message authentication code from the encryption.
This allows us to verify both bits, the raw encrypted data on disk, that it's not corrupted.
And the Mac ensures that when we decrypt the data, we got the right data back and that it hasn't been tampered with.
This means that unlike whole disk encryption, where if you mount the file system, you have to have entered the decryption key and now everything's decrypted.
With zfs, if you have your home directory and somebody else's home directory, those are encrypted with different keys.
And when the other user is not logged in, they can unload their key and that data is actually at rest and encrypted and can't be accessed by anybody until they come and enter the key, maybe by logging in over SSH and using their passphrase.
And so that allows that data to actually be protected.
Whereas if you just used whole hard drive encryption, that really only protects you against somebody stealing the physical machine.
And then when they reboot and try to access it, they don't have the key and they can't decrypt the hard drive.
Whereas with zfs, you can unload this key.
And now nobody can access this data without entering the key first.
So it allows you to keep the data at rest and have the protection from encryption without having to power off the server to get the protection.
Right.
But importantly, because we split that checksum in two, it means that we can still run the repair of a failed hard drive on the encrypted data without needing the encryption key.
Oh, right, okay.
And so it can, it knows from the first half the checksum that, hey, that block is not right.
The hard drive flipped a bit or something and, or, you know, we've drive failed and we put in a new one and we can rebuild all the data and get the array back healthy without ever having to have the encryption key.
So it means the storage administrator doesn't need all the encryption keys, which was a big advantage.
Right.
So a use case that came up with this for one of our customers being a law firm was, you know, we have all this discovery evidence from a case.
The case is now over.
We have to destroy that evidence and not have a copy of it anymore.
But, you know, we can't be sure that our flash drives are actually going to erase anything we erase.
So overriding is not necessarily going to be good enough.
And if we use whole disk encryption, we'd have to Reformat the whole hard drive in order to actually ensure that it's gone.
But by having a different encryption key for each case's data set, they can just unmount it, destroy the key and never be able.
That data is not recoverable now, and it makes their life much easier for doing that.
So you mentioned two types of, I guess, unlocking.
There's the key and there's the passphrase.
One thing that at least I've been bothered with, both in the Linux world and in the BSD world, I guess, is that in most other operating systems you can use a TPM to unlock and lock and do disk encryption.
That is not quite the case.
It's kind of possible on Linux.
I don't know how it is in the BSD world, but do you want to say a few words about the narrative around that?
If that's something that is being worked on or particular encryption, I guess, yeah.
I've not really looked at the full disk encryption stuff in a while because I've been more focused on zfs.
But I did do support for the full disk encryption in FreeBSD's bootloader years ago.
So yeah, there is interest in doing the tpm.
So what ZFS has right now is you can load the key basically by typing it in by having a path to a file or having a URL to like an API.
So you can actually have, as the machines boot up, they call some other machine to say, hey, it proves with a certificate or something that I'm part of your network.
And that server says, okay, here's the key to decrypt that data.
So you type it in.
Yeah, but we would definitely be interested in using the TPM to store the key material.
Just no one's come with that use case.
So if you would like ZFS whole disk or dataset encryption to use the TPM, then definitely go to Cloudystems.com and click on the ZFS button and tell us you want that feature and we'll talk to you and definitely get that built because there's some interest in it.
We just would need somebody who has a commercial interest in it.
Yeah, no, I had a more personal interest.
I use ZFS on my Just Home server, my Boxbox server.
And just when you reboot the server and you have to manually type in a passphrase and you don't have a KVM or something hooked up to it, that is a pain point, right?
Yeah, exactly.
And that's why some people use that HTTP thing to fetch the key from another machine that's going to have stayed up, hopefully, or whatever, some kind of zero trust environment where before you can have this encryption key, you have to prove that you're not a compromised machine and you're the machine that's supposed to have this decryption key and so on.
Yeah, exactly.
Cool.
We talked a bit about what we kind of alluded to, scrubbing.
I guess maybe you should just knock that out as well, what scrubbing is and how it's different than other profile systems.
Right.
So that kind of depends on checksumming.
So as ZFS writes, any block of data you give it stores.
So in the indirect block where it actually says, you know, the fifth megabyte of that file is on this offset on disk, it has information about it like when we wrote it and the checksum.
And so a scrub is basically a patrol read where it goes through your whole disk and reads every block and checks that the checksum is still what it's supposed to be.
And so this allows it to detect bit flips on your hard drive or bitrod or any kind of corruption.
And so ZFS scans all the data and makes sure that the checksums are correct and fixes them.
They aren't.
Because if you have bitrot, it usually kind of.
It's creeping.
It starts small and gets bigger and bigger.
So when you catch it and fix it means that, you know, if you have a RAID Z2 where you have, you know, eight hard drives and two of them are providing parity, if you find a bit flip and fix it, when you find the second bit flip and fix it, you had all the data you need to reconstruct it.
Whereas eventually, if you had three bit flips, you wouldn't have enough data to fix it if all three were in the same file.
But if you once a month are reading all the data, making sure the checksum is correct, it means you fix each of those errors as they come up and they don't accumulate to the point where you actually have data.
Right.
Yeah.
I had Brian Kantra from Oxide on the show a few episodes ago, and he was giving some story from Illuminus when they were running that, and how they discovered driver issues and issues with IO controllers, essentially where they basically was, because of the intelligence of zfs, they were able to pick up things that otherwise would have gone undercover.
Really?
Yeah.
And we've hit similar cases.
We had one customer using like an early version of an all Flash array, and occasionally ZFS was finding this corruption.
And it turned out there had been A firmware bug and a rounding error or something meant that certain writes would get written to the wrong place.
So when you read from the right place, the data wasn't there.
And when you read what was supposed to be at the wrong place, it had been overwritten with this other data.
And ZFS was able to point it out and we eventually found the pattern and they were able to fix the firmware.
Nice.
Snapshotting is not a thing you've hinted at.
Yes, the really interesting one.
So other file systems in the past have had snapshotting like you think of.
Even KVM with Qcow2 has this concept of snapshotting.
But in other file systems, the way snapshotting worked is, you know, you have your normal file system, when you change a file, we just overwrite it in place.
But if snapshotting is enabled, then we'll detect that and we'll, oh, write the new change to a different place instead and keep both versions.
So that meant there was this cost.
So like, if you use the snapshots in lvm, as soon as you have one snapshot, your performance gets cut like in half.
And then you have a second snapshot and it's half of that.
And so if you have like eight snapshots, you're at like, you know, 10% of your original performance.
It's pretty terrible.
With ZFS, it works differently.
Every time we make a change to a file, an existing file, we always write the blocks to a new place.
If you have no snapshots, the old place after a couple of seconds just becomes free space and can be reused later.
But if we have a snapshot, then we know to keep that data.
So basically I mentioned one of the bits of metadata we have along with the checksum is the time when the block was written.
And this is measured in transaction groups since the pool was created.
So when you create a snapshot, it's mostly just remembering the time of the snapshot.
And so when you overwrite some data, if it's before that snapshot, then we need to keep it, and so it won't erase it and it'll just keep going.
So it means when you're writing while having snapshots or reading, there's no extra work to do.
We're actually doing less work because we're not freeing that space because it's referenced by the snapshot.
Then later when you delete the snapshot, it can go through and say anything that's older than the now oldest snapshot.
We don't necessarily need that anymore.
And so it can go and free it and make the free space.
That's very cool.
So the copy and write nature of ZFS also provides the other big thing.
Especially in the early 2000s, there was this problem where if your server ever got rebooted unexpectedly, when it came back up, it had to run an FSCK and check all the stuff to make sure your directories weren't corrupt or whatever.
Yeah, on bigger hard drives that could take days.
And have your operating system not fully running for days was not an option.
Eventually that was mostly kind of solved with journaling, but to a limited degree with zfs.
The way it works is using those transactions.
Any changes you make get accumulated in a transaction group and then written out.
And by default that happens every five seconds or more frequently if you're pushing a lot of data and it's getting full.
But this means that, you know, in an overriding file system, if you're in the, you know, just made a bunch of changes to a big Excel file and hit save and the power goes out halfway through saving.
On a regular file system, you would have overwritten the first half of the original copy of the program or the Excel file with the new version, but the second half didn't get written yet because the power went out.
So when you boot back up, you have half the new file and half the old file.
And correl Excel is just going to say, that's gibberish.
I can't make sense of that.
Whereas zfs, the new file was being written over here and we only have half of it.
And so that transaction didn't finish, so the checksum won't match.
So when it boots up, it'll just say, okay, that one didn't finish.
We'll go back to one previous that is completed and do that.
And then ZFS has this concept called the ZFS Intent log or zil, where any time an application asks for a promise that the write I just did is on disk before I continue, like a database would, that gets written to the xil and then when you reboot after the crash, the XIL will replay those changes and make sure that everything we promised actually got finished.
Like a wall login postgres.
Exactly.
But that for the file system and this way we go from the file system was perfect to the file system was perfect without ever having to be in that in between state.
Because it's in between state, we kind of roll back like a database.
And so it doesn't have this need to fsck.
Other file systems do super interesting.
The last feature that I wanted to cover, which I think is a very neat feature as well, is the ZFS send, which is kind of clever feature as well for like, shipping data across systems, really.
So maybe speak a bit about that.
Yeah.
So basically it allows you to serialize a file system into a stream that you can send over the network.
Technically, you could send it to a file and then receive it on a different computer much later, but usually it makes more sense to just do it directly, because having a giant file full of a stream of the file system isn't as useful as if you receive it, you have that same amount of space, but as a usable copy of the file system.
Right.
But its power really comes from its ability to do incremental replication, where after you've sent the whole file system based on a snapshot, if you make a newer snapshot, you can send the difference just between those two.
And that depends on what we had just talked about, those transactions and the transaction group id.
So when you created the first snapshot, were at, say, transaction 1000, and when you created the second one, were at, say 1100.
So now when you say send the difference between snapshot one and snapshot two, ZFS just has to look for blocks that have a birth time between those two numbers.
Oh, neat.
Okay.
And so we can just scan all those blocks and say, okay, any block that has a birth time greater than a thousand, but up to 1100, we're just going to feed it into the stream, and on the other side we'll just receive those.
And so compared to doing a backup with something like rsync, where it's going to walk through every single file in your system and run stat on it and be like, when did you last change?
When did you last change?
And only.
And if a file is huge, but it was touched, rsync has to read the whole file on your side and read the whole file on the far side and check the checksum of the chunks and then to figure out which blocks changed.
Whereas ZFS just looks at the time on each block in the metadata without having to read the data and says, okay, only these seven blocks change.
So it just sends those seven blocks and it's done.
Interesting.
And so ZFS is able to basically saturate the network and do the backup in a couple of seconds, where Rsync might take a whole day to copy that same couple of blocks, because it has to check every file, and then if the file is newer, it has to read the whole file on both Sides and decide which blocks are actually different.
Whereas ZFS just natively knows because it has a timestamp for each block of the file instead of just the whole file.
Right.
And I guess if you send it on a network, you would just pipe it through something that's encrypted and then receive it and decrypt it.
Yeah.
Okay.
Well, also, if your data set was encrypted on zfs, you can do a raw send where it'll send the encrypted version without decrypting it to the other side.
And then that way the data is never decrypted on the far side.
And so if they don't have the key to decrypt it, they only have a backup of your data that they can send back to you, but they can't ever mount.
Right, okay.
The thing lots of people are after.
Yeah.
Are there any hosted services I know obviously a fellow Canadian call in is running tarsnap.
Are they supporting that ZFS send as torch now?
Because that would be an interesting service offer to basically ship.
They're not.
So the two services I know of are ZFS Rent, where they will basically rent you a hard drive in a VM that you can do that to and then rsync.net has a thing where they will basically stand up freebies to your Linux in a VM and sell you five terabytes of space that you can ZFS receive your encrypted data sets to.
Or not encrypted if you don't care.
Interesting.
All right, so the last thing I want to round up the episode on is Tales from the Trenches.
Obviously you've been around the data world for quite some time and I'm sure we've seen a lot of horror stories around this.
So I kind of want to hear from your side what's the craziest and more like bizarre data recovery missions you guys been on trying to recover data on GMS clusters or in general?
Yeah, we've done quite a few different ones.
We just did a webinar on Halloween that we'll drop a link in here for, but if you go to clarasystems.com and click webinars, you'll find our Halloween horror stories.
So I won't cover one of those because they're all really good, but I just did them.
So another one we did, there was an interesting one at a university where they had just made a bunch of changes and we had helped them with those and it was fine.
And then so they had a high availability system where they'd have a JBOD full of hard drives connected to two different servers so that if one server goes down, they could import the pool on the other one.
And they had a bunch of these pools and they had just numbered them like everybody does.
This is not the same as the war story in the Halloween one where a similar thing kind of happened.
But after importing them, after doing the upgrade, they accidentally did import pool 4 on server 4 and then went to server 5 and accidentally typed import pool 4 instead of pool 5.
So now pool 4 was mounted on two machines at the same time, which isn't supposed to be able to happen because they didn't enable the feature that stops it because there's a trade off for it.
But so now every time there's a new transaction group, server 4 is writing to the hard drive and then server 5 is writing different information with a different transaction group number over top of that.
And so they're just sitting there writing over top of each other, somewhat being correct and somewhat not.
And also meaning that if you wrote different data on different servers, they're going to not know that the other one has used a certain bit of space.
So think it's free and allocate it and basically overwrite something Server 4 just wrote with something different that Server 5 just wrote.
And you can really corrupt things really badly.
They managed to catch it after I think it was about 25 minutes or so.
And so they shut everything down and called us and things were quite broken and we had to invent some new tools to be able to fix it, but eventually got to the point where we could do a version of ZFS sen to copy all the data to a spare machine they had so that they could get most of their data back, except for obviously the newer stuff they had overwritten and a couple of pieces got damaged.
But we managed to get most of the data back by basically being able to do ZFS send.
And the big advantage there over copying it some other way was just the speed, the fact that we could saturate their 10 gigabit network.
And when you're talking about hundreds and hundreds of terabytes of data, it's really slow to copy.
So you really need a way to try to do that quickly.
Good stuff.
This has been very helpful and I think hopefully opened the eyes to ZFS to a new audience.
And I think it's well worth shouting about because I think it's a fantastic file system.
So I'm very happy to wider adoption for it.
So thank you so much for coming on the show, Alan.
Very much appreciated.
Any last words about Clara?
Something you want to say, you want.
To draw attention to?
If you have.
If you want to learn more about zfs, I do a weekly podcast where we tend to answer quite a few people's questions.
They write in about ZFS.
So that's 2.5 admins.com so 2 and a half admins.
It's a podcast with myself and another ZFS admin, Jim Salter, and a host who's trying to become a sysadmin.
So we have two and a half admins.
So if you're into sysadminning in general, or ZFS specifically, you want to check out that podcast that comes out every Thursday.
And there's also we have a website, practicalzfs.com where we have a discourse set up replacing the old Reddit rzfs.
And so it's a place where people ask a lot of ZFS questions, we answer them and kind of collect a whole thing there.
But yeah, if you need support with ZFS or feature development or same for freebsd, then do hit us up at clarasystems.
Com.
Amazing.
Again, thanks so much for coming on the show, Alan.
Have a good one.
Talk soon.
Cheers.
Bye.
Found an error or typo? File PR against this file or the transcript.