Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
coreboot Uncovered: BIOS Security and Vulnerabilities with Matt DeVillier and David Hendricks
Play On
24 MAR • 2024
1 hour 11 mins
Share:
In this episode, I’m joined by Matt DeVillier (Mr. Chromebox) and David Hendricks to explore the fascinating world of coreboot. Their combined experience from companies like AMD, Facebook, Google, and Amazon offers unique insights into how this open-source BIOS technology is transforming firmware development.
We start with Matt’s journey from hardware enthusiast to coreboot expert, and David’s early work with project founder Ron Minnich. What particularly caught my attention was the contrast between coreboot and U-Boot, especially in how they’re used in Chromebooks and servers. Their explanations of Secure Boot, verified boot, and UEFI Secure Boot reveal the critical role BIOS plays in system security.
The conversation gets especially interesting when we dive into recent BIOS vulnerabilities like LogoFail and PixieFail. Matt and David share candid insights about supply chain security and the importance of transparency in firmware development. Their discussion of Software Bill of Materials (SBOMs) highlights how crucial firmware integrity has become in modern computing.
I was particularly intrigued by our discussion about transitioning to coreboot. Their emphasis on early engagement with Original Design Manufacturers (ODMs) and the potential of RISC-V shows both the challenges and opportunities in open hardware. We also explore how coreboot contributes to sustainable computing, especially in developing countries.
If you’re interested in firmware security, open-source development, or sustainable computing, you’ll find plenty of practical insights here. For those wanting to dive deeper, here are some valuable resources:
- coreboot’s homepage - Documentation, consultant links (coreboot IBVs), and hardware vendor information
- MrChromebox.tech - Matt’s custom coreboot distribution
- Converged Security Suite - Tools for Bootguard provisioning
- goswid - SBOM generation tool for coreboot
- Developer Information for Chrome OS Devices - Comprehensive guide for ChromeOS devices using coreboot
Transcript
Show/Hide Transcript
Welcome back to Nerding up with Victor, where today we're delving into the realm of coreboot, the forefront of open source BIOS technology.
I'm excited to introduce you to two exceptional guests.
First up, Matt De Villier, a firmware engineer and open source enthusiast, celebrated for his contributions to coreboot at Purism and amd.
In the community, he's also recognized as Mr.
Chromebox, a moniker under which he has made significant strides in custom firmware development.
Joining Matt is David Hendricks, a respected figure in his influential work at giants like Facebook, Google and now Amazon.
His expertise spans a broad spectrum of hardware world with a special focus on coreboot.
Together these experts are poised to illuminate the complexities and significance of coreboot in today's tech landscape.
Let's dive in and no doubt welcome both of you.
Thanks Victor.
Super excited to have you on.
Super excited to have you on the show.
And is there anything you guys want to add to the introduction?
Anything that listeners or should know before we dive, before we start the conversations?
I'm David.
I have been involved with coreboot kind of on and off since almost the very beginning of the project.
I started off as an intern working for Ron Minick, who was the project founder.
This is around the year 2000, 2001 ish.
So I've been involved for a long time.
I was a high schooler at the time, didn't know anything about coding, didn't know anything about bios, but I like to build my own computers from scratch.
I was a gamer and overclocker, and at the time Los Alamos National Lab, which is a high performance computing lab, needed cheap labor to build servers using off the shelf components.
So my job there was basically to purchase processors and RAM from sites like Newegg, just off the shelf stuff, put them together, throw them in a rack, and then the rest of the team would put the clustering software like Beowulf on it.
And of course we'd run coreboot and show that you could actually build very good high performance computing systems out of cheap parts.
Flash forward, I managed to land a job at Google.
A couple years into that, I found my way into the Chrome team.
They were looking to make devices like laptops.
And so I was the first firmware engineer on the Chrome OS team.
Stayed there for about seven or eight years, then went off to other companies to try and find my roots back in servers again and get open source firmware pushed on servers.
So that brought me to my current place and my previous employer.
So that's about it for me.
Matt, of course is Mr.
Chromebox, so I'm sure people want to know a lot about his background too.
Well, my background's very similar to David's.
I grew up being a hardware tinkerer, building my own computers, overclocking, playing games, things like that, the Doomquake, the early days.
And out of college got a job just doing coding day to day and didn't really code too much as a hobby.
But I also was into home theater at the time and building custom media centers for people using XBMC, which is now Kodi.
And so around early 2010s when the first Chrome OS devices came out, the first Chromebox or actually the second generation Chromebox was a really excellent candidate for a media center.
And so I bought one of those with the intention of hacking it and turning it into a custom media center that I could resell to my customers.
And I didn't know anything about firmware at the time, but you know, there was somebody else online who offered firmware that you could download in Flash and the guy didn't support the new Chromebox, but he built something for me and gave it to me to Flash and it bricked the box.
And he's kind of like, well, you know, I don't know a whole lot about these devices.
I just know how to compile and distribute them and support, you know, the one or two devices that he had.
And so I was kind of left to my own devices to figure out, you know, what went wrong, how to unbrick it.
So I had to go out and order a bus pirate to be able to reflash the firmware.
And the Corebook community on IRC was extremely welcoming and helpful, especially a lot of the Google engineers who frequented it at the time.
And so I just started playing around with that as a hobby and eventually got the box working, made some improvements, submitted some patches, and essentially distributed my work for other people who are looking to do the same thing, use the Chromebox as a media center.
And you know, as time went by I just added support for more and more devices.
You know, there were some other community members who kind of supported some Chromebooks and eventually I just absorbed their work.
And you know, as of today I now support every intel based Chrome OS device since 2012.
So a good decade plus worth of devices, it's 250 and counting.
Wow.
Any estimate on how many people use the Mr.
Chromebox version?
You know, I used to track downloads, but I don't anymore.
You know, it was over it was a couple hundred thousand, last I checked, as well as some commercial companies that use it for repurposing chromevox for their own digital signage purposes and things like that.
I know one company that I did some custom work for that has, you know, 50,000 devices out there just alone.
So, yeah, it's substantial.
Yeah.
One of the interesting places that Chrome OS and by extension coreboot have found themselves is in things like kiosks, point of sales devices, digital signage.
That was actually one of the last projects I worked on at Google because, you know, it boots very quickly.
You know, sometimes you might walk around or you might be driving around and you see a sign that's, you know, it's like Pixie booting or it's at the blue screen of death or something like that.
And, you know, advertisers hate that because they're not making money on that billboard space.
So, you know, little Chrome boxes and coreboot fit really well.
Something that's very simple, very fast.
If, for whatever reason, the hardware goes down, it comes back up and you can be up and running again in a couple of seconds.
Absolutely.
I mean, we've been working with Signage for the last decade, and definitely it becomes a bit of a confirmation bias when you see all these broken screens around the world.
Right.
So absolutely, I can relate to that.
Cool.
Let's dive into coreboot.
So.
So let's start with a perhaps silly question, but why does it matter?
Why does coreboot exist and why does it matter?
So I guess I'll start because I was around towards the beginning, and then Matt can probably fill in details from later on.
So at first, as I said, actually, this is, I think, the project's 25th anniversary.
Oh, wow.
Yeah.
Yes.
It's been around since, you know, the very late 90s, essentially.
And of course, back then, if you wanted x86 BIOS was.
It was just legacy BIOS all around.
Even UEFI didn't really take off until the late 2000s.
And so for.
So.
So the story at Los Alamos National Lab was they were.
They were trying to bring up one of these clusters built with cheap x86 components.
They had, I don't know, like 1200 nodes or something, usually a power of two.
So like 10, 28 nodes, 2048 nodes.
They rolled the racks into the data center, they powered them on, and each and every one came up with a prompt that said, no keyboard found.
Hit F1 to continue.
So the way they had to solve that at the time is you give a team of about a dozen PhD computer scientists and physicists, keyboards, and they fan out and go out to all the aisles, plug them in, hit F1 to continue.
Problem is, the high performance computing software at the time was not very resilient to failure.
And so oftentimes if one node had a problem, you know, like maybe a, you know, a hard drive crashed, you know, they didn't have NVMEs or SSDs at the time.
So these were a big point of failure.
You know, you'd reboot one node and you'd have to restart the whole cluster and then go through that process again, send a bunch of PhDs out to the aisles to plug in keyboards and do whatever.
So Ron was basically like, you know, screw this.
And he thought, hey, you know, what if we just put Linux in that Flash chip?
Linux, you know, even at the time it, you know, it was pretty smart.
It could do a lot of PCIe initialization, did a lot of hardware initialization.
So the very first version of coreboot was basically a function that would, it would set up the C runtime environment and then just basically try to jump straight to a Linux ELF image.
And it did not work.
It turns out initializing hardware is a little bit more complicated than turn it on, jump to Linux.
But that's kind of the genesis of the project.
That's why it was originally called Linux BIOS up until I want to say around 2007 or 2008 was because the idea from the very get go was do the absolute minimum you need to do to get Linux booting.
The idea kind of took off and people started using it for other operating systems.
You know, they even got Windows booting pretty, you know, maybe mid 2000s or so.
And so the name Linux BIOS, you know, it wasn't necessarily Linux centric at the time and it wasn't a bios.
So the name Linux BIOS didn't make sense, so they changed it to Core Boot.
So that's how it started was kind of the inflexibility of fixing some of these intractable problems with the bios.
And then later on people are like, hey, this is kind of cool.
It's open source, we can hack on it, we can customize it.
You don't have to pay tens of thousands of dollars in royalties or consulting fees just to get one little piece of hardware running.
And then from there it kind of grew and it was really fast.
No legacy code.
One of the things that at Google, when I was showing coreboot off to the Chrome OS team, I was Showing it off on early Intel Atom boards and Viya Nano boards.
Viya was another x86 CPU vendor.
These things could literally hand off, you know, boot up and hand off to Linux in, you know, within 250 to 500 milliseconds.
Wow.
So that impressed the team.
And that's what got it to be, you know, on the plan of record for Chromebooks at the time when, you know, a typical Wintel Netbook would maybe take, you know, 10, 15, 20 seconds before it would actually, you know, even begin to start booting Windows, much less get you to a prompt where you could log in.
So Google was very keen on getting people to, you know, getting people to the prompt, logged in and browsing the web as quickly as possible.
And how did that transpire from the early version of the Chromebooks?
Were they running on coreboot from the get go or how was that adopted internally?
It actually took a couple of generations.
The first two generations used a proprietary UEFI distribution.
And I'll give them credit.
The company that we worked with, actually, they were very responsive.
Typically in the BIOS world, you know, companies have a bad reputation for not responding to emails or not fixing bugs for most of the time.
But, you know, at the end of the day there, you know, the team, you know, they were pretty keen on open source.
They liked the idea of open source.
Coreboot was still able to boot faster on the hardware that it worked on at the time.
And we also wanted something that could be ported to other processors very easily, the proprietary bios.
At the time, it really only worked with Intel Nanos, I'm sorry, Intel Atoms.
And basically if you wanted it to work on some other intel processor, you had to get another version of the code base.
Or if you wanted to work on an AMD processor, you had to work it on yet to get another code base and forget about arm.
So ARM was something that, you know, of course, Chromebooks being relatively cheap, inexpensive products from the get go, you know, were interested in evaluating arm and we didn't want to have to deal with a whole other code base, whole other set of bugs, whole other security model.
So the portability of coreboot really helped out a lot.
And, you know, we started on x86.
Coreboot was good at x86.
U boot was there as well.
It would at the time, it would have taken a lot longer to get intel support added to U boot than to have ARM support added to coreboot.
But, you know, they're both fine code bases.
Yeah, walk me through the like.
Because obviously if you play with any embedded platform, you've surely come across coreboot and you've probably.
Well, at least you come across U Boot and definitely, most likely coreboot.
Talk to me a bit about the difference between the two.
And like coreboot is.
Well, U boot is a bootloader, not just necessarily a bias measure.
Like walk me through the difference between the two and how they fit together.
Sure.
So they're both open source firmware projects.
And this is something Matt and I were talking about the other day is probably every year or so we see somebody come in with the idea, hey, why don't coreboot and U boot just merge together?
Yeah.
And I think the main thing is that they're both good at what they were designed for.
Coreboot originally ran on servers, not just Xeons like old x86 Xeons, but also DEC Alpha, digital equipment Corporation Alpha, if anyone remembers that, Blast for the Past and PowerPC.
But they were servers and they were meant to have PCI and so a lot of the resource allocation, the device discovery was built around pci, whereas with YOU boot it was really targeted more at embedded platforms where it's extremely simple.
You basically, you know, you're just directly peeking and poking at registers through known MMIO addresses for the most part.
And coreboot brings along a fair bit of framework to deal with PCI architecture.
U boot is very condensed, it's very compact, and the U boot people liked it that way.
I don't think they wanted to pollute U boot with any x86 isms at the time.
Okay, so that's fair.
Yeah, let's let Matt jump in with a few words on this as well.
Yeah, I think that's probably one of the biggest differences that coreboot is designed to discover connected hardware at runtime and dynamically configure it.
And U boot assumes a fixed hardware configuration.
And so when you have that, you can make a lot of assumptions and you can eliminate a lot of possible code paths and you know, it's essentially a simpler design model in a way.
And I guess the other main difference between coreboot and U Boot is designed to both initialize the hardware and be the bootloader, whereas coreboot is only designed to do hardware initialization and then to hand off to a second stage firmware payload which does the actual bootloading.
And that, I guess, scope of design is also a big differentiator between the two projects.
Okay, yeah, that definitely help clarify things.
And in coreboot today, what are the most common use cases.
We spoke about Chromebooks.
I know there's plenty of support for the newer thinkpads.
What are the most common use cases?
Is Server the most common use case or is it Chromeboxes or Chromebooks?
What does that deployment footprint look like today?
Or by guesstimate, I guess I would say by far.
Right now Chromebooks are the majority of the deployed devices running Core boot.
There are other x86 laptop vendors which ship devices running core boot.
But you know, by shipped volume of devices.
The nothing touches Core Boot or Chromebooks.
Just there's I don't know how many millions sold every year.
But yeah, it would dwarf any other class of devices.
Yeah, it is kind of.
I was just going to jump in.
It is kind of interesting though.
You know, we are an open source project and we don't build any telemetry or information gathering into the firmware at all of course.
So occasionally we do come, you know, we do hear about surprises where you know, someone's using coreboot in a, you know, some industrial machine.
I actually, Werner Z, he's one of the core Boot leadership members, works at Siemens, the big industrial machinery company.
And they put coreboot in a number of their products.
You see them, you see some core boot systems and like thumbprint scanners like at airports, aerospace applications.
So it's very distributed.
It's in probably more places than you think.
But as Matt was saying that the single biggest well known distributor is Chromebooks essentially and then followed by, you know, followed BY probably the Mr.
Chromebox distribution going out to a bunch of people and then you know, system 76 laptops and some other, you know, niche products.
Yeah, okay.
So I must admit that I had stopped paying attention to biases after I guess switching to macOS in the early 2000s or mid 2000s.
And then I guess didn't really think about biases until a few years ago when we started to build our own hardware screenly and we started doing like security assessments.
And that's when it dawned on me when you started do secure boot and all these things that actually I'm trusting this blob that I have no idea what it's doing and I have no idea how it's configured to do all the most sensitive things on the entire device.
And if I can't trust that, I can't really trust anything that happens past that step.
And that's really when I started getting excited about coreboot really.
And that's why we are trying to get coreboot running on the next iteration of our signage player, because it's really, like, the only way we can, with confidence, say that I trust what's happening from you power this device on until it enters user space.
Right.
Because I think it's almost crazy that nobody really talks about that, at least from my vantage point.
And I'm not sure what you are.
How you guys have seen that in general, your reflection on that in general.
Yeah.
I got to give the BIOS vendors some credit because oftentimes these PCs and a lot of devices, they work well enough that most people don't really have to think too hard about it.
But it's when stuff goes wrong or you start actually digging into the surface and realizing that, oh, there's no way I can really tell what it's doing under the covers, then you start to realize, oh, wow, this is actually a huge problem.
And especially if it's your job to make sure that it's secure, that it.
You know, we've been seeing some vulnerabilities come out lately.
I'm sure we'll talk about those later on.
But, yeah, when.
When it's your.
When it's your job to make sure that doesn't happen and you realize, oh, I can't.
I can't audit this.
I can't look at the source code.
I can't verify that it's doing what it's doing.
I can't verify that, you know, somebody way deep in the supply chain didn't put malware into it.
Exactly.
Then it's called Learn.
Learn to Helplessness, I think is the term.
Right.
Where you just kind of give up thinking, well, you know, it's.
It works well enough, and if there's a problem, then I.
I'm really not the guy to solve it.
Yeah.
And I.
And I think unless you are of the size of Google, Dell or Lenovo, you do not even have a direct relationship with your BIOS vendor.
Right.
Your contract manufacturer would have a direct relationship with them, but you would not even know.
You would not have the software to even configure the bios.
They would do it on your behalf.
Right.
So it uses another failure point in the supply chain sector.
Right.
So that's kind of like why I was really keen on having you guys on the show here to just explore core bios.
Open core.
I can't speak right now.
Core boot.
I mean, and just, I guess, raise awareness for as well, because I think there's so much interesting stuff going on in here.
So we kind of talked a bit about security already with some vulnerabilities.
But before we dive into these vulnerabilities that we have seen the last six months or so, let's talk Secure Boot and how that actually works.
Like what happens when I turn on Secure Boot and what walk me through like from boot process all the way through how that happens and how you can actually trust what happens and maybe we can open with that.
And I'm not sure what the best format to do that in, but I'm curious because they're walking that through the entire chain for serious events that happens.
Now, when you say Secure boot, are you referring to UEFI Secure Boot or one of the many Secure Boot implementations that exist within Core Boot?
That's, that's a good point.
I was thinking about generic Secure Boot, but we, I'm happy to explore either of those or like how they differentiate as well, because that's an area I don't have a ton of expertise in.
I've also met.
Well, I think David can probably speak better to the Google Secure Verified boot model that is used on Chromebooks.
Okay, so yeah, let's see.
This is a fairly big topic.
So we'll probably do, you know, we'll probably jump around a bit, so try to bear with us.
So, yeah, so two main concepts.
There's measurement, where you know, you have one, you have the first thing that runs measure the second thing that runs before jumping to it.
And measurement, you know, just means, gets a, you know, calculates the cryptographic hash and then if the hash matches what's expected, then okay, cool, the, this works.
I'll jump to it.
It's essentially a glorified checksum.
And then that measures, the next thing jumps to it.
That measures the next thing jumps to it and so on, you know, from the reset vector on up to your operating system.
And then there's the second concept, which is signature verification.
So that doesn't tell you about the integrity, it doesn't tell you anything about like the contents of the binary.
It just tells you who it was distributed by.
Right.
So like if you're just like if you sign an email and send it to a friend, you know, the signature is not necessarily checksuming your email, it's just telling your friend that yes, this email came from Victor because it's signed with his keys and I can validate.
Yeah.
So we do support a couple schema in Core Boot to deal with these.
The Verified Boot scheme that Chromebooks used, I was in on the early design of that I'm sure they've made improvements since then, but one of the main things we wanted was to avoid hardware like locking ourselves into a particular chips security model.
At the time I think bootguard was just kind of on the roadmap at intel and of course they were talking that up.
The ARM vendors of course had all their own secure boot schemes and we decided no, we want something that's simple that we can reuse across the product line and not have multiple security models.
So I pointed out that Flash chips like Spy, nor Flash chips that are pretty much ubiquitous for x86 firmware typically have a write protect pin.
So you can partition the Flash chip.
You can say I want the upper half write protected and the lower half will be rewritable like field upgradable.
And so that's what we did.
We put a read only copy of Coreboot in the upper partition on x86.
The upper part is where the reset vector is.
You hit the power button, the processor starts executing code that's basically in the read only partition that verifies that the next thing that's going to be jumped to is valid and then it jumps to that.
So the read only copy verifies the rewrite copy and then it jumps to it as soon as possible so that we can push field upgrades.
And this was a very simple model that allowed us to use it across architectures, you know, amd, Intel ARM platforms.
There was even a MIPS platform in the mix at some point and the same security model worked all throughout.
And another advantage was that it was easy for the owner of the device to take over.
This is actually something very important.
I got to give props to the original Chrome OS team because at the time the PlayStation 3 had just been hacked and Sony was touting it's oh, it's unhackable or whatever.
And then this teenage kid, George Hots, Geo Hot is his username I think managed to root it and own it.
And then from there you had, you know, universities buying PlayStation threes below cost because of course they make their money on games and they get this pretty good hardware that they could run their high performance computing simulations on.
So, so eventually so.
So the idea tying that back to Chrome OS was like, okay, people are going to try to hack this thing.
We don't want to lock ourselves in, we don't want to, you know, back ourselves into a corner.
We want owners to be able to unlock the device.
So early Chromebooks actually came with a physical switch.
You could open up the case, void the warranty that was totally cool.
Flip a switch to disable write protection and then put the case back together.
That would also have implications like the keys will get wiped out and things like that.
So when the user booted up, if someone was trying to do this maliciously, they would see a screen that would warn them essentially.
And they couldn't log into their Google account because the keys were wiped out essentially.
And that was just to thwart physical attackers who might try to put a malicious BIOS on and install a key logger and things of that nature.
So that was the idea with Chrome OS's verified boot early on, was that we wanted that redundancy both for security and for fail safe reasons.
We wanted it to be simple, we didn't want to tie ourselves to any particular process or vendors implementation and we wanted people who were interested in owning the device to be able to open it up, flip a switch, they own it, they can wipe out any trace of Google, they could install Windows, they could do whatever they want.
Right.
So that's kind of the genesis of that model.
And, and Matt of course he's been doing a lot more with UEFI Secure Boot so he can talk more about that.
Well, I don't want to speak too much about that because I'm, that's not an area of my expertise.
I would say my Mr.
Chromebox implementation of Coreboot with a UEFI bootloader does support UEFI Secure Boot, but I wouldn't say that it meets all the technical requirements.
It just implements enough of the feature that Windows says okay, secure Boot is on and enabled.
Which is kind of interesting that Windows just simply trusts a firmware flag and a couple other things to ensure that your OS and your device is booted securely and that there's no real verification there.
But the, I guess the main difference between the Google Verified Boot and uefisecure boot is that Verified Boot is from the moment the processor turns on and everything is verified forward, including the bootloader.
Whereas UEFI Secure Boot depends on other security technologies to verify that the BIOS is secure or not compromised.
And UEFI Secure Boot really just looks at the bootloader stage.
Right.
And for uefi that's where you have the concept of sign with Microsoft's root.
That's only some vendors do have their.
I know Canonical got their ability to secure boot with Microsoft keys, but if you install your random Linux distro, you have to ship your own keys.
Do you want to speak a bit about the concept between the two and like enrolling your own keys versus using Microsoft keys and how that fits into coreboot in general.
Well, one of the more interesting features of Google's verified boot is that you can go ahead and replace the keys with your own should you want to run your own S, your own OS and still have all the security features that were designed there.
You don't have to weaken the security model in any way in order to be able to do your own thing.
I wouldn't say that's terribly different from UEFI secure boot model, where you have Microsoft keys and other vendor keys, and then you also have user keys which can be supplemented.
But I don't think there's any way to completely eliminate the existing keys other than the revocation list, which you're dependent on your.
Your BIOS vendor to keep updated.
And that has been a major pain point in the UEFI world, where Microsoft will blacklist certain things in the revocation list, but that doesn't get pushed out to the IBVs will update it.
Maybe the ODM and OEMs don't.
But how long does it take to actually get into the end user's devices?
Right, right.
Yeah.
I would imagine the vast majority of consumer devices, unless there is a automated updating process, they probably never updated by us.
Right.
So that's a very good point.
Yeah, cool.
Yeah, go ahead.
Just to riff on that point a bit, Hardware does not age very well.
It is not like fine wine.
If you have a laptop that's two or three years old, it's probably considered to be end of life by the vendor.
And so BIOS updates are going to be very rare.
And of course, anytime you push an update, there's a chance that some number of devices just won't come back up.
So that's another reason vendors are very reluctant to push updates.
I've seen this in data centers a lot.
We think everything goes smoothly.
You reboot the server and it just doesn't come up for whatever reason.
And then a bunch of tickets get filed and you have to debug.
so.
So yeah, it.
It's one of those learned helplessness things where you see new vulnerabilities come out and you're just like, man, I hope my vendor, you know, pushes a firmware update.
And it could be days, could be weeks, could be months.
You never know.
Whereas, you know, it, in the ideal world, if everyone could actually just, you know, manage this themselves or at least trust a vendor like Red Hat to push a firmware update through Firmware updates.
I don't know how to pronounce it then.
Then at least you could be like, okay, well, at least I'm pretty sure that, you know, Ubuntu or Red Hat or whoever's going to push an update in a timely manner, even if the hardware vendor has already forsaken this thing.
But that's just not the world we live in.
So, yeah, let's talk, because I had.
One of the things I wanted to cover was exactly that.
Firmwepdate, or however you want to pronounce that.
Right.
It's.
It's something that I think very few vendors actually adopted.
I did look into it slightly and it's not that challenging to kind of enroll yourself, but it seems to be very little interest by vendors.
I think Lenovo does it, I think for some of their devices, and probably Dell does for some devices.
But do you guys have a gauge for how popular that is for core boot updates or optics in general?
Like what?
Adoption of that in general?
I would say the adoption rate in general is probably in the single digits, percentage wise.
You know, as you said, there's a couple of vendors who do support it for some newer devices, but it's by far from universal.
And I think it's probably our best hope for a universal distribution method for devices running Linux.
But for Windows, they do have the option of distributing those updates via Windows Update as well.
I believe as long as the BIOS supports UEFI Capsule updates.
Okay.
For coreboot devices.
There actually are some vendors who do support that for their core boot updates.
That's not something that Google does for Chromebooks because they have their own mechanism and since they control the entire stack, it's not something they need to use.
It's something I do plan on using for my Mr.
Chromebox updates now that the scale is kind of getting out of hand for me to do it manually.
How have, how far.
I mean, in terms of adopting that, is that something that you reckon is a massive hurdle for adopting in the distro, or how have you.
Have you looked at that so far?
Because I'm curious about how easy that would be to actually adopt.
When you say from a distro, you mean from a given Linux distribution?
Yes, from your distribution.
Yeah.
So enrolling the bias, imagine you have a fleet of devices, you might have your own BIOS for those adopting that workflow into your distro, that if you do control the distro, like in your case, Mr.
Chromebox, you can adopt that, right?
Because you control the entire operating system.
Like how.
What's the difficulty level of actually enrolling that?
I don't think it's very high.
I mean, I think there's some XML metadata that is used to version, hash, describe the update mechanism that's used because the firmware update daemon does support multiple update mechanisms, including using Flash rom, which is what most coreboot based firmwares use for updates.
So I don't think it's that challenging to do, you know, for a handful of devices doing it for, you know, a large number of devices, it's just me setting aside a couple of days to set up some scripts to generate the necessary files and that sort of thing.
I think it's probably a fairly small invest or it's a large upfront investment for a, you know, a large return down the line.
Right?
That's fair enough.
That's fair enough.
Cool.
Let's dive in a bit more into supply chain security because we kind of touched a little bit on that already.
And obviously to me that's one of the biggest value propositions for coreboot in general is the whole supply chain.
You can trust everything.
And David, you sent over some content to me to read about how you guys think about that at coreboot and how you kind of trust the entire flow.
Maybe we can start with that.
How does supply chain security thinking look like inside of coreboot and I guess how transparent that is.
So you can trust what's going on there.
Yeah.
Yes.
This is a interesting topic and I think it's getting a lot more attention these days, especially since in the US there's executive order from the White House to direct, you know, some Alphabet soup agencies to start paying attention to where their software is coming from.
And SBOM is spelled out as a requirement.
So it's no longer good enough to just say, you know, we bought a bunch of PCs from such and such vendor, it's running Windows, we're just going to trust it.
So, a bit.
Sorry, I lost my train of thought for a moment.
So basically the difference between coreboot and, you know, what you typically get on a PC, it's very much, you know, just the typical open source story of you have all the software in front of you can compile it, you can audit it, you can do whatever you want with it.
Most people aren't going to do that, but it's nice to have that capability.
Whereas with proprietary firmware, of course, you get a blob from your vendor and you trust that they didn't put malware in it.
Or you trust that it's patched, you don't really know, but you just kind of take their word for it.
And you have no change log whatsoever, usually.
Right?
Yeah, yeah.
You might get some handwritten release notes that say, you know, we added this feature, we fixed a bug.
But, you know, you don't have a git history the way you do with coreboot or U boot or other open source firmwares.
And, you know, this is something that people, you know, C level executives are starting to pay attention to.
Thankfully, it's taken them long enough.
But so, yeah, I mean, the way we think about it from coreboot's perspective is it's open source.
We encourage people to take advantage of the security features that we have.
Where you can measure your firmware, you can see how it boots.
You can measure each component, you can store hashes in a TPM and read it back to verify that you're running the thing that you think you're running.
You can sign your firmware with keys to ensure that it's coming from a trusted source.
You know, you can layer on these levels of security.
Whereas again, with proprietary firmware, you just take what the vendor has.
They probably fuse their key into, you know, the intel me or something, and you not only have to trust them, but you're stuck with them even if you decide that you don't trust them.
So if a vulnerability comes out and you think, oh my God, you know, this pixie fail thing or this logo fail thing, we might be impacted, you got to throw, you know, it's time to throw away that hardware and buy something new.
So we do have.
Yeah, go ahead, sir.
Oh, yeah, I just wanted to call out some work done by one of our core boot vendors, 9elements.
If you go to GitHub.com 9elements, goswid, g oswid, that's the tool that it's supported in Coreboot now to generate and view software Bill of materials.
So when you build coreboot, it'll actually try to pick up as much of the metadata as it can about all the constituent parts and licenses and it'll give you that information that you need to form an SBoM.
I tried it a few times and I think there are some rough edges, but I know I've used it on like an Intel Archer City reference board, which is Sapphire Rapids, and you actually get most of the info that you know.
It's satisfactory.
Could still use some improvements, but I think it's a pretty good start for anyone looking for a soft to ship an SBoM with their core boot distribution.
Oh, and they'll.
Yeah, go ahead.
I'm sorry.
The, the same company also has the 9 elements converged security suite and that's also on GitHub.
It's open sourced, so if you want to install a signature, you know, that'll actually provision bootguard for you.
And they do list AMD support, but I'm.
I'm not sure how far that goes yet.
Okay.
Yeah.
And are those the firmware versions of the bios?
Are those fully reproducible builds right now or so they are.
So you can actually trust the checksum and you can rebuild yourself to ensure that it fully matches.
Yeah, yeah, this is something.
Matt may have seen this in his profession, but I've seen it as well, where you get a vendor's, some code base supplied by your hardware vendor, like your odm, and you build it once it produced, you know, and then you do an MD5 sum or sha sum just to kind of see that you're copying it around correctly or whatever.
Then you build it again.
And of course it's different because, you know, they embed timestamps and the way that the UEFI Flash file system is constructed might be different depending on small little variations and phases of the moon and whatnot.
So you could easily end up with different hashes for what's essentially what should be the same firmware.
Whereas reproducible builds and coreboot have been a big thing since, you know, over a decade.
Oh wow.
If you clone coreboot and you by default it will actually build the entire tool chain necessary to build it.
So it'll compile a specific version of GCC and binutils.
And so even the tool chain is hermetic.
So this means if you get coreboot at a specific hash, then it'll pull down the specific tool chain, you build it, you'll get a reliable hash every.
Time, way ahead of its time.
And then optionally, for people who don't want to wait for all of GCC to build, there is a menu config option where you can just tell it use whatever your distro ships with.
So if you don't want to wait an hour for all the GNU utilities to build, you can do that too.
I think there's also a Docker image you can download that already has those pre compiled to save a little bit of time.
Yeah, that sounds like a large toolkit to build every time you want to compile it.
I do get that.
Question from time to time, people saying like, well, why should we, you know, trust Mr.
Chromevox?
It's like, well, you don't have to.
You can clone my repo, you can run the build script that I provide, and you will get the exact same hash as the files that you can download from my server.
And so that, I think that gives people who would otherwise be weary of just flashing a random binary from some guy on the Internet, right.
A little bit more confidence that the software that they're using is credible.
I mean, I think that's where the industry at large is heading.
Right.
Particularly with SBOMs and reproducible builds.
That seemed to be like a big Trend.
Seems like Coreboot beat the trend by about a good 10 years, but it's something that I've definitely seen more and more in the last year or two.
Yeah, that now I think this is, it's an even bigger advantage for coreboot now because when you think about the software supply chain, you know, coreboot's a.
It's a hardware centric project.
UEFI is a centric hardware project.
U boot's, you know, hardware centric project.
It's really tied in with the hardware vendor ecosystem.
So basically you have probably four, maybe five entities between your processor being produced and your laptop being shipped to you.
So you know, a lot of the code, you know, whether we're talking U boot, uefi, coreboot, typically it starts off in the silicon vendor, right?
If you're in the ARM or embedded ecosystem, you typically get like a board development kit or software development kit from your vendor and then you customize it and then you ship, you put that on the product.
In the x86 world, typically what happens is the silicon vendor, like intel or AMD will write a whole bunch of reference code and then they'll hand that off to their BIOS vendor partners.
These are companies you may have heard of, AMI, Inside Phoenix, etc.
BioSoft in China.
They'll additional features, things like maybe some embedded controller support, or if you're on a server, they'll add Support for various BMCs, RAS features and so on.
And then the BIOS vendor will license that to an odm, which are, you know, hardware big hardware manufacturing companies typically based in China or Taiwan that most people probably haven't heard of, like Foxconn, Quanta, we win, inventech, compal, more.
And then they'll partner up with an OEM original equipment manufacturer like Clevo, System 76.
Let's see who else?
Lenovo, Dell, Whoever you know, the names that you see it on the store shelves.
Right.
And then they'll package that all up and sell it to a customer.
So there's actually many entities along the way that are touch.
That are potentially touching that BIOS code.
And so just the supply chain going from one company to the next, you got to think you're not just trusting the code, but you're trusting the people who work at those companies.
And I have a joke.
So much effort.
You see Binarly in these companies putting so much effort into finding backdoors.
What about going through the front door?
How hard would it be to get hired at one of these companies and just start inserting malware and shipping it to people?
And the vendor's key is fused, the customers can't do anything about it.
Yeah.
I'd be surprised and maybe a bit disappointed if the three letter agencies haven't already thought to do that.
Yeah, I mean it's.
Yeah, exactly.
I would be surprised if they don't have their certificates in there one way or another.
Right.
And in particular if you're the depending on threat model at the end of the day.
Right.
But there have definitely been scenarios in the news about similar attacks.
Right.
And I would imagine that anybody in the chain can insert a certificate into the buyers, any point that attack.
So yeah, it's absolutely a very valid point.
And you have no way of knowing that it even happened because you can't even check it.
Yeah.
And the rise of larger companies, cloud services, et cetera, means, you know, that there's a lot more data online at this point that could be pilfered, essentially.
So.
Yeah.
So this is why I think, you know, companies, you know, occasionally you hear a big news story about, oh, a bunch of user data was stolen from, you know, such and such company.
They may face some kind of legal or financial penalties because of it.
So it's sad that it takes that kind of thing to wake up the CTOs around the world to.
Your infrastructure is important.
You don't necessarily need to build everything from scratch, but you should at least have some insight and ability to audit and control what's going on.
There's a paper by, I guess, a guy who used to work at this company that's had some issues with quality assurance lately.
About 20 years ago, there was an engineering paper and the guy was saying that the final product can only be as good as all the inputs that go into it, essentially.
So in this case it was an airline company and they were in the midst of outsourcing A lot of their component building and quality controls and the guy was raising the alarm, hey, you know what, we really got to make sure we have insight into these processes.
Otherwise we're going to start building some pretty dangerous aircraft.
Now that doesn't mean that you shouldn't outsource or that, you know, you can't hire consultants to help out or whatever, but you do need to, you know, have the ability to know what's going on and have some quality controls.
I fear a lot of companies and a lot of manager types are willing to say we're just going to rely on our ODMs to do things for us.
And if things break, we'll file a ticket with them, we'll call them up or send them an email.
So walk me through the so imagine I am a company that resells computers that I probably bought from one of the OEMs, right?
I'm some kind of mid level software vendor and I want to move to coreboot.
What's the process like?
Are there consultancy firms out there that can help you with that?
Or walk me through the process of like how the complexity looks like as well.
I would say yes there.
Go ahead Matt, I'll probably jump in later.
Yeah, there are a number of vendors on coreboot's website listed under the consulting section that can help a company who is looking to go down that road.
You know, a lot of times with hardware that you purchase off the shelf, you don't necessarily have the option of changing the firmware, you know, because of security implementations that the vendors put on there like intel bootguard.
So if HP or Dell's key is already fused onto the processor, there is no way to change that, unfortunately.
So ideally if you are looking to use your own firmware, whether it's core boot based or something else, you need to be looking much earlier in the manufacturing process or looking to purchase hardware where the bootguard keys have not been fused and you have the ability to fuse your own before you ship them to customers.
And you knowing those kind of things is where the, you know, consultants like Nine Elements and other companies come into play and can help you figure out where in your product development you need to be looking at to make those decisions.
Right?
And I presume you need to go to the ODM rather than the ODMs OEMs to even get to that level of control, I would imagine.
Yes.
As David mentioned, I or I believe you mentioned I used to work for Purism doing the open source firmware for their laptops and servers and other devices.
And Purism would work directly with ODMs in China to have the devices built to our spec and then shipped open to where we could install our own firmware and give our users the ability to put their own keys on should they want so that the ultimate control would be transferred to the owner of the device, not necessarily to the company that was producing it.
And walk me through the challenge with that.
I would imagine that's a very.
If you work with manufacturers in China, that's not really what they are used to dealing with, I would imagine.
Walk me through the complexities are you guys facing when at Purism there.
I'd say there's a lot of different size ODMs and some of them do cater to the larger corporate customers like Dell and Lenovo and things like that.
But there's a lot of smaller ones as well that do cater to, you know, smaller customers who are just looking to do a run of, you know, 10,000 laptops or something like that.
You still have to buy in a sufficient quantity to get their business, but they're much more willing to work with you and you know, that is their business is catering to the smaller customers.
So I wouldn't say there were any specific challenges there other than we would want to have more control over the hardware design.
And doing that is a much more expensive endeavor.
You know, if you want to completely engineer your own board, not using an intel or AMD reference design or something that's based off of that, then there's a significant NRE cost there.
And that poses challenges when you're trying to bring a product to market that you're only going to sell 10,000 of versus a million of.
So the MOQs are pretty high for even a mid size or small size software company and want to do a hardware component.
Yeah, I don't remember exactly what the MOQs were, but they're substantial enough that David and I wouldn't want to personally go and have our own computer built.
Right.
Yeah.
But there is cause for optimism.
We do have a lot of good momentum with a number of silicon vendors, including the x86 vendors.
So if you're a company looking to create a new product and you think ahead a little bit, you know, a lot of the work has actually already been done.
We, there's coreboot ported to basically all the latest intel and AMD processors, at least for client servers.
We're, we're actually getting there.
Intel's been doing a great job with the XEONS lately.
AMD has a proof of concept with their Genoa processor.
Yeah, if you have a little bit of foresight, you could actually get up and running with a new product with coreboot pretty quickly, pretty easily.
Not terribly expensively compared to paying the NRE for a proprietary BIOS.
Check out coreboot.org consulting or reach out on the mailing list people who are interested in servers.
You can contact me directly.
D Hendricks D H E N d r I xoreboot.org and I'll help you get connected with some of my contacts at the ODMs, particularly we win Quanta and Inventech I have contacts at.
But that would be more for servers where maybe you're a up and coming cloud service provider, hyperscaler, whatever, and you want to get racks and racks of servers with open source firmware that your own, you know, DevOps or SRE teams can audit and maintain.
So there is, there is cause for optimism.
But you got to be forward looking, not backwards looking because as Matt said, once you have a processor with someone else's keys fused in it, you don't own that.
Right.
Okay.
So, so that kind of leads me to the next topic I wanted to cover, which is we have seen a push towards open hardware more and more in the last few years.
I guess it won't truly happen for the x86 world, I would imagine, but more towards RISC V is probably where my hopes are up, at least for truly having open hardware.
What does the world look like for coreboot with regards to RISC V and open hardware in general?
Yeah, yeah, that's a tough one.
I think a lot of people pinned a lot of their hopes on RISC V being open, but a lot of the contributions and a lot of the real development these days and the governing bodies of RISC V tend to be from companies that want to keep things proprietary.
So yeah, the instruction set architecture is open, but that does not mean that the implementations will be open.
And I think a lot of people have gone into this thinking that everything's going to be open and then kind of backed off, a little disappointed.
That said, you know, there are, I think there are fully open implementations out there, but they're not the majority at this point.
So it's, you know, you got to set your expectations.
It's an open hardware architecture.
You're going to have, you're going to have open source fanatics working on it.
You're going to have companies looking to, you know, put a bunch of their IP in this and want to keep that secretive.
You know, that's just part of being open.
It's like, you know, a lot of people might think everything is more like the GPL in RISC V world when in actuality it's more like the BSD license.
Right.
Mass analogy.
So.
So yeah.
Oh yeah.
Just to cap that thought, you know, basically it's, you know, it's turning out to be no more or less open than arm.
So I wouldn't look to a RISC V as like this is an open alternative to arm.
It's more like, you know, it's something that I can run with and not have to pay an ARM license if that's important.
Yeah, I was just going to add, you know, one of the things David mentioned was, you know, vendors wanting to insert their IP blocks and looking at just how complicated a modern processor or a modern board is and how many different components it contains, like, you know, DRAM and initialization or USB 4 and thunderbolt.
And there's not a lot of companies that have expertise in all of those areas.
Now even the major SOC vendors don't internally have all of those components under their own ip and even if they did, they might not want to share them.
So there's a lot of licensing issues that would prevent a fully open and fully modern SOC from existing.
At this point somebody would have to invest millions of man hours to get it to the point to where it matched the existing functionality.
And there needs to be a very strong use case for a capitalist company to do that.
Yeah, that's fair enough.
And that doesn't even bring us to the complexities of GPUs I guess either, which is a very different subject as well and much more complicated.
Cool.
The one thing I also wanted to cover was we spoke about system 76 before.
I don't know if you want to do any more shout outs about vendors that are really embracing Coreboot.
Outside of Google Systems over 6 came up.
Purism came up.
Star Labs I know is another one.
Any other ones that people should look into for buying core boot ready hardware?
I think on the EU side Nova Customs is also offering some.
Trying to think offhand, there's a Polish company called 3M DEB 3MDEB and they're actually a coreboot consulting company, but they do support a couple off the shelf products.
Like they've been pushing what they call their Desharu distribution of core boot and that currently supports a pretty recent MSI gaming motherboard.
So people who are looking for a Nice gaming lap, you know, gaming desktop system.
They can plug a couple GPUs in it.
Maybe they want to do crypto mining or something and they want it to be owner controlled.
They can look up 3M, Deb and Desharro.
Yeah, and I call it.
So one of the interesting things about firmware is that these days it comes in distributions.
So there's upstream core boot and then there's the desharu coreboot distribution.
UEFI is much the same way.
The way to think of it is there's upstream EDK2 and that part is open.
But a lot of what actually gets put onto hardware that's sold to consumers is a distribution by one of the BIOS vendors where they add their value adds and sell it to OEMs.
Yeah, as far as coreboot vendors go.
Yes, let's see, we named a few.
There is a vendors page linked from coreboot.org, and I think they might have a few other suggestions there.
Okay, circling back to the coreboot distributions topic, one thing that is important to note is that coreboot isn't a binary release.
It only has source releases.
So it is up to, you know, the implementing vendors to produce core boot binaries for their devices and to distribute them.
It's.
And coreboot doesn't have any sort of actual stable releases, they're just kind of timestamped releases and they follow the.
I think right now a what, three month cadence.
I think we've covered all the topics that I wanted to cover.
I think we've covered why coreboot is important and why right now, and particularly in light of logo fail and pixie fail and all these things that.
Yeah, go ahead.
Circular economy.
yes, that's another topic that is very interesting as well.
It's like exploring why sunset hardware can be repurposed both with obviously Linux in the user space, but also coreboot in the bio space and making sure there's actual secure.
Yeah, yeah.
So just a quick word on that one space that coreboot kind of found itself in recent years is circular economy or some people call it, I think you called it upcycling.
And basically the idea is there's a lot of hardware out there that it runs perfectly fine, but people don't want it for whatever reason.
Maybe you're upgrading your laptop or more recently, what.
One of the things we saw, I've seen in the Open Compute project is the concept of circularity where big hyperscaler companies, you know, the large cloud service providers that have tens or hundreds or Millions, you know, hundreds of thousands or millions of servers.
You know, they always want to get the latest and greatest thing right because they have to meet increasing demand and get more compute power.
Problem is, data centers are very expensive.
They can only exist in a financially sensible sense in certain locations.
It's got to have cheap power.
It's got to have fast Internet connections.
It's got to have the right climate because cooling a lot of servers is difficult.
So anyway, data center space is very constrained, but companies need the latest and greatest hardware.
So what they'll often do is they'll decommission perfectly good servers after as little as three years, and then they'll kind of give them to some companies.
It renew is a big one in the open compute project space.
They were bought by Iron Mountain, and there are some others.
But essentially these other parties, their job is to try to reclaim as much value as possible from that server.
And there are a couple ways to do this.
One way is, you know, you take off the cpu, you take off the dram, you take off any resellable peripherals, and then you just kind of sell them at whatever the market rate is, and then you scrape off all the valuable materials from the motherboard, you know, gold, indium, whatever else you got there, precious metals, and try to reclaim as much value as you can from that thing.
But another approach is to resell these, you know, whole.
Basically, you can kind of think of it like if you go, if you're in the market for a new car, you roll up to the car dealership and the first thing they'll ask you is, would you like to trade in?
Because they want to buy it from you.
They know you're looking to get rid of it and upgrade to a newer model.
So they want to buy it to, you know, they want to buy it from you and then sell it to somebody else at higher price, you know, at a little bit of profit.
And that's actually the best way to do it, if you're thinking from an environmental sustainability standpoint, because there's no waste, you know, that somebody who maybe they can't afford a new car, you know, maybe it's, you know, a teenager getting their first car or whatever.
They get a product that they wouldn't otherwise be able to afford.
The parts stay in service for at least a few more years before they eventually get scrapped and they're, you know, the similar thing happens with servers nowadays where it's nice to be able to take a whole server and then sell it to maybe Research institutions that don't have a lot of computer, you know, don't have a huge budget for brand new computers.
Developing countries is another big thing.
You know, there are a lot of up and coming economies in like Africa, South America, etc.
And you know, even Asia, you know, there's, even though a lot of the hardware gets built there to begin with, there's still a lot of people out there who don't have Internet, you know, who aren't really connected to the Internet or the modern economy.
So anyway, it's a great concept, but one of the problems is, to borrow an old phrase, hardware without software is, you know, a chip without software is just sand, right?
So if you've got a server with, you know, firmware that has intractable problems, maybe it won't boot for some unknown reason, variables might be corrupted or it has known security issues and you wouldn't want to try to, you know, you can't really sell it with a support contract because that's where vendors will get a lot of money is, you know, here's a used server and I'll sell it to you and you know, 100 bucks a month gets you tech support or something like that.
Right.
If you don't have firmware that you control, then, you know, the server loses a lot of that value proposition.
So this is where coreboot and open source firmware in general can help a lot.
You know, companies and governments are starting to look a lot more about sustainability initiatives, trying to put less e waste out there.
I don't want to say it goes to landfills because I know these companies work really hard to try to prevent that from happening.
But if the hardware can't actually be used by someone else downstream, then you know, it's got to go somewhere.
And I think Matt actually did something similar with Mr.
Chromebox, you know, running on older hardware as well.
Is that right?
Yeah.
So I actually work with quite a few recyclers who will take, you know, decommissioned Chromebooks from schools.
A lot of them are either right at their edge of end of life or, you know, past their end of life.
So they don't get updates anymore, even from Google.
And to Google's credit, they have been improving the support lifetime of newer Chromebooks.
They've extended it out to 10 years now.
But a lot of the older ones that shipped within the past 10 years don't have that long of a lifetime.
And so they've already reached their end of life.
They're not getting OS updates, they're not getting firmware updates.
And so the schools, when they're done with them, they've gone through three or four or five very hard years of kids using them.
You know, they'll.
They'll dump them on these recyclers.
And the recyclers will, you know, try to physically refurbish as much as they can.
But then from the software side, what a lot of them will do will be to use my Mr.
Chromebox Coreboot firmware on them and put a.
Another OS on them and then send them over to a developing country where they can be, you know, resold at a price that's affordable for them but also makes sense for the recycler.
And so I've helped a lot of recyclers with that, providing some custom firmware, providing tools to help them, you know, diagnose and upcycle their hardware.
And so that's one of the things I'm most proud of, the success, not necessarily the number of downloads, but the amount of machines that we're able to keep using and not have to go through the very expensive process of extracting precious metals from them simply because they're not supported by the manufacturer anymore.
Yeah, no, that's really fantastic.
And yeah, they would probably do the job just fine for basic use cases, even though their operating system is out of end of life.
Right.
So that's.
Yeah.
But I was very happy to see Google's extension of 10 years as well.
That's kudos to Google for doing that.
It's glad to see they've been leading the way there at least.
So, yeah, good stuff.
Cool.
I think we covered a lot of ground today.
I think we've had a lot of interesting conversations.
So thank you both so much for coming on the show.
Much appreciated.
And yeah, looking forward to speaking to you guys soon.
Thank you so much, guys.
Thank you, Victor.
Appreciate it.
Take care.
Found an error or typo? File PR against this file or the transcript.