Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Revolutionizing Firmware Updates in Linux: A Deep Dive with Experts
Play On
05 MAY • 2024
58 mins
Share:
In this episode, I’m joined by Richard Hughes from Red Hat and Mario Limonciello from AMD to explore the fascinating world of firmware updates in Linux. Their work on the Firmware Update Project has fundamentally changed how we handle firmware updates in the Linux ecosystem.
We start with Richard sharing the origin story of fwupd, which began with his work on Colorhug, a free software color sensor. What particularly caught my attention was how this seemingly simple project revealed the broader need for standardized firmware updates in Linux. Mario then adds his perspective from Dell, where they were trying to match Windows Update’s capabilities for Linux users.
The conversation gets especially interesting when we dive into the technical details of LVFS (Linux Vendor Firmware Service). Richard and Mario explain how they’ve created a centralized system that not only distributes firmware but also ensures its integrity and security. Their insights into supply chain security and the role of SBOM (Software Bill of Materials) in firmware updates reveal the complexity of modern system maintenance.
I was particularly intrigued by the discussion of how major vendors like Dell, Lenovo, and HP have adopted fwupd and LVFS. The impact of Google’s “Works with Chromebook” initiative on consumer device support shows how far the project has come. We also explore the challenges still facing server firmware updates and the potential role of Redfish in addressing these issues.
If you’re interested in system security, firmware management, or the evolution of Linux infrastructure, you’ll find plenty of practical insights here. Richard and Mario bring both deep technical knowledge and years of real-world experience to the discussion, making complex firmware concepts accessible while maintaining their technical depth.
Transcript
Show/Hide Transcript
Hello and welcome back to Nerding Out Victor.
In today's episode we gotta dive into the world of firmware and I'm joined by Richard and Mario from the Firmware Update project.
And we will be speaking about all things firmware and how that looks in the Linux world really.
And perhaps we can help our viewers and have you guys do a quick introduction about yourself and tell about who you guys are and what your backgrounds are really briefly.
So Richard, you want to give a start?
Give a go?
Yeah, sure, no worries.
So my name is Richard Hughes.
I've worked for redtap for like 17 years or something.
I've worked in open source for about 25 years.
It's horrendous.
Over that time I've built stuff so I've built like GNOME Power Manager, upower, which is like a power management thing.
I did Package kit, which is a package management thing and then GNOME Package Kit and then GNOME software.
I did color stuff, color D, GNOME Color Manager and now I'm doing packaging, now I'm doing firmware stuff.
So yeah, it's been like quite a wild ride.
I live in London, I've got two kids.
It's great, amazing.
Thank you for the intro, Richard.
Mario, do you want to do a quick intro?
Sure.
So my name is Mario Limoncello and I'm currently employed by amd.
I've been with AMD about three years now.
Before that I was with Dell and I was with dell for about 13, 14 years and I've always been working on Linux ever since I left college.
My professional life at Dell I did our enablement for different client computers and so that involved initially engineering and eventually architecture.
I had a big focus on power management related things and making all the things work.
And at AMD I have a role that I help to enable all of our AMD's customers that want to be shipping Linux on their machines.
So that's like our customers are, you know, the Lenovo's, HP's, the Dells frameworks, those kinds of companies.
My particular interests in free software are making power management effective and making sure that the hardware runs really well.
Richard and I crossed paths back, what was it, 2015 or so, 2014.
Yeah, back when were trying to come up with a solution for firmware updates in Linux on Dell's machines and there happened to be a good intercept at that time that were starting to do capsule updates in Windows with Windows Update and there was a lot of interest like hey, why don't we come up with a story for Linux for this.
And we crossed paths and have been interacting ever since.
Random story.
Do you know where came from?
No idea.
So before I worked in open source stuff, I used to work for a defense company building stuff for military flight computers.
And I really enjoy making stuff.
Like end of the day you'd have a tangible thing.
And then when I went to software, you basically, at the end of the day you've basically changed zeros to ones and ones to zeros.
You haven't built anything.
And I missed that.
So it was like when I started doing the color management thing, I needed a test device that I could use for just measuring spot colors on the screen.
There was nothing that was free software that kind of worked.
And so I built this thing called Color Hug.
And Color Hug is like a little tiny free software, free hardware products that you stick on the screen.
It tells you the color and as part of that you don't make one.
When you make a PCB, it's not like now where you can just make one, but you used to make 50 minimum order quantity.
So I made like 50, put it on my blog and said, hey, am I interested?
And I think we sold 2,600 over a couple of years or something.
These little tiny color sensors on them were firmware.
And so I kept having to do firmware updates to fix bugs, add functionality, fix things with color panel changes and stuff.
And so in doing that, I wrote a little program which updated the firmware.
And I thought, this is shit, right?
Everyone who writes, everyone who produces these devices have to do exactly the same thing.
Write the front end, make it pretty, make the permissions UDEV rules policy kit, all that kind of stuff.
Wouldn't it be kind of cool if we had this centralized thing that we could just plug in the protocol to and rather than having everyone reinvent the wheel all the time.
And that's when I crossed paths with Peter Jones, who invented this lib FWPD, sorry, lib FWPTate library, which allowed you to do the Update Capsule.
And so the very first version of FW UPD came out and it had a Color Hug plugin and it had a Update Capsule plugin.
And that's literally all it could do.
And super early, like Mario said, like Dell said, yeah, we want some of that.
And then from then it's just exploded.
It's brilliant.
It's super interesting.
I must admit I only crossed paths with the tool a few years ago and it looks like it's gaining a lot of momentum.
But I guess that leads to a really good segue to the origin story you kind of mentioned Richard there.
But what's the vision in general, more than for the project?
Is it to support all sorts of things, hardware?
Is there any.
Well, yeah, let's talk to me about the vision of like how you guys see this going forward.
So I guess there's two things, right?
So one is like the deployment.
So this is like I have a firmware file, I need to get it on the device, right?
Be a mouse.
NVMe drive.
Yeah.
UEFI system firmware, Redfish, BMC.
Doesn't matter, right.
Actually, deploying it is probably the easy bit that's like the.
You have some bytes, you do some transfers, you get some error codes or some success back and the device is updated.
The harder bit is where you get that file from.
And that's kind of where the RVFS project came from, which is like getting the vendors to upload firmware in like a standardized way somewhere where we can extract the metadata, send it to users.
The actual deploying the firmware is the uninteresting bit.
The interesting bit is telling half a million people you have a critical security vulnerability.
Absolutely.
And I got reignited by this product.
I gave a talk at OpenCon a few weeks ago in London and I kind of talked about my backstory of why I sort of get excited about open source biases.
For instance.
So coreboot is a project that I had the guys before coreboot on the show a few weeks ago.
And what I realized is exactly what you're saying, Richard, is like, okay, you have these fleet of devices out in the world and you have a logo fail or pixie fail.
All these things happen.
And it's like great for screen for us, for instance, like we can't go wrong with USB stick and update the bios, right?
It doesn't work like that.
You have to do at scale.
And that's kind of why I got really excited about the combination of core boot and firmware Update.
Or actually, let me ask a question.
What is the correct pronunciation of Firmware Update or FW Update?
Because I have heard a lot of people say different ways.
That's a very good question.
And the answer is I really don't care.
So I personally call it FW upd.
Yeah.
But I also called it fwupd if I'm another one, if I'm really trolling a vendor.
Like I've had one vendor that's really pissed me off.
And I went through the entire one hour meeting calling it four Whopped, which I described Afterwards as the sound jelly makes when you tip it off a plate.
All right, yeah, that's fair enough.
Yeah, that's fair enough.
So that's, I mean it's solved for real problem.
Particular for biases where I really started getting excited about this project because when things like this happen, the logo failed, the pixie fail.
We've seen more of all these in recent years, right?
Because at least to me before we started doing hardware, I must admit the last time I thought about firmware was a very long time ago.
When I built PCs in my teenage years, that was the last time I actively dealt with these things.
But when you do it at the scale now, you realize that from a security perspective, the bias is just a blob that you basically just trust without knowing what it is.
It's worse than that.
It totally worse than that.
So like below the BIOS you have this thing called csme, which is what intel me and that has full control of your PC even when it's turned off.
You can do stuff, plug it into the Ethernet jack and you can like.
So one of the checks we do in FWD is a set of checks for hsi and one of the checks is your CSME critically vulnerable?
I.
E.
Could I remotely turn on your computer, flash a firmware, turn it back off again without you knowing?
Right.
That is pretty terrifying.
So for me was always the root of like the worst possible compromise for device has always been the bios.
Because if you can tap with like the TPM or the rng, whatever, like it's really game over.
You can't really do anything in user space that recovers that.
But you say there's even worse, which is even more terrifying.
I think something that we've done with WUPDI is that we actually go and introspect any of the security stuff you can get from AMD's PSP or Intel CSME to say that the hardware is at testing that these security features have been enabled.
And if those aren't enabled, then you can tell the end user this isn't enabled.
You need to go to your manufacturer, get this fixed or you have a big risk here from this.
The reason I'm smirking is like there's a major vendor which I don't know, it's MSI and they shipped most of their notebooks in what's called manufacturing mode, which basically is how they come off the production line.
And it means that anyone can do anything to anything, which is horrendous.
And so with the idea of HSI Was if we can do a scan at startup and say if the vendor can screw something up, can we detect whether they've screwed it up or not?
So is Secure Boot turned on?
Is the platform in manufacturing mode?
Is the tpm?
Is the tpm.
If you replay the TPM events, does it actually give you the final hashtag?
Anything that can screw up?
We check.
Interesting.
And if you do detect these abnormalities and you detect these severe instances, are you able from user space rectify these problems or can you just notify?
It depends on the individual problem.
Some of them we have fixes you could apply where if there's a kernel command line option that'll lock something down harder, we'll do that.
If we'll say there's a BIOS setting, you could toggle and we have a knob toggle it, we'll offer them toggle it.
But some cases we have to say, we can't do anything about this.
You got to talk to manufacturer about it.
Yeah.
Or you wouldn't want to turn on Secure Boot, for instance, if.
Because that would break the system.
Right.
If you don't have space level for the booting.
So, yeah, we have security checks that we put in place before we do any sort of setting that, hey, is this going to cause a breakage?
Make sure that you actually have shim there.
Make sure you have a signed bootloader that you can actually turn it on.
So a good example there is the DBX update.
So DBX is like this list of hashes essentially from Microsoft, which say, this stuff has known vulnerabilities.
And if you are running, say, I don't know, like RHEL7Box, you've never updated and you just blindly deploy this DBX thing.
Your next boot, you can't boot because the bootloader that you're using is in the DBX and so your system won't boot it.
So one of the things we do before we deploy anything is go through everything you've got in your ESP and say, is any other stuff that you've got there in this update?
Because we're not going to deploy it if we're not going to boot.
So we're like, the problem is with flashing firmware is if you brick someone's computer, they'll remember it forever and they'll tell all their friends, they'll tell everyone on the Internet.
And so we have to be really sure that we're doing the right thing and we're not going to brick something.
So on that when we deploy an update, One of the things we, if you're on the command line, we'll say to the user, the updates happened successfully.
Do you mind if we upload this result to the RVFs?
And so vendors can straight away see, right.
Well, the updates are going out with like 99.79 success rates.
And then if they start coming back with more than.
I've got the percentages, but I think like 5%, if they come up with that 5% failure rate, we kill it.
Like, we kill the deployment because it's, it's not hitting what it should do.
I think you've gone a step further too, with LVFs that now the vendors have to put a signed report showing that it worked before they're allowed to promote it.
Yeah.
So this is something that came out of the Google thing.
So essentially what happens, the vendor has to prove to us that they've updated.
So.
So what tended to happen before was that the vendor would do all their tests on Windows or maybe not even at all, and they would upload the file to Windows Update, they would upload the file to LDFS, it would go out to loads of people, and 100% would come back and say, this didn't deploy correctly.
And we're like, you didn't actually test this because this never could have worked.
This is for the wrong model.
Right, Right.
And so now we actually require the OEM to actually install the update on the target hardware, sign that report, and then send it back to the RDFs.
And that's a public record.
So you can see that like, Lenovo has tested this on ROM 9 and Ubuntu and sleaz and AMD has tested this other update on whatever, whatever.
And so you can sort of see that a, that everyone is testing what they say they're testing.
And also if there is like a Hardware Quality Lab or Red Hat QA or Google or whoever, they can additional reports saying, yeah, we tested this too.
Right.
And I guess there is no rollback either in that sense for a lot of firmware.
Right.
Once you brick it some.
But sometimes you don't have a graceful rollback that you could do.
Like, this is a phrase I'm going to use a lot in the next hour.
Yes, I know.
So in some cases, yes, you can go back, like system firmware, sometimes you can go forward, and if you type FWMGR downgrade, you can go back to the old version.
It's fine.
Okay.
In some cases you can't.
So some cases for good reason.
Like you might, if you're doing a firmware update on your system bios, it might be updating some data table to some new database schema and you just haven't got all.
There's no way of going backwards.
And there's more of a bullshit kind of reason where like, for like csme, where there'll be like a, like, I don't really call it like a token which can only go up and which means you can only.
You can't downgrade, which makes it very difficult to test this stuff.
Right, of course, yeah.
Okay, so there have been a lot of references to lvfs, the Linux vendor firmware service already.
I was going to dive into this, but let's I guess start unpacking that.
What is LVFS and what's the objective of it and how does it work?
So LVMS is just a dumb website, right?
Right.
It's a website where, I don't know, 140 different vendors have user accounts which can upload firmware.
So if you're Lenovo, you can have your ODMs, your IPVs, all the people that are providing binaries for you, they can be uploading to the rvfs.
And then when the binary gets there, we unpack it and we verify it and we do all these tests and even like static analysis and stuff, like crazy stuff.
And once it's there, we build this catalog of metadata.
The metadata goes out to end users.
It's like, I think it's up.
80 million users download the metadata a day.
And then if your system locally matches anything from the metadata, we provide the CAB archive to the end user.
Right.
And then if the update happens successfully, or if there's a failure, we send the report back to the lvfs.
It's all completely completes the circle.
Right.
And so for some history of when LVFest started, there's a reason why it's CAB files.
That's because were wanting to replicate what Windows Update did because were focusing originally on capsules.
And so that was the format that you would put up into Windows Update.
We wanted to make it really low barrier to entry for OEMs to put up the files.
So when Dell was doing the very first files, those were capsules that were put into caps just because of that.
And it's evolved from there.
Right.
Okay.
And this is a Linux foundation project or is it independent or who actually owns and controls this project?
Yes and no.
When it first started it was.
This is not even a joke.
It was literally A server underneath my stairs.
Oh wow.
Which was sending out metadata around the world over a cdn and lots of people started using it.
Like lots of like three digit US agencies and lots of big customers and banks and stuff.
And lots of people started freaking out that there was one random guy in London with this server under his stairs and they're building all this like infrastructure.
Talk about attack vector, huh?
Yeah, exactly.
So went to the Linux foundation and said, look, can you help us?
And they're like, yeah, this is critically important for Linux as an ecosystem because we want hardware vendors to provide firmware updates for Linux users so they're first class citizens.
And like long.
And the short of it is I make a pretty shit sysadmin.
I'm a pretty good C coder.
I'm a really like.
Because the Linux foundation team basically microserviced it and it's now like deploying to ECS and it has all monitoring and people get paged when it goes down and stuff, right?
And they were saying, well, how do you deploy it in your server right now?
And I'm like, well I ssh into the server, type sudo git pull and they were just like, oh my God.
Yeah, that would not make for a good sysadmin.
Now I give them that.
Absolutely.
This is terrible.
But it was all like the RVFS itself is just good enough.
So you could say there's lots of other ways to do it, but there are other good legal reasons.
Like because a lot of this firmware does contain strong crypto, we have to be very careful about.
For instance, we can't just mirror anywhere because of things like ITAR and EAR and all the like, you can't send stuff to South Sudan, Iraq, Iran, et cetera.
And so we have to do goip lookups, that kind of thing.
We do allow mirroring kind of like internally in terms of.
So if you're someone like the DoD or the UK government or something, you can mirror the entire LVFS completely for free.
That's fine too.
Public files from LVFs, not the entire LVFs.
Correct.
Because also I was saying like the LVFs, when you upload files, you upload it to this private space where you can access it and we'll do all the tests and stuff, right?
And then it moves to this embargo state, which is kind of like anyone in your vendor group can see it.
And then it moves to a testing state, which means that anyone publicly can see it, but only people that opt into the testing repository and Then once it's gone through testing, it'll then be probably by QA user be moved from testing to stable where it's available to hundreds of millions of people.
Okay.
I mean that, I guess the obvious thing to come to mind is kind of go back to the security attack vector there.
Like if you can tamper with a firmware and deploy that to millions of devices, like that's just gold dust right there.
How is that mitigated?
So we actually sign the firmware, but we use a detached signature.
So when we sign the firmware on the lvfs, we're basically saying this came from the lvfs, we're not actually signing the firmware.
So it makes sure it goes on the right device.
We kind of rely on the device vendor.
So if you took a random capsule file for your system, it wouldn't deploy on my system A, because the capsule file's doing model checks, but also because it needs to be signed by the right manufacturer key.
Right.
So even if so, worst case scenario, someone got into the RVFs, started pumping out hundreds of millions of capsules, hundreds of people.
The very worst case scenario is we'd say there's a firmware update available, user would click it, they'd reboot, nothing would happen.
There's been a few instances back when I was at Dell that they would accidentally tag the wrong GUID for a firmware binary and they would try to apply it to that sit to a different system.
It doesn't work.
It's a signed file, it verifies the signature, but then says the GUID that's within the file for this binary doesn't match the system, it rejects it.
It's actually really good segue to the other tests we do.
So when we, when you upload a file to the LVFs, we actually look at the capsule and say, is this the right guid?
Is this the right system?
Does this have the right flag set, does the metadata match, what it should do, etc.
And then we started doing some other super dumb tests.
Like we began searching for things like if you're just using like UTF8, UTF16 Le UTF16 be scanned for the words begin private key.
And that found a industry wide CVE where Intel was shipping some reference keys which then got copied into almost every OEM's firmware image.
And it was an 18 month disclosure period for finding the words begin private key.
Wow.
It's kind of, it's exploded from that as well.
So we do other tests as well, like dumb Checks like if you're uploading DFU file, we check whether there's a DFU footer.
Like really dumb stuff.
But we started working with a company called Benali.
And so we take the firmware capsule, we explode it into its file volumes, we take the file volume, we explode that into EFI binaries.
Then we use red air to do static analysis on each binary and look for specific suspicious byte sequences.
So if you have a known firmware weakness, we can say, well look, this is affected.
So when something like logo fail comes out and Cert says, okay, well what vendors are affected, what models are affected?
We can just run one scan and we say, right, all of these vendors are affected, all of these models are affected.
Which is great from a discovery point of view, but it's also good from a preventing a regression point of view.
So what tends to happen is if you have two firmware teams, one's a security team, one's like enhancement, normal new product team, it could be that a new microcode is released by the security team which goes up a version like this and then the product team puts the old microcode back in for the next update.
And so you actually, you've regressed the security fix.
And so if we're doing things like checking the microcode version, that's a very easy thing to check, or checking that the static analysis check was cleared and then was not cleared means that you can actually flag the firmware before it goes out to millions of people.
And so there's been quite a few putting it politely fuck ups that we've caught before the firmware ever sees the light of day.
And that's exactly what should happen.
Yeah, I mean it's really cool.
Is that because there's this embargo channel that all the OEMs have access to, they can do all this during their development.
And if it's going to be a pre production product, they can use LVFast, they can use WebD, they can make sure that as they're doing their weekly monthly, whatever cadence BIOS is that everything keeps working and they don't have those kinds of fuck ups before they launch their products even.
Yeah, so it's basically a CIC DLL pipeline for firmware essentially.
So yes, because the ecosystem that we're in is kind of broken in that there's lots of IBVs and ODMs and lots of stuff gets.
So a good example is the ibv, the BIOS vendor might build a BIOS and then the ODM like the Wistron, the Foxconn whoever will take that block and they'll add a few things to it like a trackpad driver or a TPM update driver and then they'll give that to be branded by the oem.
And so you lose all this kind of like CI CD ideal, build it in one place and think Dell does things differently.
Dell has their own BIOS team and so they don't use ODMs the same way other companies do.
My first exposure to the OEM ODM world was when we started doing our own hardware in China.
And that's when I really started realizing you just get this blob that probably been touched by five different people and there is absolutely no sember in that whatsoever.
You have no idea.
They're like, well we made some changes.
You're like, cool.
How do I know?
I have no idea.
There's no paper trail.
Hugely relevant.
I've had, I've been arguing about this for I think maybe three years.
So I built this project called Usewood and it's basically a way of embedding software bill of materials into firmware blobs.
Right.
Which is kind of.
It started off being like peripheral firmware like Wacom, Logitech, etc.
But we need this for EUFY firmware.
Yeah.
So I've talked to lots of different IBVs and got lots of different nonsense answers, but now I'm working with one of the USWG EUFY working groups building an SBOM so that they can be compliant with the various different cyber resilient acts, etc.
And the two biggest problems I have is that they, the IBVs and the ODMs don't want to disclose where they got stuff from.
But the single biggest problem is that they don't actually know where everything's come from.
I can imagine that.
And they're usually not very software savvy either.
I would imagine a lot of the ODMs they are more like.
So it's built usually most of the stuff like the Dell and the Lenovos and the hps of the world are doing things properly.
Right.
The cheaper brands like who remain nameless, they literally email things around to get signed.
There's no HSMs, there's no sort of proper security in place and they're getting random DXCs from random places and they don't care if the authentico checksum expired five years ago.
And it's just terrible.
Yeah, no it really is.
It's really, it's really like it's so far behind how modern software is architectured.
That it's kind of terrifying, but I had sbombs on my list of topics, so you kind of nicely segued to SBOMs in general.
But so SBOMs, obviously for modern software stacks is fairly well understood, I guess, because we know, I mean, it's a JSON blob with some dependencies in some modern programming language.
I guess my interest around used SBoMS for firmware is how much of those firmwares are actually using, say, open source libraries where you would Even find these CVEs in the dependency tree.
Or are these all like that are.
Not part of open source libraries?
We have CVEs at AMD all the time.
Oh, no, of course, I meant.
I meant more in the sense of an SBoM where you look at the dependencies of your software stack and you have a CVE in one of the dependencies that will be picked up by an SBOM analyzer.
But if it's all proprietary soft source code in the entire BIOS or firmware, how much was an SBOM actually add value there?
Because you could do analysis.
I have a personal opinion here that SBOM is still very valuable.
If you look at it from like an AMD perspective.
AMD has something called the platform initialization that we give to the IBVs and they integrate into their BIOSes and they give that to the ODMs, the OEMs.
We're the start of that stack for AMD systems.
I think it makes a lot of sense that if we have a CVE within one of our releases, we should have an SBOM that says the CVE was in this release, it's fixed this next one.
And then when the IBV goes and takes this, they can have that metadata, they can pass that on along the chain.
All right, so you're treating the BIOS as a dependency in a sense, then?
Like this particular version of that.
Yeah.
Is specifically the platform initialization code, because the BIOS is a little bit.
It's a piece of the bios.
Right, right.
Okay.
So you create your own CV database essentially with their liquid mapping.
So I'm way more militant.
So the typical UEFI BIOS is maybe what, seven, 800 different EFI binaries.
Now, my stance is that every single one of those binaries and every single one of those microcodes, every single one of those blobs that get imported from somewhere else should be an entry in the SBOM table.
Now, if you do that, and this is kind of basically what we've agreed with the USWG EUFY working group, is that you can do some cool stuff.
So if you enumerate every single microcode that's being used on your system, rather than having to go back to Lenovo and saying hey, is my system affected by the cve?
Because they'll just ask their odm, who will ask the ibv, who will then return the answer back to the odm, who will return their answer back to the oem, who will return their answer back to you.
Three months later you can just say to intel, can you create a VEX file for all the affected microcodes?
And intel can say, right, sure, here's the vex file.
And then you match it against your local SBoM and say, yeah, I am affected by this.
You've cut out the ODM and the.
I have a bit of a different.
Opinion and the reason is that there's non obvious dependencies between different pieces of the bios.
If you go to a modern AMD system, you have so many microcontrollers that are within the soc and you might have awesome code that's on this one microcontroller, awesome code on this other one.
You put them together, you have a vulnerability.
How do you say where that vulnerability is?
If you were just going enumerating single parts, you can never catch that.
You need to have a put it together and there's a vulnerability type of methodology.
So personally I think that's just a rule, right?
That's just like Boolean logic, right?
But I'm saying that you need to have several layers here.
You can go and do what you're saying, but I think you need to still look at the layers as they get put together.
I think in the common case where it is there is some proprietary code that's doing, I don't know, insecure SMM access or something.
We know that same DXE with the same file hash, ie, the hash of the C files and the H files that built it is being used in a dozen vendors on 100 different models.
So if we had a, if you're a security researcher, you can be like, okay, well what systems are affected by this SMM escape?
And you can run that file against your local SBoM and you know straight away, yes or no.
But you can only do that if you list every single DxE or PIM that is included in the system firmware.
And I think this also comes down to the lens of where you're executing the code.
Because when you're talking about Dixie code that's running on x86 cores stuff I'm talking about is running outside of that.
It's running on other microcontrollers.
And so the attack vector is very different.
The type of escapes you have there are very different.
Yeah, I think it makes sense though, because you can like.
So I was talking to intel about psp, not psp, the other one csme, and they're quite.
No, it wasn't fsp.
Fsp.
There we go.
I was talking about FSP and they're quite happy putting a SBARM entry for their FSP and they're quite happy putting the SBARM entry in the microcode itself.
So I think if we can embed stuff as low as possible, even stuff that's not running on the host cpu, just include that.
Yeah, interesting, because I think that was in the fallouts of both Logo Fail and pixiefail.
One of the biggest things was like there is no public database of all affected biases because we don't really know what they are.
Right.
Because they are.
They know.
I think AMI was heavily affected and some other vendor was heavily affected, but because of the supply chain is so complicated, there's no easy way to say you're affected or not.
So one of the things I'm really pushing for on the LVFS is shaming the vendors that don't include the Nest BOM and praising the ones that do.
So literally, for every single product page on the lvfs, when the vendor uploads a firmware, we look for the at SBoM, we validate it and it gets a green tick or a red cross.
And most of the firmwares right now have a red cross.
It's either not provided or it's a token gesture.
There is one thing in it which is not true, right?
It's microcodes, there's FSPs, there's all sorts.
And so if there's anything that a marketing team doesn't like more, it's a red cross next to that product.
You're right about that.
Okay, let's go back to like, the fundamentals.
If I am a new vendor and I want to get my firmwares into the database, what is the procedure for doing so?
So LVFS website itself is hosted in GitLab.
And so the.
So the.
Go to fwt.org, read the details about how to apply for an account.
The accounts are completely free.
We usually ask for quite a bit of details like what's your psert team?
Can we use your logo agreement to sign?
And stuff like that.
There's Like a legal agreement that we are legally allowed to not only provide your firmware for download, but the people we provide it to are then allowed to redistribute it themselves as well, which sort of solves the mirror thing.
So there's various sort of agreements that need to be done, but it's pretty easy.
We have to do some due diligence as well.
Basically checking that company is legit that it's not a big giant trick to try and get an account and then do exploits from the inside or something.
So you might involve literally ringing someone up or talking to someone's boss or whatever.
And then once the user account is created, the legal stuff can be done kind of asynchronously.
After that, when the legal stuff's cleared, the user's allowed to push stuff to testing and stable.
Before that, they restricted to private embargo.
Once there is a vendor account, the vendor manager can then create as many users as he needs or she needs.
So there can be different account levels.
There can be like a basic uploader which is just able to upload firmware, nothing else.
There could be a QA user that can move stuff from embargo to testing.
There can be other vendor managers that can add users or take away privileges, etc.
So it's quite a useful easy thing for vendors to do.
What blows me up?
At least for vendors that have plugins already supported by fobd, like if there's somebody's a new UEFI capsule they want to put out, then they can go follow this path really easily.
If they have a new kind of device that they want to support, it's a little bit different because you have to have a plugin written and you either have to write that yourself, get a consulting company to help, get one of us to help have some specs out there.
There's more steps to make all that happen.
But I mean, we're very forthcoming about trying to help as many vendors as possible.
If you look at how many types of devices are supported now, it's astronomical.
But what is amazing, this is the thing that I keep getting asked as well by vendors, they say, well, look, we want to get our dock on the ivfs and it's got a Synaptics MST chip in it and it's a Thunderbolt Goshen Ridge and it's got a via usb.
What code do we need to write?
And I know we've written all that code.
You need to add six lines to this file with your vendor ID product IDs, it'll magically just work.
And they just sort of look at you blankly as if to say, I don't understand.
Like, we expected like engineers and code and NDAs and stuff.
And you're like, no, it just works and it just explodes.
Okay, well, the client side works.
We've tested it.
How much does it cost to be on the RVFs and distribute, say, a million firmware updates?
Well, it's free.
It's actually free.
Again, they're just like, I don't get it.
Right.
So they're like, how does this work?
And I was like, well, the cdn provides like 100 million downloads and it costs like $60 a month or something.
Right.
And like, it's.
It's like, I think they're highly heavily.
Cacheable assets because they're usually small blobs.
Metadata is hugely cacheable.
Yeah.
And it's GEO replicated and it's fine.
And the firmware isn't that big.
And like, even like 100 million downloads of the firmware is not hugely expensive.
My time is paid for by Red Hat.
Red Hat obviously owns me and kind of what I do.
And the Linux foundation provide all the sysadmin stuff.
And so there's not really any fee required or necessary.
Right.
So I want to bring back to something you mentioned earlier, Richard, about.
You said the flashing part is the easy part.
Right.
The part between flashing and downloading is identifying what you're actually trying to upgrade.
You kind of alluded to the IDs and the various things.
Can we speak a bit more about how that mapping happens for various types of firmware?
Yeah.
Mario, do you want to do this or shall I?
So, at least for UEFI firmware, the way that we do it is that there's something called the EFI system resource table esrt and the vendor will create a unique GUID for that system.
And that means that if dell has a 14 inch and a 16 inch of the same family, they've been to the same GWID because it's the same binary.
But if they have a 14 inch of one year and then a 14 inch the next year, they'll probably change it because it's different binary.
And so we'll go and match that in the client or in the daemon to see whether or not that's going to be the right system to apply the firmware to.
For other kinds of devices, for usb, we go and look at the vendor id, the device id, PCI devices, we look at vendor id, device id, and in Some cases we go and add extra guids in place that are for the individual OEM that's going to be doing something.
As an example, we have Support for AMD GPUs and we have extra OEM boards.
And so you could say MSI has this board and ASUS has this different board.
They both have the same Navi31 chip in it.
How do you make sure you only apply the firmware to the ASUS board?
That's because we can go look up the board ID and build a GUID from that board id.
And we apply that same concept across all kinds of devices, across flopd.
Okay, so imagine, okay, I'm just gonna bring it back to my own scenario.
Like, we have an OEM in China, we make our own boards, we have our own bios.
The bias is an AMI bias behind the scenes.
And if I want to get my bias updates into the database, what needs to happen for me to brand that as my device versus well, the identifier that the OEM provides, is that a bias vendor ID in the bios or for instance, or how does that actually look like.
Like you're saying branding.
Like, where do you want this branding to show up?
Oh, no, I mean saying, like, if we wanted to do bias updates, for instance, on our devices, and we have the firmware, we have the blobs, but we want to map it so we can use FW update for bringing those bias firmwares down to devices.
How can, how does that mapping.
What needs to happen for the mapping?
So do you have to have like our own company firmware update in there?
Or if I'm making my own hardware, I need to have that firmware update mapped to that so I can have my own entry.
Or how does that.
If you feel like a smaller.
You would basically be making XML metadata that matches the ESRT entry that's on your system.
And when you go and upload this to lvfs, it would go and show up as an entry over there and you can put whatever string you want to title it there.
And so when somebody sees an update available, they'll see ACME Device right there.
If you call it ACME Device, the user doesn't really ever see the esrt.
That's just how we internally map it.
Okay.
Okay.
There's also another thing maybe you're alluding to as well is it could be that there's two systems, both with the same ESRT GUID or both the same VID and PID or whatever, and something like a Touchpad.
Right.
A touchpad on a laptop.
It could be exactly the same i2C address on Dell and on Lenovo, but they have different firmware.
And so for an embedded device like that, what we can say is, we can say match that i2C address, VidPiD, etc.
But in the firmware you can say only match if your DMI data is del.
Right.
Only match if your DMI data is Lenovo.
Not on the firmware, in the metadata.
We say that the metadata.
Sorry, yeah.
And so on the lvfs, you can add that directly and just edit the firmware on the lvfs.
Or you can do that as part of the pre upload.
Yeah.
And this is something else that exists in Windows today as well.
This is how Windows does Windows update to make sure that in case this ever happened, that you had two things with the same identifier, that you can only apply it to certain models.
And we use the exact same concept and we generate the strings the exact same way that Windows does.
So there's the opportunity that if these companies are putting out UEFI updates for Windows, they'll say, this is they call the chid.
This is the CHID we use on Windows.
We use the exact same CHID on Linux when we do LVFs.
Yeah, that's hilarious.
Like, there's a whole blog post on how I reverse engineered how Windows generates those chids.
That's.
That's funny.
No, so the, I guess what I'm getting at is like all the people that resell Chinese boxes, because that's kind of how the, a lot of vendors, harder vendors use it.
Right.
They buy from some OEM in China and then they get some buyers blob from there.
But because they're not Dell or Intel or they don't really get those firmware updates over there in the same way.
So that was kind of like where I'm alluding to.
Like, how do you kind of solve that problem where you can tap into this ecosystem.
But I mean, I think that for those kinds of systems, you have a process problem.
You need to build that relationship with where you're getting the BIOS from and make sure that you're getting quality bioses out.
You don't want to be pushing out random stuff that you get that hasn't been tested.
So you build that relationship.
You make sure you have QA to go with it before you start doing this.
Oh, no, that's not what I meant.
I meant more.
There were probably 20 different products with the same hardware used for different configurations.
So that's kind of like.
So when you map on the vendor ID, for instance, it might be the same across 20 different use case for this particular board.
Tricky.
So if you have like 20 different vendors all selling the same device, with exactly the same firmware, with exactly the same hardware IDs like the DMI string, etc.
Exactly.
That makes it really hard.
And I don't think we've got a very good solution for that because we don't.
If it's not in the DMI data in smbios, to us, it's identical.
There's no way of telling apart one white label versus another white label because they are electrically and from a software point of view, identical.
Yeah, exactly.
And that's the tricky part.
So let's go back to the upgrading flow because we spoke about biases and we spoke about various other things.
So what kind of flashing, I guess groups of flashing procedures, upgrade procedures are supported currently by.
By the project.
Are there.
Yeah, like speak a bit more about that, dude.
There's like 140 different protocols.
Oh, wow.
Okay.
Can they be grouped in some way or other generic trends or are they completely different though?
I guess, like in.
In trends, I guess the Update Capsule is probably a big one on its own.
But even with Update Capsule, there's actually three different protocols within that.
There's MV ram, which is like the kind of more clunky way of doing it.
There's Capsule on disk.
Yeah, very common.
There's Capsule on disk, which is more common with the arm 64 boards now.
And there's also.
You can chain load Grub as well.
So there's three different.
There's like three protocols within one protocol.
So some Wacom.
Wacom have two protocols.
One for their raw i2C tablets, one for their USB tablets.
Logitech have, I think four different plugins for four different product ranges, like Bolt Unifying, there's Rally Bar and others.
But it seems like every different.
Like the biggest problem is with vendors is everyone thinks their way of updating firmware is unique, beautiful, and better than all the other vendors.
It sort of comes to a head because you can go into a meeting and they're like, we want a FW plugin.
And I'm like, well, tell me your update protocol.
And they're like, well, we can't tell you what our update protocol is because it's proprietary.
And I was like, okay, well, let me have a guess right now.
What?
So let me.
So, okay, well, Is it a USB device?
Yes.
Does it use hid?
Yes.
Okay, well, I guess your protocol is you've got a magic command which turns it from runtime mode to bootloader mode.
You've got another command that erases a block, you've got another command that writes a block, you've got another command that reads a block and you've got another command that turns it from bootloader mode to run loader mode.
And they're like, how did you know our protocol?
Because they're all the same.
Well, I guess there are finite ways.
To do it, right?
Yeah, absolutely.
So there's lots of plugins have been developed using Wireshark.
I mean going and doing a PCAP in Windows, see what it does.
And like, oh, we're doing this wrong, let's switch this over.
Literally it'd be like, oh, it's CRC32 using a different initial start or it's MD5 or SHA1 or we've done the.
The numbers have to start at one rather than zero.
Like there's so many variations, but they're all basically the same.
Right, that's interesting.
Let's switch gear a bit to vendor support.
Right.
So it's by no means generally supported to have my BIOS updated or my drivers updated for my firmware on a, I don't know, pick insert blank vendor.
The support is, I mean Lenovo has some, I think the decent support for.
So for the car X1 carbon for instance, I think has decent support for it.
But speak to me about my like how the adoption looks like there.
What's the current supporters name and shame some vendors are doing good and somehow doing well.
So I guess like it kind of comes down to when they joined.
So like Logitech joined fairly soon and they had quite a few devices supported including the unifying the little receivers as well, which there was a big vulnerability for.
Dell joined very early Lenovo, then hp, what's really driving it now is works with Chromebook.
So Chrome had this, Google had this problem where on a Chrome OS device a good chunk of their NNSC root file system was firmware updaters.
Because every, I don't know, Western Digital would say here's an NVME updating program, it's only 50 megabytes in size and it updates all Western Digital NVME drives.
And then you'd have, okay, well Samsung would have the same thing, 50 megabyte.
And then Wacom would say here's our 50 megabyte, everything statically linked blob and it was ridiculous.
Both in from an auditing, the code point of view, from maintaining the code point of view, from just a disk storage point.
And so when we said oh yeah, we have an NVMe plugin, it's 25 kilobytes in size and it updates all NVME drives, it's just like it blew them away.
So I think it was last year, maybe the year before they added FWD and LVFS support as part of works with Chromebook.
Oh wow.
Which means that all of the vendors that were supplying stuff for Chrome OS were basically like, oh wow, we have to do this now.
And because the code and the LVFS and everything is shared, we get this for free.
Right.
So all of this is why Red Hat's obviously interested in me working on this.
Because if offender's providing the firmware update for something on Chrome os, yeah, we get it for RHEL and Fedora and everyone else gets it on ubuntu and Debian, etc.
Yeah.
And so it's really nice little tie in.
Something else that's kind of interesting with Google is that they didn't start using FUPD inside Chrome OS until Dell did the WD19 docs.
Those were the docs that shipped with the Latitude Chromebooks that were done a number of years back.
And we did all the development for the Dell DOC plugin while those docs were under development and we notified Google about it at that time and that was their bring up to fluffd and said Chrome OS and they said oh wow, this is such a great experience.
And it's just grown that they want to support everything because as Richard said, all these other devices are supported as well.
They should just have one experience for updating all devices.
Right now there is, it's literally built into Chrome os.
There's like a GUI where you can click stuff and go.
At the moment only a few devices kind of allow listed but like the plan is this year to like blow the covers off.
It's kind of one of the reason for the signed reports is that if a vendor's done a signed report on Chrome OS, it automatically goes out to Chrome OS users.
It doesn't have to be gated by Google anymore.
That's really cool.
So obviously Google is not necessarily a hardware vendor, but who's the most supported hardware and vendor, I guess who's actually has as near complete coverage as possible in terms of what firmware is that are updated on these devices.
So it's a hugely commercial decision.
So Lenovo support, Linux inverted commerce on specific laptops, like most of their ThinkPad range, ThinkCenter.
The plan is Think System as well.
Dell support it on some of the Latitude Inspiron, but not everything.
So it's a commercial decision because you have to have QA people testing staff and there has to be like, commercial need for it.
So some of the smaller vendors, it's hard when they're like, okay, well, although it's a free service, we still need to test stuff.
Yeah.
And there's now Windows Update.
So most events don't even use Windows Update because it's firmware.
Not important.
Right, right.
If you look at the view from like a hardware vendor perspective, you're selling a $500 laptop, are you going to go and spend extra money on a QA resource to test this with linu, when that's probably going to be sold at best buy for $500 and people are buying that just for their kids to use?
Yeah, yeah, absolutely.
And that brings me to another thing, which is we talked about desktops and laptops, but there's obviously a whole range of server in the server world as well.
How does that landscape look like?
Is that option bigger there, smaller there, or how does that actually look like?
So I'm smirking because there's so much politics there.
Right.
It's like, this is the sort of thing I could tell you in a bar, but I couldn't tell you here.
So it's.
Well, I'll catch you on the next conference.
Yeah.
So, like, the short answer to the story is there's not much server stuff, but one of the major OEMs spent a lot of time and money making the Redfish plugin work, which is a way of like, basically doing like.
Like, it's like the successor to IPMI.
And now there's three tier one OEMs that have been testing this stuff, but they haven't actually committed to making it.
It's so frustrating because there's so much stuff that's there, but it's not public yet.
Right.
And so this year, maybe last year.
No way.
This year, maybe next year, certainly.
And I'm kind of waiting for the first vendor to sort of like, say, yeah, we're doing this, which then elevates them from a purchasing point of view and then for the other vendors to be forced to do it as well.
I mean, it's kind of what happened with the UEFI capsules, because Dell was doing it on so many.
It got picked up by all the other big OEMs as well.
It is a kind of chicken and egg chain reaction as well.
So you need one vendor to do it first and we're tantalizingly close to having the vendor because I think what will happen is realistically it'll get put in some purchasing requirements somewhere and one vendor will do it and they'll win the contract and the other vendor will just copy.
You mentioned something interesting, which is where you said Redfin, which is.
You said it's replacing IPMI Redfish.
Yeah, and you said it's a.
I've never heard of that actually before.
So it's an IPMI replacement you said.
But that's actually something very interesting because IPMI on a traditional server lives in the firmware space, not in the software space usually, which is.
Makes it very well, I guess it's an in between lan but makes it very interesting use case for updates for the firmware as well.
Which I guess never really or rarely really happens.
Right.
It gets more mad than that on two different levels.
So to actually talk via Redfish.
Redfish is usually talk to Redfish either through like an external like network connection to the server.
We literally separate RJ45 Jack but most people just use the internal network card.
So the host OS can talk to the BMC which is running the host OS through this internal network card.
And then you have to use a username and password.
Hopefully ephemeral, but not really.
But you have to use the legacy IPMI to provision yourself a Redfish account so that you can then switch to Redfish and then do the firmware update stuff.
So that's super fun.
The other where it gets more crazy is someone inside Google Data center said rather than the host OS running fwupd talking to the BMC and then doing firmware updates via the bmc, could you just run FWUPD on the bmc?
All right, which I don't know if that's genius or insane, but we did it all right.
So I built OpenBMC and included there's a few little fixes we needed for FWT to just build and run.
And it runs like it actually runs as part of OpenBMC now just the YOCTO thing.
So you can actually run it on the BMC itself independent to the host os.
So you would use as like an update operating system almost to.
You could communicate with.
Yeah, I'm not sure.
Like it basically means that all the flashing tools that we've built for the host OS, things like NVMe and things.
Things that might be accessible to The BMC can be updated from the BMC rather than from the host os.
It all depends on who owns the resource.
I'd really be interested to see what they really flash there though, because, like, if you think about NVMe, we rely on kernel IOCTLs, and those are in band dioctyls.
If you're trying to do NVME through the bmc, it should be a very different flow.
Yeah.
So the use case that was presented to me was mtd.
When you were literally using exact same part.
You're updating the system firmware over MTD from the bmc, but you're using all the same MTD stuff that you normally use on the host OS but on the bmc.
So it's bizarre.
I'm not sure if it's like, I don't know if anyone's actually done anything with it, but when we break it, people notice.
And we tried to run CI configurations for all these weird things that exist because we don't look at these all the time, but we want to make sure that we do our best to not break them.
Yeah.
So a good example is Windows.
Someone said, look, can you build FWD on Windows?
I'm like, probably.
And so we added it as a CI target.
So now you can just install like the FWD setup exe and you get a command line only, but you get a command line thing that can run some of the protocols.
Like not NVME because it uses ioctals and stuff, but dfu.
All the vendor USB protocols.
Yeah, all the stuff that's like the vendor specific stuff, so that's kind of useful.
And then someone else said, look, could it run on Mac os?
So we added the missing bits to make it work on Mac OS and it now works on Mac OS too.
So it's part of the BREW system as well.
Nice.
That's really cool.
So the last thing I kind of want to chat about is what's the most memorable or craziest use case so far in the whole project?
Either like insane reverse engineering required or just a bizarre use case for like some really bizarre hardware.
One pretty weird one was a smart mirror.
There's a company that makes a smart mirror and they said, we want firmware updates on our smart mirror.
I was like, okay, that's a pretty bizarre one.
Wells.
There was an oil and gas company that wanted to update firmware with the nodes are buried like 60ft underground and they wanted to make sure the AB booting worked okay.
Other bizarre ones.
Yeah, interesting ones indeed.
Guys, we are Kind of on time here, so I don't want to take much more of your time, but this has been super interesting for me.
And anything else you guys want to add to the show before we wrap up?
Anything?
Call to action.
People need help for testing.
Anything you want to say thank you.
To Mario because Mario's been like, co maintaining right from the start and I only recently added him as an official maintainer.
But without Mario, I would have given this up years ago.
So this has been awesome.
Yeah, it's been a fun project to work on.
I mean, it's not my day job that I do this all the time.
It's just something that I do part of the time and I have other stuff I do at AMD all the time, and it's just fun to be part of.
So for the record, there's only one person in this, on this planet who can tell me that something that I've worked on for two weeks is complete shit.
And that's Mario.
Amazing.
Well, good stuff, guys.
Very much appreciate you coming on the show.
So thank you so much for taking time out of your busy schedules and I'm sure I'll bump into you guys at some conference sometime soon.
So thank you so much, guys.
Have a good one.
Cheers.
Bye.
Found an error or typo? File PR against this file or the transcript.