Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Unlocking Firmware Secrets with Christian Walter: BIOS Vulnerabilities & Security Insights
Play On
30 JUN • 2024
1 hour 4 mins
Share:
In this captivating episode of “Nerding Out with Viktor,” host Viktor Petersson sits down with Christian Walter, a renowned expert in firmware and IT security from 9Elements. With over six years of experience, Christian offers an unparalleled deep dive into the world of firmware, shedding light on the most pressing issues facing the industry today.
As they kick off the conversation, Christian shares his background, highlighting his role as the leader of the firmware development department at 9Elements and his involvement with ventures like a company focused on firmware testing and the Open Source Firmware Foundation. This glimpse into Christian’s expertise sets the stage for an engaging discussion that delves into the intricacies of firmware security.
The conversation takes a fascinating turn as Christian delves into recent high-profile firmware vulnerabilities, such as LogoFail and PixiFail. With his exceptional technical knowledge, he breaks down the technical aspects and broader implications of these vulnerabilities, underscoring the significant security challenges they present. Viktor and Christian engage in a thought-provoking discussion about the urgent need for improved security measures in firmware, making it an essential listen for anyone passionate about IT security.
The conversation then shifts towards the complexities of BIOS security, with Christian highlighting the reuse of code bases across vendors, the challenges in auditing, and the lengthy disclosure periods for firmware vulnerabilities. This insightful exploration of BIOS security provides listeners with a nuanced understanding of the industry’s current state and the pressing need for innovation.
As they delve deeper into the world of Trusted Platform Modules (TPMs), Christian provides an engaging overview of the different types of TPMs, their role in hardware security, and recent vulnerabilities. He shares a startling revelation about how attackers could unseal Intel TPMs by reassigning pins from user space, making it a must-listen for anyone invested in the future of firmware.
The conversation takes an exciting turn as Christian introduces the innovative Firmware CI project at 9Elements. This revolutionary project aims to transform firmware testing and validation by offering a user-friendly, scalable framework for continuous integration and testing on actual hardware. Viktor and Christian engage in a lively discussion about the importance of modern tooling in firmware development and how this project bridges the gap between traditional firmware practices and contemporary software development methodologies.
As the episode draws to a close, Christian shares his thoughts on the mission of the Open Source Firmware Foundation and its efforts to unite the industry in advancing open-source firmware. This inspiring conclusion underscores the potential for collaboration and innovation that exists within the world of firmware.
In this enlightening conversation, listeners will find valuable insights into the world of firmware security, BIOS complexities, TPM vulnerabilities, and innovative projects like Firmware CI. Christian Walter’s expertise shines through as he provides a unique perspective on the industry’s current state and its future possibilities. With Viktor Petersson at the helm, this episode is an absolute must-listen for anyone passionate about IT security, hardware, and the future of firmware.
Transcript
Show/Hide Transcript
Welcome back to another episode of nerding out with Victor.
Today I'm joined by Christian Walter from nine elements and many other things.
And today we'll be diving deep into the world of firmware and hardware.
So welcome to the show, Christian.
Hey, thanks.
Hi, Victor.
Thanks for having me.
So maybe for the audience, do you do a quick introduction to yourself so people understand a bit more your background and just paint a little bit of the conversation I guess we'll be having in the next 45 minutes an hour?
Yeah, sure.
Yeah.
My name is Chris, obviously, and I'm, you know, I've been around vmware for a couple of years now, six years, seven years now.
So my background is a little bit on it security.
Right.
I studied it security a decade ago now and, yeah, worked, but I was always amazed by hardware.
So I did like a lot of hardware security in terms of side channel techs and this kind of stuff, and then kind of emerged into the firmware space.
And since five, six years, I'm at nine elements now leading the department that does firmware development and also do a bunch of other things there.
So I founded another company now that does firmware testing as a product.
A couple of years ago, we founded the open Source Firmware foundation, which is a nonprofit organization in the US and all these kind of things.
So basically my day, my week, Monday to Sunday, that's all about firmware and everything in between.
And the focus is really on open source, pushing open source firmware forward.
That's kind of the amazing.
Yeah.
So I was introduced to you by David Hendricks from Coreboot and the reason why initially I wanted to have you on the show before knowing more about your background, which made me even more interested in having you on the show, was around bias security because I asked David, who I should speak to kind of chat about the whole new wave of bias vulnerabilities that we've seen in the last 18 months or so by now.
And your name came up as a good person that can both speak about it from a technical perspective, but it's also good at understanding the big picture of the whole ecosystem.
So maybe that's a good starting point.
So we've seen in last twelve months, we've seen logo fail, we've seen pixie fail.
Let's just start there.
Let's talk a bit about them.
How did they, what happened?
What are they?
Let's start with Logofell, perhaps just describing what that is and why it's so, I guess, so critical and.
Yeah, like why it really should be drawn more attention to.
Yeah, yeah.
Okay.
That's a lot of pressure now that you're bringing in here from David, right?
Yes.
So logo fail is a vulnerability that got, you know, recently published I think from binary really on a couple of firmware winners allow you to know, customize your bios, right.
Firmware that is basically, and how they, how you can do that is that if you boot up machines you probably want your own, you know, boot logo showed in the boot process.
Right?
So it's nice for, I don't know, corporate companies let's say, right, you're a big company and you want all your corporate machines having the same logos, right?
So you want customize that it's.
Right or it's nice for people like us, right.
You want your hacker logo shown why you're putting up your Lenovo instead of the red Lenovo sign.
And so, and these companies actually give you a way on how you could customize that by uploading basically images into the firmware, right.
Saying we have a couple of restrictions, it needs to be that in that format and that's the only restriction and resolution.
Probably two resolution.
Probably two, exactly.
However, there is a vulnerability in the parser of the images.
How that works is you write that into a specific partition in your firmware, in your flash drive where the firmware is stored and it picks it up from there and tries to display.
And while parsing it doesn't check the boundaries of the image.
It only cares about is it a png, is the file N png?
And if not, off you go.
This leads to that.
You can basically execute any code there because you can trick, you can work around the parser and you can basically execute code out of that memory space that is dedicated for that.
So you didn't even need to fake the mime type.
It's just so naive.
It just looks file extensions, that's how rudimentary these checks were.
Yes, exactly.
This is a huge problem and I think that has been there for many many years.
And this was recently discovered by binary, which is part of the, or who are part of the current development that so many vulnerabilities actually get exposed.
Right, because they are doing a pretty good job in actually doing.
Absolutely, yeah.
So that's, and the blast radius for this vulnerability is pretty massive, right, because it's basically game over once you can get to that level of intrusion in the device, right, exactly.
I mean if you think about it from a security perspective, right?
You have a couple of let's say measurements in firmware that you can then actually can do to counter any attacks that are happening in firmware.
There's like the whole secure boot process, right?
There's these, you know, into boot guard and all these kind of technologies that you have, however you circumvent all of them basically because you are not implanting anything in firmware as you're not overriding anything, but rather you're just executing it in memory.
Right.
And then, and that means it's really not preventable in that sense.
And you don't catch it with any other, with any of the technologies that are actually there.
So.
And this really like has a massive impact on your, on the security of your systems.
Right.
And as you said, I think all big ipvs basically were affected by that.
Right.
And then that shows you that some probably share the same code base, right.
So, and that's like a huge issue.
Right?
And this, this image pardon algorithm, let's say, that gets used on, you know, all platforms basically, right.
It's one of the core things basically that is, which is part of the value add that you get by going through these ipvs.
And that is really, that has a mess of it.
And that was something that came up in the episode I had with coreboot.
So with David and Matt.
And one thing I didn't really quite know about the bias structures is how much of this code base that is reused cross vendors.
So it's not just like AMi would have their set of binaries, they share across their biases, but it's almost like open source components that are dealt between vendors as well.
So the blast radius span so many vendors.
And to make matters worse, I guess at least from an audit perspective, is there's no easy way to tell if you were affected by it because there's no s bomb or something that you can say, oh, this component is used in my bios.
It's just like you were affected.
Maybe, maybe if you have a good bios vendor they will tell you.
But more often than not, if you bought your device through an ODM or some supplier, you would most likely not even know that you're available.
Right.
Which is biosecurity is fascinating that way.
Yeah, exactly.
Right.
And this is, I think David explained that pretty good in the other episode, right.
This is how because of the whole supply chain on what you have.
Right.
So, and actually there's an interesting paper from the uefi.org, right.
That talks about this as well.
But what they're trying to do is trying to make a case to.
So if you fight a CVE.
Right.
There's normally something like a 90 day period.
Right.
Where after 90 days it gets disclosed.
Right.
And, yeah.
You have to fix it for U of I security vulnerabilities.
It's 300 days.
Right.
So that's like, wow, three and a half.
Right.
Or something like that.
Yeah.
And that paper kind of talks about this.
Right.
Why this is exactly the case.
And this is because there are so many parties involved in that ecosystem.
That supply chain is so complex, basically that tracking down who owns the code.
Right.
And then it has to be fixed somewhere.
And then it has to kind of flow back through all the stations again until you can actually release it.
It takes so long that 300 days should be like the period until we disclose or they disclose these kind of things.
Right.
And that means a security issue that gets discovered now, right.
Yeah.
They fix it next year.
Right.
In March.
So this is, and this is pretty amazing.
Right.
And I think David really explained that a lot.
Right.
You got like the silicon vendors right there doing code.
Right.
You got like the ibvs, they do value audio.
The oems.
The Oems, right.
And then it goes happen, then you burn it on flash at the end.
And this is like, that whole chain is extremely complex.
Everything is proprietary in that.
Right.
Sharing code is really sometimes on a zip file basis and this kind of stuff.
So that supply chain is brutal.
And if you find an issue there, it has a huge impact because fixing that issue takes so much time and so much effort that sometimes it's not worth it in that sense, because they say, okay, it's an ado machine, whatever.
And also one thing that was fascinating to me is the amount of regressions you're dealing with as well, because you might fix something, but then because things are not version, we're not talking to git repo here.
The most sophisticated vendors, they might have a git repo for their source code, but so much of this just dealt with files on some file share.
So you pick that and you patch it, but then you forgot the other patch and then you have a regression instead, right?
Yes, yes.
Yeah.
I mean, we're all humans, right?
And naturally we make these mistakes.
Right?
Yeah.
But we have some tooling that could help us to prevent these kind of things.
And I want to dive.
Yeah, I want to dive into that in a second.
I want to kick that kind of liver down the episode.
But, yeah, absolutely.
That's, that's another thing that I wanted to cover here.
So the other thing is pixie fail, which is another one that happened.
I think it was disclosed what, three months after logo fail or something along those lines I would imagine.
Talk a bit more about what that is and how that attack vector looked like.
Yeah.
So pixie fail, I think it's a bunch of vulnerabilities.
I think they released nine in total or eight.
Eight or nine in total.
So it was a chained attack.
No, it's different vulnerabilities that have impacts basically, but they kind of all, I guess explore the same concept, let's say as probably know how to pixie boot a device, right.
So that's kind of the DHCP server.
So it's basically network booting.
Right.
That's like the core thing on pixie booting and what happens under the hood is the DHCP server when they send out their responses.
Basically there are fields in that response that tell your device where the PKC server is.
So where can I load my images from and this kind of things.
And there again, Washington, you could craft a DHCP message that when the, when your firmware parses this message you can kind of escape the parlor and do code execution again because they're also, again, we didn't check for, or they didn't check for boundaries correctly and you're kind of escaping that and can basically execute everything.
And the crucial thing here is you don't have to have access to the device rather to the network.
Right.
This is kind of, which makes it for me at least a little bit more critical in that sense because for logo failed you have to have access to the device and probably elevated access in terms of you need to be able to run the tooling that you can update the boot logo for Pixie fair.
You don't have to do that.
You have to be, you know, part of the network and you have to be able to send these messages out and so on and so forth.
And, but this gives you just like whole other privileges that you need basically, and access that you need to device.
Right.
And it's basically remote code execution which is pretty brutal, right?
Yeah.
Again it's, I think the vulnerability was found in the upstream EDK tool repository and EDK two is an open source implementation of the UFI spac and that also gets consumed by the SoC vendors and then again drips down the supply chain basically and lands in your device.
And that's again one of these attacks that has huge blast of, you know, really everyone is affected on that and we need to fix the supply chain, you know, walking up the whole thing chain, fix up the upstream repos and then it kind of drips down into each and every device.
And was this an attack vector purely exploitable by the fact that I guess most modern motherboards have built in NIcs on the motherboard.
If you had say an external network interface, would you still be vulnerable?
Or is that kind of like that tag vector is that you get in and because it's kind of in the biospace so to speak, why it's vulnerable or is that unrelated entirely?
I think it's not related to if you have a NiC on board or if you have a separate PCI card that has a NIC on it.
Right.
Because once you, I mean, UEFI is a whole operating system basically, right?
You have all the drivers available.
You.
Have all nics available most of the time.
You have all the USB ports available.
So basically you could also, if you have these USB dongles that you can plug in the network cable, you can basically boot from that as well.
It's really not bound to one of the nics.
You can in firmware, of course you can dictate everything and said, okay, we only do pixie boot from that specific nic.
This is for sure possible.
However the attack itself is not limited to that.
Right.
So it's just a channel into the BIOS essentially boots into a special boot shell essentially that you then have direct access to the BIOS essentially, yeah.
Okay.
Okay, so the most interesting question to me when we look at both logofile and Pixi fail is why now?
We've been having biases for ages.
But then over the span of eight weeks, eight months or six months, whatever it was between them, we had two that are like, holy shit.
Vulnerabilities in the BIOS space after.
I think Logofair was the first real severe BIOS vulnerability that I ever heard of.
And now we have two in the span of like six to eight months.
So why now?
What changed?
Is it just more eyes on it?
Is there something that changed in general or.
Yeah, what do you make out of that?
I think it's more eyes on it.
Right.
So really I do see that.
So for a very long time, firmware, you know, had one job, right?
Boot up the device.
That's right.
And no one cared about it.
Right.
You have it there, right.
Everyone cared about.
Once you hand it off to the bootloader or to your operating system, you know, you care about that stack basically that is running there.
Everyone tried to make that as secure as possible, as open as possible.
But firmware was just there, I think, and it was just, you know, it had the job on booting the device that.
Right.
And I think the us government, you know, made was there was like a presidential act that stated firmware is critical software.
Right, right.
We should look at it, right.
And there's this nisten, I think it's called the NIST 800 paper or guidelines, which kind of stack.
Okay.
We need to take care more security firmware.
Right.
There should be certain things that firmware needs to deliver.
Right.
Or there are certain promises that firmware needs to stand upright.
So you need to be able to downgrade, upgrade and all these kind of things, transfer of ownership and so on and so forth.
That whole topic just gets more attention.
Right.
There are a couple of companies, binary for example, did like a huge financial round, right.
And did some marketing around there, finding exploits and this kind of stuff.
So there's really more going on in that space, I think.
Right, so firmware.
But firmware attacks themselves, right?
They're here for years already.
Right?
Yeah, I saw, I think I saw a talk about 1012 years ago where they were attacking AMD Cube, this kind of things, right?
So, yeah, it's there for a very long time.
I just think that people care more about it now, right.
Because it slowly drips down into the brains of the people that security starts in firmware, right.
It has to, because everything is rooted there.
And if you don't take care of that part and do it in a right way, basically, then it will get you.
Right.
And there's no way to circumvent that.
There's no way against this.
So I think that's, for me at least one of the main reasons.
Yeah, I mean, that's the reason why at least I got excited about things like coreboot and various tool open source buyers and all these things like last year when you started to look into exactly this user space.
Security is completely moot if you can't protect the first boot process and biases and all things firmware, really.
So I think it's very good that we've seen a lot more attention drawn towards this in particular around tpms as well, because that's tightly coupled with the bias, at least with discrete tpms.
So maybe that's a good segue into tpms because I saw a post, you post on LinkedIn the other day, which I think was really interesting about Intel TPM attacks.
But maybe let's start with a TPM like what's the primer like?
Talk about the different types of tpms, and then we can dive slightly deeper into this news that you shared with these attack vectors.
So TPM itself is a trusted platform module.
So.
And what it is, it's a specifically hardened chip, let's say, which should anchor your security, right?
So the idea is that, okay, all you have on your normal cpu and whatsoever, right, that's like general proposed stuff.
And you need to have something that is specially made for security, which is tamper proof, at least tamper evident things.
When it comes to security, you probably need that.
So you use that as anchor for your whole boot process.
If you think about booting a platform, it's always that you need some kind of anchor or root of trust where you start from.
There's always something that you need to trust.
There's basically no way around it.
There's always something that you need to trust.
And then you build up your chain of trust basically from there.
And the TPM has a couple of interfaces to your cpu, let's say.
And from a functionality point of view, you can actually measure things into a TPM.
So there are a couple of registers in the TPM.
And how that works is you basically write a hash to the TPM.
And what it does under the hood is it takes the value of the register that is already there, concatenate the new hash with it, and hash it all together again, so that you just do not just copy hashes over, but this should prevent replay attacks in that sense of.
And yeah, so you have these couple of registers, you have like an NVRAm where you can store stuff.
And TPM has a couple of functionalities that you can use to do that.
And there's of course like a discrete TPM, which is like a dedicated chip, right, that you have there.
There are firmware tpms which are, for example, intel provides that they have in the intel me, right, they have a software emulated TPM, basically that you can use, or you can use root of trust modules, which are more than just the TPM.
They provide a little bit more functionality around to basically secure your boot process.
And so this vulnerability that you mentioned was in that LinkedIn post, maybe talk a bit about that.
But intel tpms being essentially you could unseal the Intel TPM by reassigning some pins from user space.
So maybe that, I think that was fascinating.
Yeah.
So the concept of sealing and unsealing secrets in tpms is basically.
So we have a bunch of registers that what I explained before.
Right, where you can measure things into.
Right?
So you write hash waves into that, get conkey, they get updated.
And the process of sealing and unsealing is that you can put a secret key into the TPM and seal it against the current values of the pcrs.
These are how these registers are called, right, and they call platform configuration registers.
And so that means you have to imagine that like a lock, right?
And you have like all the numbers, right, basically on them.
So you have like 61160, right?
And that's.
And you see your secret key again.
And to unseal that key again from the TPM, that would only work if you have the same PCR values again and you can unsee it, right?
And this is essentially how bitlocker works from windows, right?
So if you boot into windows, there's Bitlocker, which has like, which encrypts your whole hard drive.
And they measure, firmware measures the boot process, right?
So you always take the chunk of the code, put that into the TPM, right, and they sealed a key prior in the TPM.
And if all the registers are right, so that means your firmware is in the state that you want it to be, and they unseal it, right, and then boot it.
And this, the TPM itself is connected to pins to your, you know, to your SoC.
The SoC has to have a way, you know, there's like LPC or Espy to actually talk to that TPM, right?
And send these measurements over.
And on intersocs, you have a couple of gpios, right?
And they have, they are multifunction gpios.
So these gpios have multiple functionalities.
So they can either be an espy reset line or they can be anything, something else.
And you can basically define that on what you want that to be.
And what that guy found out in the article that I posted there was that majority of the firmwares, or he couldn't find any that are actually locking down the GPIO configuration.
So this is written in firmware, but it needs to be locked down saying, okay, this cannot be changed anymore.
And then this is not done on more or less any of the devices.
So that means you can reset the TPM at any state, and you can basically try to replay the, writing the hashes again into the TPM, try to replay what the firmware actually does, and then kind of unseal what secrets are in the TPM.
And you can do that from the operating system, because intel gives you guidelines on how to lock this configuration down.
But if you haven't done that.
You can do that from the operating system.
And so whatever is in your TPM that's not safe anymore.
And imagine you have access to one device where TPM is basically attached to.
If you can boot into any other live operating system whatsoever, you just reset the TPM and replay the whole thing and get the key out of it.
And then you can boot into your other operating system which is basically encrypted maybe, or for whatever you use the key for.
And then you have it.
Right.
Because normally you store something like a symmetric key.
Yeah.
So that means that screenly we use the tpms for our mtls, that's where we store the private key for all mtls.
So that means that if you're vulnerable to this kind of attack vector, you could extract that private key and you could mimic mtls or not mimic even, you would actually do proper mtls traffic based on that private key and mimic the device.
But I guess it also means that it completely spans encryption technologies, I guess in the sense that it spans encrypted technologies in the sense that it doesn't matter if you're using looks or something on Linux stack, if it's Bitlocker, it's all the same under the hood.
It's just some kind of certificate that you have or privacy that you have stored in there, which makes it kind of terrifying from attack vector.
Right, yeah, exactly.
Right.
And this is someone just handed you over the key.
Right.
It doesn't matter, you know, whatever you have afterwards.
Right.
You have the secret.
Right.
And this is.
Yeah.
Your cipher or how you encrypt, it doesn't really matter at that point.
Yeah.
Right, right.
It really doesn't matter anymore.
Right.
So if someone handed you the keys, you know, to my house.
Right.
The lock doesn't matter anymore.
Yeah, you got the key.
So, and this is why it's, it's for me I think critical.
And what I find even more astonishing is, you know, he explains, you know, in the article, you know, on how it works and you know, what he did to actually, you know, find that out and he, you know, he contacted intel.
Right, saying I found this.
Right.
And I think it's a security issue.
Right.
And inter kind of responded.
Yeah, no, we have in the BIOS writer guide which you have on the NDA, there's a section which says this is how you lock it down if you don't do it.
Right.
That's not our problem anymore.
So there's no CVE for that.
And then they agreed on the disclosure date, right.
And he said, okay, I think, like two months ago or something like that.
He said, yeah, I will disclose things June 1.
And everyone was like, yep, fine, right.
And then.
And then there was no disclosure process.
And this is.
I don't know.
Right.
So I would have hoped for more responsibility on, you know, on an intern side, saying maybe it's not our problem, right.
But we are getting in the people because we see in the wild, no one is doing that.
Right.
So we are going to our vendors saying, this is an issue, guys, can we fix that up?
It basically comes down to sensible defaults at the end of the day, why is this a default configuration when it will have massive security implications for the vast majority of the odms and oems that do not reconfigure these super low level settings?
So I guess that brings me over to one of the projects that you've been working on, the firmware CI project.
So maybe we can speak a bit about that and speak a bit about what it is in the first place, how that fits into trying to solve for problems like this and similar attack vectors in general, and bringing modern tooling into the firmware world, I guess.
Yeah.
So, I mean, nine elements is an ifv, right?
That's how we call independent.
And while we work on a lot of firmware and with a lot of customers, we notice that not a lot of customers actually have a proper CI system for their firmware, because firmware is.
Because firmware is the first code that starts on every device.
So that means there is no abstraction layer between firmware and the hardware, because firmware is the abstraction layer.
But everything that you test on the operating system run on the operating system.
You have an abstraction layer, drivers that abstract the hardware for you and for firmware, that's just not possible.
So that means firmware has to be tested on hardware.
There's no way around it.
So you can test limited things, you know, in QMU and this kind of things, right.
And if you have enough money and people, you can write your own simulator for your hardware that you build.
But if you are a normal company, that's kind of out of scope.
But even if you do, like, you're still building those assumptions.
Assumptions essentially.
So it's still not as valid.
Of course.
Yeah, of course.
I mean, you get certainty, I think, to a very high degree.
Right.
If you build your own simulator.
But, yeah, it's not the real deal.
Right.
Because in the end it has to.
And, yeah, that's why we thought of.
Okay, maybe we can come up with a test framework that makes this easier.
Right.
So certainly there are test frameworks already out there, a lot of out there.
And, you know, some are used for testing firmware, right?
So there's the open BMC test automation.
Tool.
And other things that you can actually test firmware with.
However, my view on this was always, okay, it has to be easy for the end user.
So the end user is the guy or the girl that we're actually working with, and he or she needs to be able to easily set up the whole thing and write tests and things.
So we built a framework that makes this much easier.
And we.
It's a server client model, right.
And the server orchestrates everything and the client is there to interface with the hardware.
Right.
So if you have a new image, you write to it, you get like the serial logs or you get display output or whatever you have from the device under test, and you perform your tests.
And with that, we saw that this is useful for us, nine elements in the first place, because this really helps us scale better.
And we also see that for customers that there was a need actually that they wanted to have that, because reality was the customers that we talked to, a lot of them do manual testing.
We are talking to one company that every, they're not even doing firmware development, but they want to validate the firmware that is running on their devices.
Right?
So that means when they get a new firmware drop from their, you know, OEM, dell, Inovo, whatever, they want to put that on the machines, and then they have like tests that they want to run on that seeing, you know, does this, does everything still make sense?
Right?
Is it configured the way that we want it to be?
Right.
Can I switch the variables that if I turn on secure boot, can I still turn on things on and off, or is that prohibited the way it should be made?
So basically writing unit tests against hardware.
Essentially, yeah.
Right, right.
You could say that.
And that helped them.
For example, that was like a process for each firmware drop, for each hardware.
Five days, one guy kind of sitting there and trying to do all the tests.
You know, I have some extra seats and this kind of stuff and writing it all down.
And they automated the whole thing down to 36 hours, 40 hours, because the test still need to run.
Right.
This takes time, but no one needs to sit there.
And that was like the overall goal that we are aiming for.
And it's starting out pretty good because we see a lot of feedback that this is something that people want and that they help them.
Everything is very user friendly.
I think that's what we're trying to do is here really the tools that have been used for many years already on the higher level stacks, websites development and back end frontend.
If you write an iOS application, I think you have a whole simulator and you run tests all over again.
And this kind of stuff for firmware that's just not there.
And so we try to break that and say, okay, there's an easy way to do that.
We still need hardware for sure, but you can easily get.
Yeah, I think that has been a recurring theme in a lot of the episodes that we've done around hardware is around fact that hardware firmware build development, testing and distribution is really just so far behind modern software development.
Like it's like it's kind of the works on my machine kind of setup for a lot of these.
Like some guy built the firmware on his machine, sent it to some file share and then well that's a release and that's just obviously miles away from how like away from how modern software development is done in general.
Totally.
And I mean, and this is not Joe, right.
Zip fights is still like one of the standards, right?
And this is, I don't know, you are getting, you know, I'm panicking, right.
Sometimes if I see that saying this is like where the world runs on, right?
And with all the cloud stuff that is happening nowadays, right, with all the push to AI now where this, all the whole infrastructure gets, just gets more and bigger, right?
And there's still, you know, two people sitting there.
I don't say that they're not smart people, right?
They're probably good, but still two people sitting there, right, for I principle sending zip files with the dash underline, you know, find or whatever, right?
So, and this is, yeah, and we have to think about that, right?
If we make want to lift that into 2024 and say let's do modern software development.
Yeah.
And I think I remember first exposure that I had to working with firmware or even embedded code base was like modern coding principles like linting your code base.
And these things are like, they are completely foreign concept and like unit testing, they were like completely foreign concept for the traditional firmware developer, right?
And that's kind of like it's terrifying when you do see the real implication of that with things like poke, pixie fail, logo fail, all these problems, right?
Yeah, totally.
I mean to be fair, right, for example, unit tests are hard also in feature sometimes, right?
It really?
Because for unit test you need to also be able to abstract interfaces.
Right.
To any other hardware that you have.
And this is not easily possible in firmware always, right?
It's not always easily possible, right.
So UEFI, in UFi it's better possible because they have these protocols and these handles and can easily hand that in for coreboot, for example.
That's also harder, right.
Because we have a different concept on how we access the hardware.
There are efforts and we have unit testing for parts of the code, but some parts are just not effectively, just not possible.
And then the only way is testing on real hardware.
And then I think you need something that you can easily get going.
Instead of everyone writing their own python or bash scripts again, you try to punch that into your own GitLab runner and see how then someone else does something else again.
And if we can unify on that a little bit, I think that's good.
I mean, we've had the same exact problem at screenly, right?
Like we build a device that's an end to end device that does something on a screen, but like doing that in automated end to end fashion is very difficult because like you basically end up having to like do some kind of HDMI capturing device, do assertion on the image.
It gets very complicated very quickly to do that well at scale.
Right.
Because you are chested across like ten different types of devices.
Well, that gets very difficult very quickly.
Yeah, totally understand that.
Right.
And finger.
It's exactly the same problems.
Right.
And for example, to give you an example, right, when we do this firmware validation, basically.
So what we're doing is we, for example, testing secure boot and we are also enrolling broken certificates.
So we're trying to see if the machine recognized that.
And from time to time we are breaking the machines along.
And then you have, depending on the vendor that you're working with, there's a more or less complicated process which involves copying over text from the screen and punching it into the third party tool and then getting the secret again and punching that into the device again and so on and so forth.
So you need to screen capture, right.
You need to be able to read the text that is actually displayed on the screen.
Right.
So you have serial console capturing probably as well.
Yeah.
And this is, and it's insane how complex this kind of stuff is, actually.
Yeah.
And I of course understand that this is a hard thing to do.
Right.
Think that working with hardware is indeed hard, right?
Yes.
And I think everyone that works in the hardware field kind of agrees with that.
And sometimes I would wish for a much higher level stack.
But yeah, that's what it is.
And we just have to make it more easy for people to use that because I think it all comes down to usability and how low can you actually put the barrier for people to, you know, get into that?
Absolutely.
This is part of the problem.
Why, why really the whole security is in such a crucial state.
Let's say because the barrier is so high and everyone is looking on the budget.
Right.
There's always of course, no budget for testing and especially this extensive testing that you have to do on hardware.
Everything, you know, is cost, if cost intensive to set up and so on and so forth.
That's like a huge pain.
Right.
And we have to really look into that and see how we can solve that.
Absolutely.
So looking back a little bit, what we spoke about a few minutes ago about the push towards security in software and it seems like a lot of these pushes, they've come from the executive order.
It seems like that was like the first like force for hardware vendors to actually, I guess take security serious.
Right.
And it seems to kind of go hand in hand with the whole s bomb requirements as well.
And I think that has kind of shook in the hardware industry significantly.
And I've had quite a few conversations with BIOS vendors and firmware vendors that are really like, they are, it was a rude awakening for them to like really get into like, oh, we actually need to start doing this now.
It's like secure has been brushed under the rug for decades and now it's like these are smart people.
They knew they had a problem, right.
That just was never prioritized.
But now with these new mandates from the us government and you see the same thing in other like follow on legislation in other countries.
But that's, it seems that was the tipping point that was needed for people to start paying attention, I feel.
Yddeh I'm usually not one for over regulation, but security feels like one of these spaces where it was needed to change the status quo to get to where we actually need to go.
Right.
It's almost like where the invisible hand will fail otherwise because there is no market.
Well, there's no demand otherwise.
Otherwise it's a race to the bottom.
Right?
Yeah, but sometimes I think these things have to come basically top to bottom.
Right?
Let's say like that because security is always like why do people do security, right.
I mean, why do you care about security?
Yeah.
Right.
And you don't.
Right.
The majority of the people don't care about it, right.
So it either has to be, you have to be, you know, compliant with something, right.
Very little part of, you know, of the developers or, you know, companies or whatever do security out of their own strong belief that it has to be like that, right.
Most of the people is, okay, we need to be compliant with that, right.
Because someone else is dictating that on us.
Right.
And if we don't, if we're not compliant, we're not able to sell it.
And that's the main driver along, you know, with all of these kind of things.
And yes, it's, you know, it's terrifying to see this exploits, right.
And it's.
It's terrifying to see what, in what state famous security is.
But if there's no mandate, right.
If no one is mandating it from the top saying, okay, if you want to sell something here, right, there has to be, you know, a bottom line that you have to fulfill.
Right.
Otherwise, you know, take your stuff and, you know, go somewhere else that it will not change.
Right.
And that's, that's a fact because everything is running on business and we see that in the US a lot.
Right.
The European Union also, there's like the sovereignty cloud kind of things.
Right.
So we have to own and so on and so forth.
Right.
And they also slowly get to the point that probably starts in firmware.
So I'm also not really a fan of, you know, as you said, over regulating everything and so on and so forth.
But I think sometimes it's necessary to push the companies that are responsible for that basically into the rent.
Yeah.
And I think there is a similar one on the EU side with the EU cybersecurity law because a lot of these things, they come from the Iot world, right?
They are like, oh, your baby camera cannot have a default password.
That's like, yeah, we all know this.
Like, anybody who has any sense of, like, understanding of security understands that.
Yes, of course you don't do this, but unless there is a legal push for this, the cheapest vendor in China will kind of gain market share.
So there needs to be some kind of force in the market to make that happen.
Yeah, but this also only makes sense to us.
Right.
So for us, yeah, of course you don't.
There should be default.
Right.
Or you probably change that or whatever.
Right.
But to 90% of the other people that are just using the baby camera.
All right.
It's easy.
Right.
The password is like 1235.
Right.
That's good.
I can remember that.
Right.
And I can accidentally remotely.
Yeah, right.
Oh, that's great.
Right.
I can see my kid basically when I'm at work.
Right?
Yes, it's good.
But everyone else as well.
Right.
And maybe it has an implication on, they know when you are home, probably break into your home and this kind of stuff.
And I think this, I mean, it's footwork basically.
Right.
So you have to explain that to the people why you're actually doing that, because I think sometimes this feels like you are making things more complicated than they have to be.
Right.
I mean, my mother would say it worked that way before.
Right.
So why do I have to now have like 20 characters.
Right.
And digits and this kind of stuff.
Right.
It's like.
Yeah, until their bank account get compromised and.
Yeah, right, exactly.
They're a real consequence stake.
Right.
And I think that's, and I think that the California law, I'm blanking on the name for the californian Iot provisions that started this whole wave.
And now, I mean, I see that as a positive thing, but I don't want to take too much time on that because there are so many things I want to cover with you as well because you have a lot of interesting things you've been working on.
So I mentioned, you mentioned open source firmware foundation already.
You are my founders for that.
Let's talk about that.
What's the goal?
What are you guys doing?
I think that's a foundation that should get a good shout out.
Yeah.
Thanks.
First of all, as I said, we are working a lot in the open source ecosystem.
That means we are reactive in all the communities, and we are really trying to advance the state of open source firmware.
And what we see is that firmware gets very little love and very little attention from a lot of people.
And so with that, okay, there needs to be one point of contact, one umbrella kind of thing, where we can place everything under, in that sense, where people can go, where people can educate themselves on what is firmware, why open source and so on and so forth, and where they can reach out to the individual projects.
And one thing that I deeply believe in is if we want to drive adoption in open source firmware.
So if you want to have open source firmware running on each and every device, that needs specification, right.
And that needs like.
Yeah, we need to specify interfaces, right.
How do we talk to other software?
How do we talk to hardware and these kind of things?
Because this is not existent right now.
Right.
So we have UFI as one big body, let's say.
And that's just a SPAC, right.
It's per se, not proprietary.
However, majority of the implementations are.
There's one implementation which is EDK, two, which is not and which is, you know, which is not great, let's say.
Right.
So it's okay.
And so I think, and the UFI sPac is rather complicated for a lot of use cases.
And I think we can simplify a lot of things.
And, yeah, with all these kind of things, we said, okay, we have to have a foundation that incorporates all the players that are in the ecosystem.
So the OEMs, the stock vendors, the open source firmware projects, the ibvs, and of course the communities.
Everyone needs to get on board, basically.
And we have to bring everyone together so that we work towards one common goal.
And we're trying to push that, basically.
So we have like working groups that care about, for example, firmware security, where the founder of Binary is also involved.
We have a work stream or working group that is around silicon incentivization.
If you initialize silicon, there are different interfaces, or every software builds their own interfaces on how you do that.
It would be great if you have a.
So that makes it easier for adoption and this kind of thing.
And we're trying to drive that through a nonprofit foundation because I think this specs shouldn't be owned by a company because that's also the standard.
Right now, intel has the FSP, like the firmware support package specification.
So they specify how their FSP looks like and what interface it has.
Naturally, no other SoC vendors will adopt that spec because, you know, AMD will never ever adopt the spec that is coming from intel in its own way.
So we have to find a neutral ground, basically, where we can play and work this kind of things out.
And yeah, the OSFf can be one of this bodies, basically, and it's tied.
To licensing or how do you think about those things?
Obviously, licensing is a hot topic right now.
Source world, how do you view that side of the firmware?
Where is that something that is yet to be kind of emerging in the open source firmware world?
So the foundation itself, we don't care about licensing this kind of thing because that's of course nothing that we want to push.
Our model is basically that the members that are, we have different tiers basically in the foundation.
Right.
And depending on the tier, you can kind of be part of the conversation on, you know, what specifications should be drawn, should be driven and, you know, making decisions on that part.
Right.
So you have influence on the spec, let's say.
However, licensing in general is an interesting topic in the.
For open source firmware as well.
Right?
So we have to think about what's the business model for the companies that develop open source firmware or that have open source firmware on the devices.
And for example there's a polish company, three MDAP, they are running a business model where they pay a fee basically to get the newest core boot version.
So they have downstream fork of that and you have to pay fee basically to get the newest version for your device.
That's one way to go.
Now elements takes another approach saying okay, we are just an agency and we provide support and this kind of stuff, we don't care how many devices you are, but for sure that's an interesting topic in open source firmware.
something where we can all do business.
Right?
I mean naturally again, right.
Even if it's open source, we have to think about how can we do business that is good for everyone, right.
Which is fair for everyone.
Yeah.
And licensing is one of them.
Absolutely, yeah, I think, I mean, and also like, I guess it's worth doing a shout out to the people actually involved in the foundation.
You got some big names in there.
Do you want to speak a bit about who are actually part of the members and who are part of that?
Because you got some big brands engaged in the actual foundation.
So I think it's worth doing a shout out as well.
Yeah, totally.
So one of our co founding members is Siemens for example, which is pretty cool.
Very, very big company and they give us a lot of trust in terms of coming in so early so that we can do what we can do basically, and backing us up.
Because basically when Siemens joined there were only plans, let's say, but their execution was still undergoing and we're still doing things and still moving forward.
Siemens actually provides us a lot of trust there and we have a couple of companies that are actually using open source female in their products and that are actually trying to push for open source.
One of the biggest there for sure is Google, of course, Supermicro, one of.
The big ones are super macro using coreboot on their servers or is that something on the roadmap or do you.
Know, I was curious, they have a proof of concept where they run coreboot on their servers.
Interesting.
They are eventually doing business.
Nice.
Potentially, yeah, they have it on their roadmap, right.
I mean you would be.
I think Supermicro is smart enough at least to, you know, look left and right and see UK, we have, you know, our standard proprietary vendor and there are options.
Right.
And we should, you know, look, what does that, what implications does that have on our business if we go for open source firmware?
Is there other markets that we can kind of tackle?
How does that work from a royalty kind of view?
What does it mean money wise for us and this kind of thing?
So supermarket, I think looking into that for sure.
And I imagine they are starting to get more and more nervous from another one on your roster, which is oxide taking them from the left field in terms of that.
So yeah, maybe that's a.
Oxide is.
I mean oxide is extremely radical in how they do it.
Right.
So I mean, if you ever hear Brian speak about it, right.
Brian, if you are listening to this, I would love to have you on the show, by the way.
Yeah, I got my.
Yeah.
And if you listen to him, right, he's like very radical in a good way, right.
On his views on how things should be handled and why it should be open source and this kind of things.
And actually I saw the racks in person, so I visited them a couple of times in there in the headquarters and they look neat.
So everything is custom made from the boards, from the blends basically in front of the rack, the whole rack, the bars, I don't know, everything's custom made.
And everything looks proper nerd porn, right?
Yeah.
Really?
I mean you see that.
That's great.
I don't need it.
Right.
I mean I will never ever buy such a rag, right.
I have no use case for that, but I want one, right.
Because it just looks great.
And yeah, they are long term supporters on that.
I met Brian like three years ago on the foundation did some mini summit which was in parallel to the OCP, the open project Global summit.
It was like a half day in the back of a conference.
We had peanut and nuts and Brian was giving a talk about Oxide and there's where we met him.
And it's great to see how they actually evolved right now.
And of course a couple of other companies as well.
Right.
Ss three, MDEP, of course.
Is there system 76, is there Nova custom, is there.
So yeah, it's really great to see them board and you know, if some company is listening to this, right.
Said, okay, we are doing open source firmware, right.
And we want to be a part of the foundation.
I want to at least show up on that website.
You can always reach out to Victor and he then reaches out to me.
Right in for happy to do introductions.
The last thing I wanted to cover before we wrap up, because we're running up on time here is, I'm sure this made the news to you as well.
And I'm curious about your take, your hot take on this is insides AI bios because they were being a bit ridiculed in the press.
I'm kind of curious about what your whole take on that is.
Yeah, no, I saw the news.
Wait a, I think at Computex, they announced it that they have AI.
Don't really understand what it is.
How I read it is that they, I think the customization of the whole thing is now through AI, or you can interact with an MI and it customizes the bios for your needs.
I have no clue how that would practically look like.
And I totally think that this is not a good idea because I couldn't think of a thing why you would need AI in firmware.
So I think firmware should do very little.
It's just the basic minimum, exactly what you need.
That should be very strict.
There should be no room for AI's to interpret whatever they want in between.
But rather than I want to do this and exactly this, and yeah, I'm curious to see what they're doing, but my gut feeling really says I'm not entirely sure if that's a good idea.
To me, it sounds like, it feels like taking the current AI hype.
We need to do something with AI, right?
Every, every hardware vendors does something with AI, right?
They, everywhere, they put Nvidia right in building like the new next AI machines.
And this now, and every software stack basically, which is, you know, on top of firmware does AI.
So whether you want or not.
Yeah, if you want it or not.
Exactly.
And so we have to do something as well.
Right?
So let's go.
It sounds like a terrifying attack vector from a security perspective, but yeah, let's see what happens with that.
Totally, right?
I mean, all the models that they're running on is probably proprietary.
No one actually knows what's happening inside.
And I think you can perfectly.
So I talked to a colleague of mine a couple of days ago and he said what an interesting attack vector would be.
If you have an LMMe, which you kind of train to inject exploits in the code that it writes.
Right.
Because you're asking an NI, write me a parser for XYZ, whatever.
And then who is looking at the code, right.
So you copy the code, see if it builds.
it does what it does.
Right.
That's great.
And what if you have elements that are actually injecting exploits there.
Right.
And maybe they are learned and trained on code which had exploits in it and this kind of stuff.
Right.
So, and this kind of, you know, makes up a whole new channel or vector of, you know, possible attacks where you wouldn't even know who injected the code right in the first place.
And.
Yeah, that really, I think that's terrifying.
Yeah.
To see.
I would be surprised if the likes of binary and these guys who have done some great research are not using this for similar like fuzzing attacks and similar, I would imagine, at least.
But yeah, it's definitely, yeah.
When we get into like obfuscation off backdoors and whatnot, using AI, that's where it gets really dangerous and crazy.
So.
Totally.
This has been really great.
Christian, is there anything else you want to do a shout out about before we wrap up?
I think you have a conference coming up in the fall, so maybe that's.
Yeah, that's definitely.
So if you.
The open source firmware conference is coming up in September, 3 to fifth.
It's annual conference surprise around open source firmware and, you know, current developments there.
It's very developer centric.
Right.
So really, if you want to get your hands on the topic, that's the conference where you should be.
If you want to find people who can do it.
Right.
That's the place where you can find these people.
It will be in Germany.
So it will be at the, you know, home at our hometown.
Right.
Where everything.
Where the foundation was born, where the conference was born, when an elements was born.
And yeah, if you want to.
If you want to check that out, you know, have like through our offices and all this kind of stuff, we would be happy to host you.
Osefc IO is the website that you need to check out.
Yeah, it would be great if we can see a couple of more faces there.
Amazing.
I'll make sure to link that in the show descriptions.
And thank you so much for coming to the show, Christian and I'll talk to you soon.
Thanks so much.
Bye.
Cheers.
Bye.
Found an error or typo? File PR against this file or the transcript.