Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Uncovering Firmware Security: A Deep Dive with Binarly's Philipp Deppenwiese
Play On
21 SEP • 2024
1 hour 0 mins
Share:
Today I’m joined by Philipp Deppenwiese from Binarly for a deep technical dive into firmware security. If you’ve been following the security space, you might know Binarly for their impressive work uncovering BIOS vulnerabilities like PixieFail and LogoFail.
We start by exploring attestation - a crucial yet often overlooked aspect of secure computing. Philipp breaks down how it differs from traditional secure boot mechanisms, particularly when it comes to verifying what code is actually running on a system. What I found particularly interesting was his explanation of how TPMs can facilitate attestation, creating an unbreakable chain of trust from boot to runtime.
The conversation gets really interesting when we discuss the practical challenges of implementing secure boot in different environments. Philipp shares his insights on why it can be particularly tricky in Linux compared to Windows, and we explore the entire boot flow from firmware to kernel. His explanation of how certain technologies enable secure computation on cloud stacks was eye-opening.
One of the most valuable parts of our discussion focuses on transparency in modern computing. Philipp points out how many of us “just run things” without questioning the underlying software or firmware, and we discuss why this can be dangerous. We also get into Binarly’s innovative solutions using TPMs and attestation, which I think shows a promising direction for the future of system security.
If you’re interested in firmware security, TPMs, or how modern systems establish trust from the ground up, you’ll find plenty of practical insights here. Philipp brings deep technical knowledge but explains complex concepts in a way that makes them accessible without losing the technical depth.
Transcript
Show/Hide Transcript
Welcome back to another episode of nerding out with Victor.
Today I am joined by a company that I have been keeping my eyes very closely on lately called binary.
And Philipp Diepenweisen is joining me today from binary.
Hey, nice to meet you.
Great.
So the reason why I've been somewhat fascinated and obsessing with over binary is the sheer volume of discoveries that you guys have published.
And I been trading emails with Alex, who's the founder, and yourself as well, about this.
It's just amazing how much you've rocked the whole firmware landscape I guess, over the last few years.
But before we dive into those details, maybe do a quick introduction.
Who's Philip?
What should we know?
And what is also what's finally.
Okay, then, yeah, I will explain a bit about me.
So I'm Philipp Depen Wiese.
I'm basically being in it security industry since around 15 years.
I started to work for the local government.
Like companies here, like government contracted, everything is federated stuff, military, secret agencies.
On some point I moved into a web development agency that was kind of fun because I never have been doing web development and focusing in UI UX thingies.
And then in this company, I founded my own department called Nineelements Cybersecurity.
They're still there nowadays.
I worked there for six years, passed it over like everything I did to my successor, Christian Walter, who was on.
The show a few episodes ago.
Yeah, right.
And he's basically taking that on and it works in great.
It was about open source firmware development, especially as an open source firmware IPV and that, let's say in that time I also focused on creating the open source firmware conference and the open source firmware foundation together with my team there.
And so I'm also the president of the open source Firmware foundation in us.
But yeah, after that time, I basically left for having my own startup.
I did the three ESD scores in terms of like, let's say endpoint protection.
We worked on something called remote attestation.
Can probably talk a bit about that later as well.
That didn't work out.
But yeah, after this three years journey, I did one year development of another important project in the open source industry as freelancer for system transparency.
That was also kind of interesting topic.
And after that I landed somewhere, somehow.
It's binary here.
And now I'm the head of solution engineering at Binary.
That's.
You have so many topics I would like to dive into that just briefly mentioned there because there are quite a few of those interesting things I would like to cover.
But maybe before we go in that give me a bit of backstory on binanly and just what is binary, how?
Well, what's the purpose?
And just without too much vendor pitch, but just give me some big backstory.
Really.
Yeah, so I tried to do so.
I mean, I met Alex, was it like four years ago?
And basically finally came up of his mind.
So he was a visionary.
He was coming from Nvidia.
He was one of the security researchers, the lead security researchers there basically.
And he has a huge background in security research.
He also wrote a book called Bootkats and Rootkits.
He even bought it like 15 years ago or something like that.
So he's working on the security research stuff for quite a while.
And he has also been an expert on reverse engineering and all this like lower level vulnerability and all this stuff.
And I think, like he basically, what he told me is like he basically wanted to look into this area and then he started basically binary.
And the focus of binary was basically looking into binaries.
So that's why it's also called binary.
It's like kind of a nice play, you know, don't know if it's called binary or binary anyway, which is kind of strange from the name, but it's is good people think about it.
And interesting part is what we do different from a lot of other companies is we look into the binary level, so we care more about reverse engineering.
And we use this reverse engineering technology not to uncover secrets in the binaries.
We just more look into the binaries to figure out potential risk that includes, for example, known unknown vulnerabilities, malicious code, for example, if you embedded backdoor into a binary or whatever, and cryptographic materials and stuff like that.
So that's where binary is basically focusing on.
And that's quite unique in the industry because most of these companies just look on the source code level and we don't require them.
Yeah, and I mean, I've had a few episodes around firmware and bias and the core boot guys, for instance.
We talked a lot on that episode.
We talked a lot about what goes into bios, for instance.
And I think that's, I mean, that's obviously a sweet spot for you guys to do version during our bios.
And I guess for people who have never done any of this, it's kind of a, it's a strange world, I guess.
And it's a world where modern supply chain security is not a thing.
And I've been covered that multiple times on the episode.
And it's scary to someone how little transparency there is in that world, right?
I guess Binarly's claim to fame and really big breakthrough was the discovery of logo fail.
And that was probably the first major thing to hit the mainstream media where people like holy shit.
And that definitely was my reaction when I first read it and I was like wow.
So maybe talk me through how this was discovered and maybe like for people nothing familiar with it, the impact for the world because it was, it's, the blast radius is enormous for wow.
We'll talk about other ones but just start with that one.
Like talk about what logo fail and how was this discovered?
Yeah so let's talk a bit about that.
So basically I think that was around two and a half years ago.
I like we working together with Alex in a company partnership and I already talked to Alex about issues in the field we have seen.
And one of the things where we had to manually analyze some firmware I think it was from, let me think what kind of company was a german company, Fujitsu, Siemens?
no I remember or no Japanese, I don't know, they're like japanese German, whatever company.
And the interesting part is like we already found out there's some issues with unsigned let's say boot logos, right?
It's about logo fail is about the BIoS boot logo.
So if you want to upload your own boot logo into the firma it's often possible because people love customization companies as well and so they make opportunities that you can change the logo during boot.
So it's not showing Lenovo or whatever, it's showing their custom product line whatever for example Palo alto networks if they have that and then they show that in this bootloader and you can customize it.
And during the time we already figured out these images are not signed because you want to exchange them maybe from the operating system side of you.
And this was basically I think also some impression for Alex because I told him like this seems kind of sketchy right?
There's no validation for that, you can just change it.
And they said let me look into that.
I think during the time, that was around the time two and a half years ago they already like put it on the to do list and on some points they came to the like basically to this logo fail and look to like looked into all these type of lobe images you can basically upload with different kind of parsing libraries.
So there are a lot of let's say so you can check out the Tiana core, it's an open source implementation of UEFI.
Basically it's on GitHub, you can just check it out and you can also just check out basically the image parser, right?
And there are different types of image types nowadays.
You have JPeg, you have Png to have BMP or whatsoever.
And I think it was JPeG mostly and BMP what was used.
And they already found like vulnerabilities in that by just looking at the code, right.
And then you can do further analysis, basically looking into closed source implementation.
Try to use pattern matching to figure out the same like vulnerabilities in these kind of, let's say firmware images.
And that was possible because bindly made basically this kind of technologies in the backend, and they were able to basically prove that also in production images, they had vulnerabilities for these different type of images.
So I could pull up some kind of presentation.
But I think this is like, this is just to go into deep.
I mean, there's a lot of materials on the Internet nowadays.
We also released, I think in the blog post, extensive review about all the stuff.
But basically it ended up after this type of research and responsible disclosure that there were a lot of system where you could easily just like make a malicious BMP image or JPEG image, just put it there, and then the BIOS took it, passed it, run in a buffer overflow, and basically you had code execution on the firmware.
And I mean, most people say, why is it important?
Right?
But the firmware has the most permissions inside the operating system, even before the operating system.
But even during runtime of the operating system, there's also something called system management mode.
So if you can sneak in before the operating system starts with some kind of exploit into the firmware, you can basically burn in an implant inside the UFI firmware image, and you're able to even attack the operating system site through system management mode, which is basically runtime service into the operating system.
There also have been some examples of that, like theoretical ones, and some are really in the field, but more like for the, let's say, intelligence people, where they have really advanced persistent threats there.
But let's say this already has been a huge problem because the firmware level is really not seen from the operating system part of it.
So I have two questions about it.
The first one is, like you mentioned, that you have some tools for discovering and analyzing.
You kind of already answered this a bit about discovering this part.
It was more like a gut feeling than anything, by analyzing the code.
But that then revealed that what are the weaknesses of the open source or sorry, the closed source bias world, which is.
Well, you don't know what goes into your bias at all.
Like you have no clue, right?
It's just, well here's a binary blob.
Have fun.
You may or may not even have a sember on that.
You probably don't even like.
You might.
But it's definitely not a whole lot of insight that you can actually pull from that.
And this is where it got really interesting why I think what you guys tooling, and I kind of covered that in previous episode as well.
It's like how do I even find a blast radius of this?
I know some bias are affected, but there is so little, there are no s bombs you can pull.
You can correlate this between all the vendors and there is a lot of reusable code between vendors.
So maybe speak a bit about that.
Like the tooling you guys have in place for like checking this out and.
Yeah, maybe start from there.
Yeah, so I mean this tooling, like we written a lot of tooling in binanly, but also in the already open source world there's some tooling for looking at UFI images for sure you can look at the open source implementation tano core and the source code level.
But there's for example another tool called UEFI tool which is not from us, it's from some engineer like of Apple who basically maintains that and basically wrote together tooling for looking at UEFI images.
And they also add some security checks and you could see what kind of parts, for example of the boot flow, unsigned or not.
So that's where I basically got like whoa, why it's not signed.
Right.
You can just use some kind of open source tool, look at it and then figure out, oh, this is not good, how it looks like.
Right.
That's one option.
We released FW Hunt which was basically the first version of it, which is basically a world based firmware scanner.
So it tries to find through pattern matching and some technology like the vulnerabilities in different kind of firmware image.
You can just scan any firm image, try to match this type of patterns.
We have a rule description.
So this is open source.
It was basically the first version of it.
Now we have the second version probably you did hear about it or not, I'm not sure.
So we released it during Blackhat this year is basically RISC Binary IO, which is now a full service.
So you don't have this kind of menu invoking tooling anymore.
You can just use a rest API to find known vulnerabilities in the firmware space in generating sboms out of it.
That's completely free.
You even can use it anonymized.
We don't care so much about registration and things just for API access, so you only have access to what you upload.
And this service should basically help also the community.
So we're currently working together with Richard Hughes from the lvfs.
So basically from Red Hat, he's going to integrate as part of lvfs.
So all images of lvs will be scanned by this kind of service.
And there are also other companies like blind spot software from Christian Wilder, who has been on this video, like on this, let's say, episode as well, and.
Richard Hughes as well.
Actually, he's also been on the show.
Yeah, yeah.
So, and then there's OCP, for example, open compute project, which are trying to define like open data center standards, and they probably make use of it as well.
So we want to spread it a bit more.
And the important part of it is we will also add a lot of experimental features from our, basically new research into that, so that we can like see if, what kind of benefit it brings to the masses.
And also people who want to use it for free, just try to do that and you can figure out what's inside your finger images.
And especially, as you say, generating s bombs is a big problem, right?
Because just generating it or believing in S bomb, it's a text file, bom.
It's from the hardware point of view, right, bill of materials, and then it's just a document about what kind of components are included in your system.
And the same goes for software, for sbom.
But what you need to do is basically you need to prove if the S bom is matching basically the contents of the binary and this binary, we decided not to go that way anymore to say from source code to SBOM, I don't think this is good, because you have a lot of, let's say, third party software and even binary components going into there.
So the best way to do that is basically generating it out of the binary in the end and generating an SBOM.
So you can basically prove what's in there and looking, by looking into it.
And you can, I guess, based on the hashes, when expanding the binary, I guess you can find, you could detect based on the hash, the version of a component, basically.
Because I guess you have some kind of library or index for that, right?
Yeah, I mean, there is some kind of limitation.
You need to know the most important part nowadays is to be honest, to build software reproducible.
The main issues with it is that like we produce the build.
The problem is the biggest part of the industry is not doing it.
So comparing hashes is somewhat useful.
If it's a release and this is not customized or patched software, then you can somehow compare that to basically to the public releases, right?
That works.
But for this type of like, let's say customize modification and stuff like that, it's really hard to really validate.
If it's really the binary you're searching for, sure you can check the hash, but as I said, limited.
So we also have to look into other parameters and looking at towards a binary level and checking for specific similarities there, especially for patterns in order to figure out if it's really the case.
You can for sure just check the version number and stuff like that.
But someone can fake it, right?
So it's lots of no, right?
Yeah, I wouldn't expect the version to be super helpful in this context.
So I guess the obvious question is, well, not obvious, but my obvious question is like how was this received by the industry?
Right?
For my feeling, it was not exactly welcomed by the buyers vendors.
So how did that look like the conversation between, well, when you did the responsible disclosure and what was the response from the vendor BIOS vendor community?
So let's say I've worked for a long time with this IPV folks, right?
I worked a long time with this kind of IBVs, ODMs, OEMs in my previous field at nine elements, cybersecurity.
Now I do the same again, binary.
The problem with that is responsible disclosure is quite important, right?
And everyone agrees on that in the theoretical world.
Really great, because you get two months of time, you can fix the bugs, everything goes out, everything is safe and sane.
The end user gets a fix before basically the disclosure is done and everything is published.
But the problem is with, let's say, hardware folks or companies, hardware companies, they are not really used to that type of ecosystem.
They're not even understanding software.
So once I give you an example, once I ask it, odms or oems like that, they also ship software, right?
I said like to them, you ship hardware, right?
Yeah, we do chip hardware.
But you also ship software as part of the hardware, right?
No, you only ship hardware.
And then I say it.
But wait a moment, there's like probably 32 megabytes of Biosim there, and additionally components.
Sometimes they have even a baseball management controller, which is like this kind of control unit for server systems with 128 megabyte of Linux system on it, right?
And said like yeah but this is software right?
No, it's hardware the management doesn't understand especially asian like odms and oems we're not really let's say understanding and welcoming in this area because they don't understand what it means they also don't understand not all of them for sure there are better examples like framework for example really good company so my laptop is from framework but also like they already like they have more, they have better people to understand how it goes but for all these other odms really hard to understand it and even this is not the only problem so most people don't know but probably I can talk about a bit more so the people who are writing software in this area are mostly third party suppliers so that means if you are an ODM you don't you maybe you develop a part of your firmware yourself or customize it but most of the stuff comes from another company and you just buy it in and these companies are for sure Ami phoenix insight there might be also open source ipvs in there like nine elements they are better than them these closed source companies let's say and the problem with that is you don't understand what your tech stack does because you're basically just buying stuff in and then you have to deal with them and aside from that the software quality is not so good in this area because firmware engineers haven't been at conference for last 30 years.
So I mean I started the open source firmware conference because of the reason there were no firmware conference.
You can go to some kind of trade fair and you can also find small conferences about Firbair but most of these events are not really focused on the developers and.
Right and so firmware conference was one idea to basically get them all together and this is the main issue like those people normally sitting in their let's say area or ecosystem circle and they don't go out and they don't see what else is there and this is a big problem.
So it's not like if you look at web developers or if you look at application developers where they go to big conferences and then talk and exchange about different kind of frameworks and topics it's not like that.
And all these basically play into the, into this ecosystem where it's really, it's bad kind of understanding about security vulnerabilities and even software from the hardware vendors perspective and also these people who develop firmware are not really skilled to write good firmware and also don't understand security at all.
So all of this combines into basically one result that the firmware you get there is mostly really insecure.
It's not really well developed.
Best example is, for example, a friend of mine.
This is another story, not even from the security perspective, but functional perspective.
Normally if you update your bios, basically the program tells you not to shut down the device, right.
They tell you don't shut down, don't do anything at all, just keep it on the powered on and have a power supply to it.
Because if it turns off, everything breaks.
Right?
Yeah.
And this is like super funny.
And then if you are Finbi engineer like me, and you know what to do, and you already have seen like products like Chromebooks for example, use open source firmware core boot.
They developed a model where they can never break, they can never have a break of the system.
You can shut it down all the time.
The update will not kill the, like the system basically, and there's no need to worry about that stuff.
But in this world it is.
And I even they have some kind of recovery machine and most of the time it fails.
So there you can see how really bad software development and knowledge in this firmware world is not about the hardware.
They have amazing knowledge about hardware and how to do specific things, right.
In terms of connecting firmware to hardware, but they are not good in developing software at all.
But anyway, this is just a long, let's say long discussion about that.
And I hope I answered the question there.
Yeah, yeah.
So I think, yeah, it's a cultural problem to a great extent, right?
And I think so, long story short, the disclosures were not very happily received.
And then I guess what is the state of affairs currently?
Because that was announced what, eight months ago, I think, or something like that to the public or ten months ago, probably by its public.
The UEFI forum tells you one year.
So if you look at the Tiana core, like the UEFI is the specification, right.
If you look at their forum websites, they tell you they need one year to do response, like they need one year to close the bug.
It's insane, right?
Because like one year is unbelievable amount of time.
I mean, how long does it need to, until the fixes end up at the customer side, right?
And you are right, they don't take this well.
They don't like security vulnerabilities.
They hate them basically.
And so this is, it's a big problem to do.
Responsibilities, closure.
There it's so big that on sometimes for the vendors we not going to decide to do that anymore at all.
We give them a period for sure and say like you have time, but it's impossible to just do that.
It costs you tons of money on your side because you need to have, they want to have meetings, they want to have information and stuff like that.
It costs you a lot of money to actually do responsibilities.
I mean security researchers from university for example, they do it for free because that's their part of their research or whatsoever, you know.
But if you're doing it as company.
Like, yeah, different story.
So I guess the winning strategy going forward for you guys, there will be a name and shame kind of approach.
It sounds like that.
I wouldn't say like we would, we don't want to change them basically publicly on it, on a wall of shame, let's say.
But there will be definitely people showing like metrics and showing information about, let's say hardware vendor names which are better or let's say better in terms of security, which are faster, close security bugs and stuff like that.
If these metrics comes out then I mean this is just research from our side and also from other sites as well.
It's not only us.
And then you can see what kind of hardware you should buy from what kind of render, right?
Yeah.
And I think the change cannot be introduced by publicly shaming them.
It would be super nice to be honest.
But the thing is like it's an ethical question.
And also, I mean the best thing is if the customer of these hardware vendors, if they have transparency into their basically firmware, they get from their customer, from the hardware vendor, they can see how good it is and they can see next time if it's bad, for example, or it gets worse or whatever, they can say, oh yeah, then I get a 20% discount, right.
And then like things end up not being so nice on the hardware vendor side anymore.
I don't believe, personally, I don't believe you can force hardware vendors without business incentives.
So what you need to do is to have all these enterprise companies basically giving them transparency to show like also they can save a lot of money, but they can also basically understand what kind of hardware is running.
Imagine for example, you build a product, you are parallel to networks, right.
And this is just an example, right.
But they're basically building like switches, firewalls whatsoever.
Right.
And then you buy hardware or you get hardware from your ODM, from Taiwan, for example.
And the firmware shitty.
But you don't have any tool to look into this firmware stuff, right.
You don't have any transparency you have talked about before.
Then somehow you need to figure out, yeah.
What's in there.
Right.
Because it's a risk that it ends up as a product, let's say security vulnerability on your side.
It's not like the ODM gets blamed for that.
It's your product.
And so you get blamed and you get basically the bad press about it.
And so this will change hopefully in the future when we give those companies the tools for it.
Do you see the, so the EU has been pretty proactive about legislating, well, I guess trying to fix security in a very, european way, in which some of it is good, some of it is not.
But do you think there is a.
I'm actually rather bullish on that.
The only way security will be improved is probably through legal means, because I don't think there's enough business incentive to actually solve, properly solve security.
That's my, particularly in the Iot world and the embedded world, I think it's gonna be hard to solve it without legal tools to force vendors, like you said, like to force them to actually care.
What are your thoughts around that?
Yeah, so I mean, there's one option, basically, nice phrase like that.
But let's say there's one thing you can do.
You can for sure give the, like the enterprises the opportunity to look into this binaries and, or black boxes.
And when they figure out vulnerabilities, they can basically like reduce the amount of money they need to pay to these odms.
And this is bad thing for odms.
So money is the driving thing in the ODM market.
It's one option.
The other thing is compliance for sure.
So, I mean compliance for regulations.
So you cannot, for example, buy from hardware vendors who don't comply with specific regulations.
Right.
This is another driver, I would say, and this one is, I think, only, let's say it's only helpful if it either enforces you not to buy hardware stuff there or there are hefty penalties.
And I think with the latest regulations from european unions, the Cyber Resilience act, which come into place, I think, what was it?
2025, mid of 2025.
I don't know what it's coming to, into play, actually, but it's, yeah, but it's a pretty comprehensive piece of framework.
Let's say until summer 2026, every country needs to have it in place or a bit later.
And the thing is with that, it's already making a lot of infrastructure, critical infrastructure.
That's the first thing what this thing does.
And then critical infrastructure has regulations like they need to fix security vulnerabilities.
That's for example important.
And if we can detect vulnerabilities in the third party components or in this like black boxes because we don't need the source code because they don't like they try to obfuscate all the stuff.
But if we are able to show that, then they need to fix these bugs.
And this will also drive basically or force them through compliance, I think.
And also s bomb is for example, an important role, I think, for the Cyber Resilience act.
And this will come as well.
So it will just take a while until adoption.
But it's already like making this OEM hardware business much more difficult because years of like not doing anything at all and shipping just stuff and then you are trustworthy because the company have a contract with you since ten years.
That doesn't make sense, right?
I mean, if you go to the supermarket and you buy seafood, right?
You go to supermarket, look at the seafood and then you probably smell or look at the eyes of the fish to figure out.
And it's like basically the, like, you know what I mean?
Like the english word the gills.
Yeah, yeah.
For example, to see if it's fresh fish, right?
And this is what you do as normal person, like in order to figure out if it will poison you, right?
And if you eat it, food poisoning, basically the same goes with like all this hardware render stuff.
We cannot just trust the hardware ecosystem blindly and just hope that's secure.
I mean, how do you do that when most of the stuff is coming from China, right?
I mean, well that's, that was.
Kind of my initial reaction as well is basically like, well, when we, when I got into the hardware world and started building hardware and it's just like, I have no idea what's actually inside of this.
There could be backdoors.
There could be like, I mean, I get a bias.
Binary.
Cool.
What's in there?
I can't build it.
Certainly not reproducible build.
Like I, I can't do anything with it, right.
You can't even unlike you have, unless you have tools like you guys have to actually extract that and actually figure out what's in it, right?
It's.
And it certainly is a world of black boxes.
I think these kind of toolings will force a change in that ecosystem hopefully because yeah, as long as it's treated as a black box and nobody can actually look into it and see what's in there, like nothing's going to change.
And that is a good strike.
Yeah and this is a problem, I mean a big problem.
And I tell you like for example if you don't have tooling this basically is, you cannot do it, right.
But now we got tooling, it's basically possible to look at it.
And then people also start to think about the, I mean they can, there are multiple reasons why you want to do that.
But let's say there's also a big example.
We worked once with meta like for data center, for the data center area and did there for them some kind of security stuff.
And the interesting part was they were already looking on the hardware level, not even firmware like that.
They already looked at firmware as well, but they also looked at their hardware level in order to find basically backdoors inside hardware.
Just imagine that you take a full server like you get the server shipped right on the ship in a cargo and then it comes to your country and then you put it on some kind of transportation unit, whatever is it arrives at the data center and then the first thing you do you pull out like one or two servers and then put them in some kind of screening like you know basically x raying the entire chips and the PCB's and everything.
So the motherboards and looking, if there's any type of let's say component which is probably malicious which shouldn't be there.
Right.
And this is super crazy and this is already have been done for these cloud service providers basically.
Right.
So like Microsoft Azure, Amazon with AWS and stuff like that, they already look at that kind of basically impact from the supply chain.
Yeah that was, I'm not sure you remember that story about super micro servers being backdoored, right?
A few years ago, I'm not sure that was a, I'm not sure that got stored, got pulled in the end if it was actually bogus or not.
But I remember a conference talk about that a few years ago.
Right.
I mean it was probably bogus.
I mean if you want to go into super micro hardware nowadays I would personally say I did some research on my null elements side during the time all the systems are unprotected.
You can just put a flasher on their firmware chip, refresh the firmware with something else no one figures out.
You can just like you don't even need to attack the hardware level, you can just do it on the software level by just flipping something on the hardware chip.
So I mean this is like completely insane, right?
And so in order to improve this ecosystem and also tell people it's really hard because most people are not that technical, right?
And you need to explain them, not even the vendors, whatever, even like the enterprise, like customers who buy that hardware, they are not really well trained on this kind of thread, right?
And they say, yeah, why is it important now?
It hasn't been a problem since the last 30 years, right?
So most people will tell you that, but the reality is like the firmware grows.
I talked about that a lot in my time of being at nine elements that the firmware, for example, in 1999 you probably had like, I don't know, 100 kilobyte of firmware components on your system.
In 2000 you had probably 1 firmware components in your system.
In 2010 it has been around 60 megabyte of firmware components in the system.
And nowadays we are around 32 to 64 megabytes of a normal laptop.
I'm talking about, yeah, I ordered a new motherboard a few months ago and I was like 32 megs for bias.
That's crazy.
You know, you can put a complete Linux kernel inside there.
We did this with Linux boot just for data center applications.
We run Linux inside the firmware.
And then it just costs you eight megabytes.
And I mean, people can imagine what kind of impact it has.
And also the problem is like in the next ten years, I guarantee you we will end up to 1gb on the firmware level.
And the reason for that is all those chips getting smaller, get more features integrated, their bus systems like all the buses, like high speed memory, high speed transfer for USBC, whatever kind of new technology arrives after that.
Everything which needs high speed requires a huge amount of source code to control and also to manage.
And that means like we will get more and more chips with more and more firmware, which is more like Linux operating systems or real time operating systems.
And it becomes less, not only one chip, it becomes ten on the main board already.
And so this is super insane.
It means the firmware growth is massive and in the next ten years it will grow more.
And on some point we just have all these integrated chips communicating with a lot of operating systems.
Yeah, that brings to our topic that you can hint at before, which is, I mean, you said if you buy a super microbox, just swap the firmware on.
And that brings me to attestation, right, and source of truth and validation.
And that's an entire cattle fish, right?
I mean maybe before we dive in that because PK fail, which is one, was one of the latest disclosures from you guys, kind of goes in that territory.
So talk me through PK fail and then I want to segue to have to stash at large.
Yeah, so let's say PKFA was basically the thing that we found in test key.
So there are multiple test keys by the way, not only one, I think it was 22 or something like that.
You can find that on our like outline on the blog post with some materials in the PDF, all the information.
So let's say the interesting part is that this test key have been used by a lot of companies, especially oems.
So for example gigabyte was one of them and supermic and some other, not all of them used it, but like most of them use this test key.
And normally those test keys shouldn't be used.
But the first thing is the test like keys, key materials in that ecosystem should be protected by hsms.
That's the first mistake, right?
They didn't do that.
Most of them just stored on the hard drive unencrypted and then just built from some kind of build PC or whatever.
Build this kind of, let's say, image.
The problem is also a lot of these, you can check that.
If you just check out film images downloaded from the homepage of the vendor, you just run strings command on the Linux, you will figure out the build directories are mostly Windows operating system.
Build directories Terrence, many reasons.
Yeah, the reason is like this ODM, or let's say hardware business is highly driven by Microsoft operating system.
And so it's just how it is.
They are not Linux people.
And this is the main reason why all this stuff is also not well integrated.
You cannot do good pipelining stuff under Microsoft Windows operating system.
There's no, this is why basically Microsoft Windows integrated Windows for Linux, right?
So basically like the Windows WSL environment for like working under Linux with Microsoft Windows, right?
And this is one reason.
And so this is a big problem.
And so they don't have this kind of OpSeC, they don't have this kind of automation.
And then they end up building manually releases and then ship them over some kind of distribution channel.
So that's the first thing, which is the big problem with these keys.
And the second thing is those people are not really skilled on security at all to understand what they embed are test keys and they shouldn't be used.
And the biggest problem was the test key was reused multiple times.
So it's not only like you generate, every time they got an SDK from AMI was a test key, basically they don't generate a new one.
They just ship the same test key for all devices, like for all SDK portions of.
And then you have like probably you have multiple different kind of, on some part you have different kind of SDK keys, but like 22 for example.
But with one you can already like own, let's say three form vendors and like 200 product lines or something like.
That's super crazy.
And this key basically, so for people who don't know what it is, it's basically the PK key, which is the highest key in the secure boot hierarchy.
That means there's UFI secure boot.
Normally this is used to protect your bootloader.
And there's kind of key hierarchy for basically doing security stuff is super complex.
So you have the highest level keys.
A PK, this was basically owned.
Then there can be multiple kecks, basically keys.
And then you have a whitelist database.
And a blacklist database is for blacklisting, for example hashes of bootloaders or certificates.
We have been compromised.
And you also have a whitelist basically allowing things.
And if you have this PK, you can change everything.
You can just generate a new keg under it.
You can do whatever you want.
You have the full basically options to do.
You can block people supplying bootloaders.
For example, you could say the Microsoft bootloader is forbidden, right?
So you can also not fix it.
And only my malicious bootloader, who looks like a Microsoft bootloader works then, right.
Kind of strange.
And then this has already been broken with that.
And so the important point here is you can just do that from the operating system side.
As soon you have root access to system and the system has this kind of vulnerability.
You can just get this type of private key from GitHub.
So it was leaked on GitHub on other probably multiple places.
So since it was used a lot of like by a lot of companies for sure, there have been leaks of these SDKs and you could just download this key and use it.
And then you can basically just sign another CAC integrated into the system.
The BIOS accept it and you have full, basically full features to do whatever you want on the system in terms of moot process security.
But there was a caveat, I think they were all like encrypted, was they were encrypted with like a four numerical, four character numerical key.
It was protected by basically four, I think numbers or I don't remember what it was, but it was super easy to break.
But you don't even need that.
So I already saw this SDKs, I mean, I saw the SDKs already out there with a test key unencrypted.
So it's not like you need to really decrypt it.
Right.
So this is like a brute force it, because most of these vendors just put stuff everywhere.
And this is the main problem, I told you first is how do you handle keys, especially even if they are test keys, right?
Yeah, yeah, that's the same.
So that brings us to attestation.
Right?
So because this is a complicated topic, attestation is not easy in any way.
You deploy it, right.
And it becomes, it's complicated in the cloud, but it's a lot more complicated when you're dealing with devices that you.
Well, that might be a different country that when they were initially doing their initial bootstrapping, you don't have control over it.
So, yeah, I mean, you have a background this, like what?
Yeah, maybe like do a.
Explain to me, like I'm five, what is attestation?
How does that work in the world of firmware?
Yeah.
So it's not only word of firmware, you can pull it over the entire boot like boot up of a system, let's say.
So, but attestation is basically the idea that you can do local attestation and or remote attestation.
And local attestation means you can locally prove something and remote attestation, you can remotely prove something about the system.
Some assumption, let's say, and this assumption normally, like in the most practical way seen, it's what kind of code has been executed or data have been executed on the platform, because you can also read data from the code, pass it and execute to another stage.
And the interesting part is nowadays attestation is super easy to do thanks to the improvements of the TPM.
So there's a trusted platform module, for example, on the most devices where attestation is possible to do with, there's no confidential computing.
You probably heard the name of it.
It's like some kind of technology to run something in the confidential vm.
So nothing goes in, nothing goes out, and you can do computation on the cloud stack securely.
This uses attestation as well.
And for these use cases, basically it's like if you use this with a TPM, for example, you can check up your entire, let's say, what have been executed in the boot flow from the firmware level, from the BIOS level down to the operating system kernel.
And this is one of the most important technologies upcoming in the next years.
Nowadays it's not really often used, and this was also when we had started our startup that we try to have this kind of like technology back there and have it as part of the back end.
But it was like the most people don't understand how valuable it is, because nowadays, I give you an example, people just run things, right?
This is similar with firmware, they just run it, they don't look into it, they have no tools for transparency.
The same goes with when you're running operating system, if you're running Linux in some kind of data center, how do you know no one exchange the code on the Linux system, right?
How do you do that?
Yeah, secure boot is still painful in the Linux world, right?
It's somewhat better in the Windows world, but it's painful in the Linux world even to this day, except for maybe Ubuntu, but beyond red hat.
But beyond that it's painful.
That's completely true.
But also keep in mind, secure Boot just tells you this one has been signed, but you don't know if it's a previous version or it's the newest version, right?
You don't know what kind of version has been executed, you just guess.
Like someone could for example, have an exploit for a specific version which has been signed as a part of secure boot.
And you want to forbid that, but this is hard to do.
Then you have to update, as you say it, like you have to update the blacklist to probably forbid this kind of piece to be loaded inside in this secure boot chain.
But it's also complicated because you cannot control it.
The full chain like secure boot is only the transition to the first bootloader, and then the other secure boot like, or similar secure boot or signature verification parts are done by the Linux world.
For example, if you use Linux on the system, and then you have to, like, it's really complicated to do that.
There's also like kind of shim bootloader who does it, and it's really, we looked into that a lot, and so I believe also it's super enforcing.
If you have systems which have high ability or need to be available, and someone wants really to make your life hard, they just like load something which is unsigned and then the system stops working, right?
With attestations.
This is not the case, the system always works, but you can prove if the system is in the state you expect or not.
And this has been, this goes in a different way.
We don't use signatures for that.
We use hashes.
So every part is hash before it's been executed.
And then the next part, before it executes the next thing is also going to hash it and puts all these hashes into the TPM.
And the TPM can only reset this hashes to the original value by basically restart of the system.
That means you have this kind, you collect just an append only lock with all these hashes get xored and basically combined together, so you cannot basically calculate them back.
And so you have basically a list of hashes which giving you the state of the system.
And this is attestation is used for.
And this is super useful for basically securing systems and integrity protecting them.
But most of these companies nowadays don't use it.
I know some companies make use of it, like Cisco use it in some areas.
There's other companies like in Germany have been using stuff like that.
And for sure, like people start to look more into attestation nowadays, especially with confidential computing, they're doing the same on that level.
Just is a VM.
They want to prove what's running in there, right?
Because they want to make sure nothing like, nothing wrong has been executed before you ship your credentials to this VM, whatever you do there, right?
And this is important part of it.
And so it also got lately integrated in systemd.
So if you want nowadays to encrypt your hard drive, for example, under Linux, and you use cryptid app and you want to do local attestation, for example, to bind all of this hard drive encryption to the firmware, to the bootloader and every component you utilize until basically the system, the init ramfs, spins up for decryption and asks you for the password.
You can do that.
It's now integrated.
Leonard did some great work there in that area.
Interesting.
So if you wanted to try it out by yourself, I think it's still a bit experiment material, but it's going to be integrated in a lot of public distributions like Ubuntu is going to make use of that fedora.
So things are changing lately, but it took a while to adopt remote station localization.
Yeah, just a side rant on that.
I'm still, I find it oddly painful still that you can't use proper, like TPM backed disk encryption on most distros without like bending over backwards five times.
Yeah, I mean, it's still not easy.
I mean, remote itself, attestation is less impactful because it doesn't that you can integrate in theory if you do having company and you're doing some kind of nice product and you just want to secure your product.
You can use attestation quite well because it's remote one is easier than the local one.
Local one is enforcing through encryption, remote one is just verifying it.
So I think maybe there's also remote installation already on the roadmap of system D.
So maybe there will be an easier way to utilize that.
Not by using this encryption or like combining it with it, I'm not sure yet, but the future will definitely look more into this integrity protected launch of operating systems.
But what you need for that is reproducibles.
So most of the software companies should be built reproducible and there are reasons for that.
If you later want to compare really what you have been executed, because otherwise you have to deal with so many different, let's say artifacts.
You need to compare and see because it can be built differently and it doesn't work across all things, right.
If the build server is doing things wrong and stuff like that, then you might end up locking up your system and things like that.
So yeah, I mean that's whole different topic.
That is a very complicated topic at large to cover as well because secure builds and I mean GitHub is doing some amazing work around that to make that actually accessible for a lot of people.
But doing sign builds and yeah, cryptographically sound is not a trivial feat to do.
Let's talk a bit more about other interesting findings that you guys have been revealing.
One of them is the lighty or lighthdpd vulnerabilities.
Do you want to talk a bit about that finding?
Because I thought that was the first time I've heard of Lite in a long time, but well, clearly still in use and has some serious impact.
I would love to, but unfortunately this one, I didn't look into it.
I mean I just started five weeks ago.
So to be honest, that one is.
That one is still like unknown.
I just checked up the recently ones, but I think from this, let's say the next episode, should Alex ask more about the details there because he's quite specialized on that.
I'll make sure to dive in that.
So then one more thing I would like to cover is I'm not sure you caught this, but apparently the root certificate for secure boot, probably not familiar with it.
Microsoft are the one issuing like the root certificates for secure boot across any operating system.
Right.
And I'm not sure you caught this, but apparently the root keys that are signing off secure boot expires in like 2028 and that's gonna have an interesting bombshell for the entire world.
Really.
Like have you, did you catch that one at all or is this news.
I catch it already.
So the thing is like you need to know, in the firmware world there's no time.
This is the first thing you need to understand if you are in the UI five firmware world or whatever.
Most of these like embedded firmwares have no time.
And sure you can spin up network and I mean this is dangerous, right?
If you have this kind of super like highly, let's say high permission software piece, right?
And it can do everything and then giving it network capabilities is probably not the thing you want.
So NTP and stuff like, or secure NTP and stuff like that is not a great thing to do.
And then you getting network, like, let's say you're getting network time issues, so you don't really know what the time is.
You can sure store the time at the cmos or some kind of vram variable and sync it, but to be honest you cannot really believe it because anyone in the operating system can change it value and maybe of a skew implementation.
But this can also lead to glitches in the time.
So not always.
It's guaranteed that you have the correct time on the firmware level, so time doesn't matter so much there.
So even the certificate expires, you could still use it for a longer time because the verification of secure boot is not checking time fields inside the certificates.
There's no joke.
So you can basically change, as far as I know, scary.
Yeah, but, yeah that's scary.
But the thing is like if you don't have time to use to verify against it a secure time source, how do you do that?
Maybe some firmware implementations do that and they have some kind of figure out some way and try to single whatsoever.
But to be honest, using time and certificates is like certificate in general inside embedded firmware was not a great idea.
Probably just use keys and you can never basically invalidate these kind of keys, except when you exchange entire PKI on the side of secure boot.
But this is something you should always do.
So if you listen now to me, always try to customize your secure boot.
Never use the predefined keys in there because you don't know what kind of keys are in there.
We already figured out, for example, there are keys from odms and oems you never heard the name of, and they putting all their stuff in there for some kind of product they sell or whatsoever.
So whenever this kind of keys leaked, it can be used basically for malware, right?
So you should, the first thing you should do, get rid of the factory keys and replace it with your own myriad.
But then that, I guess that's very scary for a lot of vendors because now you're on the hook for maintaining keys and unless you really know what you're doing, you might cost yourself a world of pain, right?
Because PKI management is nothing trivial.
It's not trivial, but there's an easy way.
So if you want to have an easy way to do that for your Linux operating system, I can give you some recommendation.
There's SBCTL from Martin Lindgrund, he's one of the arch maintainers.
This tool is basically super easy to use to maintain secure boot on your system.
You just need to log into your BIOS, set a BIOS password, say factory, reset the keys, disable secure boot, and then you start in the system and use this tool for provisioning and rekeying is super easy.
You can exchange your keys on a daily basis if you want, or monthly or yearly, whatever.
But this tool makes secure root provision so easy.
I can really recommend it and it's available on GitHub.
You can just download it or just as part of your distribution might be already packaged.
I meant more for manufacturing, more so than for your laptop at home.
That's when it gets tricky.
Okay, so yeah, I mean that's probably the different story, but even there you could use some kind of like really like bind to.
For example, if you build a product right, you also want to make sure this kind of product is safe and sane.
And so maybe you have some kind of remote management inside this product.
So what you could do, you can customize your like with all keys and also do raking stuff.
For example for Windows operating systems for sure is the problem because they require their keys to be there.
But what you can do is also you can use your custom key to sign the Windows bootloader.
So be smart with it.
So it's also possible to do that.
So what you can do is then basically like build up your own secure boot chain over the Windows bootloader and everything from there.
You might still need to include some certificate from Microsoft because of other reasons.
That's not super nice, but at least it reduces the attack surface.
Basically crazy because these vendor certificates, what the vendors ship on the PK and the CAC, you should check that out.
There are tools reading out all these keys or showing them basically in your Linux system.
We'll figure out they have tons of other certificates in there and it's not sure what is used for.
Sometimes it's used for drivers, sometimes use for extra hardware, sometimes it's used for their own products they want to ship on your systems.
So I can always recommend to do that, even if it's not so easy to customize your secure boot, but better to be safe.
That's a good catch up.
I would definitely have to take a look at that in detail and make sure what our device will shipping with any firmware, any keys that we are not familiar with.
I don't think so, but let's double check that.
That's a good thing to take a look at for sure.
I think we covered a lot of ground today and I think we've had great conversations.
I'm looking forward to going further into this in the future episodes and maybe have you and Alex together on an episode in the future and just go further into all these craziness.
But I think I am super excited what Binary has been doing over the last few years, and I'm super excited about keeping all these oems, odms and bias vendors on their toes to make sure we can raise the tide in terms of security for the hardware world because there is a lot of work to be done there.
Definitely.
So I mean, I would also love to be included in the next basically podcast, but I would say in general, if you're also interested to speak about that more, you should visit the open source firmware conference because a lot of people coming from big companies and big enterprises there, and probably also maybe you can find some other people doing a podcasts about it and see their view on the things.
So this ecosystem is similar to the baseband ecosystem, you know, baseband also super, let's say niche, closed source, NDA driven ecosystem and also super proprietary.
And the firmware level is basically the same, less worse probably, but because we fight for a lot of things there and we made it, but it's still like a lot of things to do.
And I think we can learn from each other what kind of views we have and how we can improve on that, because it's not always so easy.
And I said I think a lot of issues there is person like people culture in that way that these people are not really well educated enough, not because they can be really great in terms of hardware and fixing problems there and whatever, but they're not well educated in terms of like what is currently the world going into one kind of direction, right?
Even Microsoft had to admit that Linux is a major operating system for a lot of devices.
They started to use it on Azure.
So I was at all systems go last year with Leonard Pottering and all these folks and they told me like the biggest part of Azure is running on Lambda Linux.
There's no Windows operating like some stuff running still on our Windows.
But even Microsoft is endorsing Linux a lot on system and I think we need not only Linux is important I think, but the open source ecosystem which comes with it will help us to improve a lot and also it will help us to be more flexible, customize things and find security vulnerabilities and even like have responsible disclosure much easier for sure.
It also brings risk as you could see with the exact fail issue right where someone tried to embed some kind of backdoor into it and we just figured it out by accident.
But let's say there's a lot of potential and I think it's always better to work together with a lot of companies and with a big ecosystem instead of doing your own thing.
And this is what you can see in the automotive sector.
Automotive is trying to move away from their closed source ecosystem and try to go more into mainline with Linux operating systems.
Yeah, no, and I think that's transparency and responsible development is a good note to end on.
So thank you so much for coming on the show Philip, and talk to you soon.
Cheers.
Found an error or typo? File PR against this file or the transcript.