[00:01]
Viktor Petersson
Welcome back to another episode of nerding out with Victor.
[00:04]
Viktor Petersson
Today I am joined by a company that I have been keeping my eyes very closely on lately called binary.
[00:12]
Viktor Petersson
And Philipp Diepenweisen is joining me today from binary.
[00:16]
Philipp Deppenwiese
Hey, nice to meet you.
[00:18]
Viktor Petersson
Great.
[00:19]
Viktor Petersson
So the reason why I've been somewhat fascinated and obsessing with over binary is the sheer volume of discoveries that you guys have published.
[00:30]
Viktor Petersson
And I been trading emails with Alex, who's the founder, and yourself as well, about this.
[00:38]
Viktor Petersson
It's just amazing how much you've rocked the whole firmware landscape I guess, over the last few years.
[00:46]
Viktor Petersson
But before we dive into those details, maybe do a quick introduction.
[00:51]
Viktor Petersson
Who's Philip?
[00:52]
Viktor Petersson
What should we know?
[00:53]
Viktor Petersson
And what is also what's finally.
[00:56]
Philipp Deppenwiese
Okay, then, yeah, I will explain a bit about me.
[00:59]
Philipp Deppenwiese
So I'm Philipp Depen Wiese.
[01:01]
Philipp Deppenwiese
I'm basically being in it security industry since around 15 years.
[01:07]
Philipp Deppenwiese
I started to work for the local government.
[01:09]
Philipp Deppenwiese
Like companies here, like government contracted, everything is federated stuff, military, secret agencies.
[01:16]
Philipp Deppenwiese
On some point I moved into a web development agency that was kind of fun because I never have been doing web development and focusing in UI UX thingies.
[01:25]
Philipp Deppenwiese
And then in this company, I founded my own department called Nineelements Cybersecurity.
[01:30]
Philipp Deppenwiese
They're still there nowadays.
[01:31]
Philipp Deppenwiese
I worked there for six years, passed it over like everything I did to my successor, Christian Walter, who was on.
[01:38]
Viktor Petersson
The show a few episodes ago.
[01:39]
Philipp Deppenwiese
Yeah, right.
[01:40]
Philipp Deppenwiese
And he's basically taking that on and it works in great.
[01:43]
Philipp Deppenwiese
It was about open source firmware development, especially as an open source firmware IPV and that, let's say in that time I also focused on creating the open source firmware conference and the open source firmware foundation together with my team there.
[01:57]
Philipp Deppenwiese
And so I'm also the president of the open source Firmware foundation in us.
[02:02]
Philipp Deppenwiese
But yeah, after that time, I basically left for having my own startup.
[02:07]
Philipp Deppenwiese
I did the three ESD scores in terms of like, let's say endpoint protection.
[02:12]
Philipp Deppenwiese
We worked on something called remote attestation.
[02:15]
Philipp Deppenwiese
Can probably talk a bit about that later as well.
[02:18]
Philipp Deppenwiese
That didn't work out.
[02:19]
Philipp Deppenwiese
But yeah, after this three years journey, I did one year development of another important project in the open source industry as freelancer for system transparency.
[02:29]
Philipp Deppenwiese
That was also kind of interesting topic.
[02:31]
Philipp Deppenwiese
And after that I landed somewhere, somehow.
[02:34]
Philipp Deppenwiese
It's binary here.
[02:36]
Philipp Deppenwiese
And now I'm the head of solution engineering at Binary.
[02:39]
Viktor Petersson
That's.
[02:40]
Viktor Petersson
You have so many topics I would like to dive into that just briefly mentioned there because there are quite a few of those interesting things I would like to cover.
[02:48]
Viktor Petersson
But maybe before we go in that give me a bit of backstory on binanly and just what is binary, how?
[02:56]
Viktor Petersson
Well, what's the purpose?
[02:57]
Viktor Petersson
And just without too much vendor pitch, but just give me some big backstory.
[03:01]
Viktor Petersson
Really.
[03:02]
Philipp Deppenwiese
Yeah, so I tried to do so.
[03:04]
Philipp Deppenwiese
I mean, I met Alex, was it like four years ago?
[03:08]
Philipp Deppenwiese
And basically finally came up of his mind.
[03:11]
Philipp Deppenwiese
So he was a visionary.
[03:12]
Philipp Deppenwiese
He was coming from Nvidia.
[03:14]
Philipp Deppenwiese
He was one of the security researchers, the lead security researchers there basically.
[03:20]
Philipp Deppenwiese
And he has a huge background in security research.
[03:23]
Philipp Deppenwiese
He also wrote a book called Bootkats and Rootkits.
[03:27]
Philipp Deppenwiese
He even bought it like 15 years ago or something like that.
[03:31]
Philipp Deppenwiese
So he's working on the security research stuff for quite a while.
[03:36]
Philipp Deppenwiese
And he has also been an expert on reverse engineering and all this like lower level vulnerability and all this stuff.
[03:43]
Philipp Deppenwiese
And I think, like he basically, what he told me is like he basically wanted to look into this area and then he started basically binary.
[03:53]
Philipp Deppenwiese
And the focus of binary was basically looking into binaries.
[03:59]
Philipp Deppenwiese
So that's why it's also called binary.
[04:00]
Philipp Deppenwiese
It's like kind of a nice play, you know, don't know if it's called binary or binary anyway, which is kind of strange from the name, but it's is good people think about it.
[04:10]
Philipp Deppenwiese
And interesting part is what we do different from a lot of other companies is we look into the binary level, so we care more about reverse engineering.
[04:20]
Philipp Deppenwiese
And we use this reverse engineering technology not to uncover secrets in the binaries.
[04:25]
Philipp Deppenwiese
We just more look into the binaries to figure out potential risk that includes, for example, known unknown vulnerabilities, malicious code, for example, if you embedded backdoor into a binary or whatever, and cryptographic materials and stuff like that.
[04:40]
Philipp Deppenwiese
So that's where binary is basically focusing on.
[04:44]
Philipp Deppenwiese
And that's quite unique in the industry because most of these companies just look on the source code level and we don't require them.
[04:51]
Viktor Petersson
Yeah, and I mean, I've had a few episodes around firmware and bias and the core boot guys, for instance.
[04:59]
Viktor Petersson
We talked a lot on that episode.
[05:01]
Viktor Petersson
We talked a lot about what goes into bios, for instance.
[05:04]
Viktor Petersson
And I think that's, I mean, that's obviously a sweet spot for you guys to do version during our bios.
[05:09]
Viktor Petersson
And I guess for people who have never done any of this, it's kind of a, it's a strange world, I guess.
[05:18]
Viktor Petersson
And it's a world where modern supply chain security is not a thing.
[05:24]
Viktor Petersson
And I've been covered that multiple times on the episode.
[05:27]
Viktor Petersson
And it's scary to someone how little transparency there is in that world, right?
[05:36]
Viktor Petersson
I guess Binarly's claim to fame and really big breakthrough was the discovery of logo fail.
[05:43]
Viktor Petersson
And that was probably the first major thing to hit the mainstream media where people like holy shit.
[05:50]
Viktor Petersson
And that definitely was my reaction when I first read it and I was like wow.
[05:54]
Viktor Petersson
So maybe talk me through how this was discovered and maybe like for people nothing familiar with it, the impact for the world because it was, it's, the blast radius is enormous for wow.
[06:13]
Viktor Petersson
We'll talk about other ones but just start with that one.
[06:15]
Viktor Petersson
Like talk about what logo fail and how was this discovered?
[06:21]
Philipp Deppenwiese
Yeah so let's talk a bit about that.
[06:23]
Philipp Deppenwiese
So basically I think that was around two and a half years ago.
[06:27]
Philipp Deppenwiese
I like we working together with Alex in a company partnership and I already talked to Alex about issues in the field we have seen.
[06:36]
Philipp Deppenwiese
And one of the things where we had to manually analyze some firmware I think it was from, let me think what kind of company was a german company, Fujitsu, Siemens?
[06:46]
Philipp Deppenwiese
no I remember or no Japanese, I don't know, they're like japanese German, whatever company.
[06:52]
Philipp Deppenwiese
And the interesting part is like we already found out there's some issues with unsigned let's say boot logos, right?
[07:01]
Philipp Deppenwiese
It's about logo fail is about the BIoS boot logo.
[07:04]
Philipp Deppenwiese
So if you want to upload your own boot logo into the firma it's often possible because people love customization companies as well and so they make opportunities that you can change the logo during boot.
[07:16]
Philipp Deppenwiese
So it's not showing Lenovo or whatever, it's showing their custom product line whatever for example Palo alto networks if they have that and then they show that in this bootloader and you can customize it.
[07:29]
Philipp Deppenwiese
And during the time we already figured out these images are not signed because you want to exchange them maybe from the operating system side of you.
[07:36]
Philipp Deppenwiese
And this was basically I think also some impression for Alex because I told him like this seems kind of sketchy right?
[07:43]
Philipp Deppenwiese
There's no validation for that, you can just change it.
[07:46]
Philipp Deppenwiese
And they said let me look into that.
[07:48]
Philipp Deppenwiese
I think during the time, that was around the time two and a half years ago they already like put it on the to do list and on some points they came to the like basically to this logo fail and look to like looked into all these type of lobe images you can basically upload with different kind of parsing libraries.
[08:08]
Philipp Deppenwiese
So there are a lot of let's say so you can check out the Tiana core, it's an open source implementation of UEFI.
[08:14]
Philipp Deppenwiese
Basically it's on GitHub, you can just check it out and you can also just check out basically the image parser, right?
[08:21]
Philipp Deppenwiese
And there are different types of image types nowadays.
[08:23]
Philipp Deppenwiese
You have JPeg, you have Png to have BMP or whatsoever.
[08:27]
Philipp Deppenwiese
And I think it was JPeG mostly and BMP what was used.
[08:31]
Philipp Deppenwiese
And they already found like vulnerabilities in that by just looking at the code, right.
[08:35]
Philipp Deppenwiese
And then you can do further analysis, basically looking into closed source implementation.
[08:40]
Philipp Deppenwiese
Try to use pattern matching to figure out the same like vulnerabilities in these kind of, let's say firmware images.
[08:48]
Philipp Deppenwiese
And that was possible because bindly made basically this kind of technologies in the backend, and they were able to basically prove that also in production images, they had vulnerabilities for these different type of images.
[09:02]
Philipp Deppenwiese
So I could pull up some kind of presentation.
[09:04]
Philipp Deppenwiese
But I think this is like, this is just to go into deep.
[09:07]
Philipp Deppenwiese
I mean, there's a lot of materials on the Internet nowadays.
[09:10]
Philipp Deppenwiese
We also released, I think in the blog post, extensive review about all the stuff.
[09:15]
Philipp Deppenwiese
But basically it ended up after this type of research and responsible disclosure that there were a lot of system where you could easily just like make a malicious BMP image or JPEG image, just put it there, and then the BIOS took it, passed it, run in a buffer overflow, and basically you had code execution on the firmware.
[09:36]
Philipp Deppenwiese
And I mean, most people say, why is it important?
[09:40]
Philipp Deppenwiese
Right?
[09:40]
Philipp Deppenwiese
But the firmware has the most permissions inside the operating system, even before the operating system.
[09:46]
Philipp Deppenwiese
But even during runtime of the operating system, there's also something called system management mode.
[09:51]
Philipp Deppenwiese
So if you can sneak in before the operating system starts with some kind of exploit into the firmware, you can basically burn in an implant inside the UFI firmware image, and you're able to even attack the operating system site through system management mode, which is basically runtime service into the operating system.
[10:11]
Philipp Deppenwiese
There also have been some examples of that, like theoretical ones, and some are really in the field, but more like for the, let's say, intelligence people, where they have really advanced persistent threats there.
[10:23]
Philipp Deppenwiese
But let's say this already has been a huge problem because the firmware level is really not seen from the operating system part of it.
[10:32]
Viktor Petersson
So I have two questions about it.
[10:33]
Viktor Petersson
The first one is, like you mentioned, that you have some tools for discovering and analyzing.
[10:39]
Viktor Petersson
You kind of already answered this a bit about discovering this part.
[10:43]
Viktor Petersson
It was more like a gut feeling than anything, by analyzing the code.
[10:48]
Viktor Petersson
But that then revealed that what are the weaknesses of the open source or sorry, the closed source bias world, which is.
[11:00]
Viktor Petersson
Well, you don't know what goes into your bias at all.
[11:03]
Viktor Petersson
Like you have no clue, right?
[11:05]
Viktor Petersson
It's just, well here's a binary blob.
[11:07]
Viktor Petersson
Have fun.
[11:08]
Viktor Petersson
You may or may not even have a sember on that.
[11:10]
Viktor Petersson
You probably don't even like.
[11:12]
Viktor Petersson
You might.
[11:12]
Viktor Petersson
But it's definitely not a whole lot of insight that you can actually pull from that.
[11:19]
Viktor Petersson
And this is where it got really interesting why I think what you guys tooling, and I kind of covered that in previous episode as well.
[11:26]
Viktor Petersson
It's like how do I even find a blast radius of this?
[11:30]
Viktor Petersson
I know some bias are affected, but there is so little, there are no s bombs you can pull.
[11:35]
Viktor Petersson
You can correlate this between all the vendors and there is a lot of reusable code between vendors.
[11:42]
Viktor Petersson
So maybe speak a bit about that.
[11:43]
Viktor Petersson
Like the tooling you guys have in place for like checking this out and.
[11:47]
Viktor Petersson
Yeah, maybe start from there.
[11:50]
Philipp Deppenwiese
Yeah, so I mean this tooling, like we written a lot of tooling in binanly, but also in the already open source world there's some tooling for looking at UFI images for sure you can look at the open source implementation tano core and the source code level.
[12:05]
Philipp Deppenwiese
But there's for example another tool called UEFI tool which is not from us, it's from some engineer like of Apple who basically maintains that and basically wrote together tooling for looking at UEFI images.
[12:18]
Philipp Deppenwiese
And they also add some security checks and you could see what kind of parts, for example of the boot flow, unsigned or not.
[12:25]
Philipp Deppenwiese
So that's where I basically got like whoa, why it's not signed.
[12:29]
Philipp Deppenwiese
Right.
[12:29]
Philipp Deppenwiese
You can just use some kind of open source tool, look at it and then figure out, oh, this is not good, how it looks like.
[12:35]
Philipp Deppenwiese
Right.
[12:35]
Philipp Deppenwiese
That's one option.
[12:36]
Philipp Deppenwiese
We released FW Hunt which was basically the first version of it, which is basically a world based firmware scanner.
[12:44]
Philipp Deppenwiese
So it tries to find through pattern matching and some technology like the vulnerabilities in different kind of firmware image.
[12:50]
Philipp Deppenwiese
You can just scan any firm image, try to match this type of patterns.
[12:53]
Philipp Deppenwiese
We have a rule description.
[12:55]
Philipp Deppenwiese
So this is open source.
[12:56]
Philipp Deppenwiese
It was basically the first version of it.
[12:59]
Philipp Deppenwiese
Now we have the second version probably you did hear about it or not, I'm not sure.
[13:04]
Philipp Deppenwiese
So we released it during Blackhat this year is basically RISC Binary IO, which is now a full service.
[13:10]
Philipp Deppenwiese
So you don't have this kind of menu invoking tooling anymore.
[13:14]
Philipp Deppenwiese
You can just use a rest API to find known vulnerabilities in the firmware space in generating sboms out of it.
[13:21]
Philipp Deppenwiese
That's completely free.
[13:23]
Philipp Deppenwiese
You even can use it anonymized.
[13:25]
Philipp Deppenwiese
We don't care so much about registration and things just for API access, so you only have access to what you upload.
[13:32]
Philipp Deppenwiese
And this service should basically help also the community.
[13:35]
Philipp Deppenwiese
So we're currently working together with Richard Hughes from the lvfs.
[13:39]
Philipp Deppenwiese
So basically from Red Hat, he's going to integrate as part of lvfs.
[13:44]
Philipp Deppenwiese
So all images of lvs will be scanned by this kind of service.
[13:49]
Philipp Deppenwiese
And there are also other companies like blind spot software from Christian Wilder, who has been on this video, like on this, let's say, episode as well, and.
[14:00]
Viktor Petersson
Richard Hughes as well.
[14:00]
Viktor Petersson
Actually, he's also been on the show.
[14:03]
Philipp Deppenwiese
Yeah, yeah.
[14:04]
Philipp Deppenwiese
So, and then there's OCP, for example, open compute project, which are trying to define like open data center standards, and they probably make use of it as well.
[14:13]
Philipp Deppenwiese
So we want to spread it a bit more.
[14:15]
Philipp Deppenwiese
And the important part of it is we will also add a lot of experimental features from our, basically new research into that, so that we can like see if, what kind of benefit it brings to the masses.
[14:28]
Philipp Deppenwiese
And also people who want to use it for free, just try to do that and you can figure out what's inside your finger images.
[14:35]
Philipp Deppenwiese
And especially, as you say, generating s bombs is a big problem, right?
[14:39]
Philipp Deppenwiese
Because just generating it or believing in S bomb, it's a text file, bom.
[14:44]
Philipp Deppenwiese
It's from the hardware point of view, right, bill of materials, and then it's just a document about what kind of components are included in your system.
[14:52]
Philipp Deppenwiese
And the same goes for software, for sbom.
[14:54]
Philipp Deppenwiese
But what you need to do is basically you need to prove if the S bom is matching basically the contents of the binary and this binary, we decided not to go that way anymore to say from source code to SBOM, I don't think this is good, because you have a lot of, let's say, third party software and even binary components going into there.
[15:13]
Philipp Deppenwiese
So the best way to do that is basically generating it out of the binary in the end and generating an SBOM.
[15:19]
Philipp Deppenwiese
So you can basically prove what's in there and looking, by looking into it.
[15:23]
Viktor Petersson
And you can, I guess, based on the hashes, when expanding the binary, I guess you can find, you could detect based on the hash, the version of a component, basically.
[15:36]
Viktor Petersson
Because I guess you have some kind of library or index for that, right?
[15:40]
Philipp Deppenwiese
Yeah, I mean, there is some kind of limitation.
[15:43]
Philipp Deppenwiese
You need to know the most important part nowadays is to be honest, to build software reproducible.
[15:49]
Philipp Deppenwiese
The main issues with it is that like we produce the build.
[15:52]
Philipp Deppenwiese
The problem is the biggest part of the industry is not doing it.
[15:55]
Philipp Deppenwiese
So comparing hashes is somewhat useful.
[15:58]
Philipp Deppenwiese
If it's a release and this is not customized or patched software, then you can somehow compare that to basically to the public releases, right?
[16:08]
Philipp Deppenwiese
That works.
[16:09]
Philipp Deppenwiese
But for this type of like, let's say customize modification and stuff like that, it's really hard to really validate.
[16:15]
Philipp Deppenwiese
If it's really the binary you're searching for, sure you can check the hash, but as I said, limited.
[16:25]
Philipp Deppenwiese
So we also have to look into other parameters and looking at towards a binary level and checking for specific similarities there, especially for patterns in order to figure out if it's really the case.
[16:37]
Philipp Deppenwiese
You can for sure just check the version number and stuff like that.
[16:39]
Philipp Deppenwiese
But someone can fake it, right?
[16:41]
Philipp Deppenwiese
So it's lots of no, right?
[16:43]
Viktor Petersson
Yeah, I wouldn't expect the version to be super helpful in this context.
[16:47]
Viktor Petersson
So I guess the obvious question is, well, not obvious, but my obvious question is like how was this received by the industry?
[16:55]
Viktor Petersson
Right?
[16:56]
Viktor Petersson
For my feeling, it was not exactly welcomed by the buyers vendors.
[17:04]
Viktor Petersson
So how did that look like the conversation between, well, when you did the responsible disclosure and what was the response from the vendor BIOS vendor community?
[17:14]
Philipp Deppenwiese
So let's say I've worked for a long time with this IPV folks, right?
[17:19]
Philipp Deppenwiese
I worked a long time with this kind of IBVs, ODMs, OEMs in my previous field at nine elements, cybersecurity.
[17:27]
Philipp Deppenwiese
Now I do the same again, binary.
[17:30]
Philipp Deppenwiese
The problem with that is responsible disclosure is quite important, right?
[17:34]
Philipp Deppenwiese
And everyone agrees on that in the theoretical world.
[17:37]
Philipp Deppenwiese
Really great, because you get two months of time, you can fix the bugs, everything goes out, everything is safe and sane.
[17:44]
Philipp Deppenwiese
The end user gets a fix before basically the disclosure is done and everything is published.
[17:52]
Philipp Deppenwiese
But the problem is with, let's say, hardware folks or companies, hardware companies, they are not really used to that type of ecosystem.
[18:01]
Philipp Deppenwiese
They're not even understanding software.
[18:03]
Philipp Deppenwiese
So once I give you an example, once I ask it, odms or oems like that, they also ship software, right?
[18:11]
Philipp Deppenwiese
I said like to them, you ship hardware, right?
[18:13]
Philipp Deppenwiese
Yeah, we do chip hardware.
[18:15]
Philipp Deppenwiese
But you also ship software as part of the hardware, right?
[18:18]
Philipp Deppenwiese
No, you only ship hardware.
[18:20]
Philipp Deppenwiese
And then I say it.
[18:21]
Philipp Deppenwiese
But wait a moment, there's like probably 32 megabytes of Biosim there, and additionally components.
[18:26]
Philipp Deppenwiese
Sometimes they have even a baseball management controller, which is like this kind of control unit for server systems with 128 megabyte of Linux system on it, right?
[18:36]
Philipp Deppenwiese
And said like yeah but this is software right?
[18:38]
Philipp Deppenwiese
No, it's hardware the management doesn't understand especially asian like odms and oems we're not really let's say understanding and welcoming in this area because they don't understand what it means they also don't understand not all of them for sure there are better examples like framework for example really good company so my laptop is from framework but also like they already like they have more, they have better people to understand how it goes but for all these other odms really hard to understand it and even this is not the only problem so most people don't know but probably I can talk about a bit more so the people who are writing software in this area are mostly third party suppliers so that means if you are an ODM you don't you maybe you develop a part of your firmware yourself or customize it but most of the stuff comes from another company and you just buy it in and these companies are for sure Ami phoenix insight there might be also open source ipvs in there like nine elements they are better than them these closed source companies let's say and the problem with that is you don't understand what your tech stack does because you're basically just buying stuff in and then you have to deal with them and aside from that the software quality is not so good in this area because firmware engineers haven't been at conference for last 30 years.
[20:08]
Philipp Deppenwiese
So I mean I started the open source firmware conference because of the reason there were no firmware conference.
[20:12]
Philipp Deppenwiese
You can go to some kind of trade fair and you can also find small conferences about Firbair but most of these events are not really focused on the developers and.
[20:23]
Philipp Deppenwiese
Right and so firmware conference was one idea to basically get them all together and this is the main issue like those people normally sitting in their let's say area or ecosystem circle and they don't go out and they don't see what else is there and this is a big problem.
[20:41]
Philipp Deppenwiese
So it's not like if you look at web developers or if you look at application developers where they go to big conferences and then talk and exchange about different kind of frameworks and topics it's not like that.
[20:54]
Philipp Deppenwiese
And all these basically play into the, into this ecosystem where it's really, it's bad kind of understanding about security vulnerabilities and even software from the hardware vendors perspective and also these people who develop firmware are not really skilled to write good firmware and also don't understand security at all.
[21:14]
Philipp Deppenwiese
So all of this combines into basically one result that the firmware you get there is mostly really insecure.
[21:22]
Philipp Deppenwiese
It's not really well developed.
[21:25]
Philipp Deppenwiese
Best example is, for example, a friend of mine.
[21:27]
Philipp Deppenwiese
This is another story, not even from the security perspective, but functional perspective.
[21:32]
Philipp Deppenwiese
Normally if you update your bios, basically the program tells you not to shut down the device, right.
[21:39]
Philipp Deppenwiese
They tell you don't shut down, don't do anything at all, just keep it on the powered on and have a power supply to it.
[21:46]
Philipp Deppenwiese
Because if it turns off, everything breaks.
[21:49]
Philipp Deppenwiese
Right?
[21:49]
Viktor Petersson
Yeah.
[21:50]
Philipp Deppenwiese
And this is like super funny.
[21:52]
Philipp Deppenwiese
And then if you are Finbi engineer like me, and you know what to do, and you already have seen like products like Chromebooks for example, use open source firmware core boot.
[22:02]
Philipp Deppenwiese
They developed a model where they can never break, they can never have a break of the system.
[22:06]
Philipp Deppenwiese
You can shut it down all the time.
[22:08]
Philipp Deppenwiese
The update will not kill the, like the system basically, and there's no need to worry about that stuff.
[22:14]
Philipp Deppenwiese
But in this world it is.
[22:16]
Philipp Deppenwiese
And I even they have some kind of recovery machine and most of the time it fails.
[22:20]
Philipp Deppenwiese
So there you can see how really bad software development and knowledge in this firmware world is not about the hardware.
[22:27]
Philipp Deppenwiese
They have amazing knowledge about hardware and how to do specific things, right.
[22:31]
Philipp Deppenwiese
In terms of connecting firmware to hardware, but they are not good in developing software at all.
[22:36]
Philipp Deppenwiese
But anyway, this is just a long, let's say long discussion about that.
[22:41]
Philipp Deppenwiese
And I hope I answered the question there.
[22:44]
Viktor Petersson
Yeah, yeah.
[22:46]
Viktor Petersson
So I think, yeah, it's a cultural problem to a great extent, right?
[22:50]
Viktor Petersson
And I think so, long story short, the disclosures were not very happily received.
[22:57]
Viktor Petersson
And then I guess what is the state of affairs currently?
[23:01]
Viktor Petersson
Because that was announced what, eight months ago, I think, or something like that to the public or ten months ago, probably by its public.
[23:10]
Philipp Deppenwiese
The UEFI forum tells you one year.
[23:13]
Philipp Deppenwiese
So if you look at the Tiana core, like the UEFI is the specification, right.
[23:17]
Philipp Deppenwiese
If you look at their forum websites, they tell you they need one year to do response, like they need one year to close the bug.
[23:24]
Philipp Deppenwiese
It's insane, right?
[23:25]
Philipp Deppenwiese
Because like one year is unbelievable amount of time.
[23:28]
Philipp Deppenwiese
I mean, how long does it need to, until the fixes end up at the customer side, right?
[23:34]
Philipp Deppenwiese
And you are right, they don't take this well.
[23:36]
Philipp Deppenwiese
They don't like security vulnerabilities.
[23:38]
Philipp Deppenwiese
They hate them basically.
[23:40]
Philipp Deppenwiese
And so this is, it's a big problem to do.
[23:43]
Philipp Deppenwiese
Responsibilities, closure.
[23:44]
Philipp Deppenwiese
There it's so big that on sometimes for the vendors we not going to decide to do that anymore at all.
[23:50]
Philipp Deppenwiese
We give them a period for sure and say like you have time, but it's impossible to just do that.
[23:56]
Philipp Deppenwiese
It costs you tons of money on your side because you need to have, they want to have meetings, they want to have information and stuff like that.
[24:03]
Philipp Deppenwiese
It costs you a lot of money to actually do responsibilities.
[24:06]
Philipp Deppenwiese
I mean security researchers from university for example, they do it for free because that's their part of their research or whatsoever, you know.
[24:14]
Philipp Deppenwiese
But if you're doing it as company.
[24:15]
Viktor Petersson
Like, yeah, different story.
[24:18]
Viktor Petersson
So I guess the winning strategy going forward for you guys, there will be a name and shame kind of approach.
[24:25]
Viktor Petersson
It sounds like that.
[24:26]
Philipp Deppenwiese
I wouldn't say like we would, we don't want to change them basically publicly on it, on a wall of shame, let's say.
[24:33]
Philipp Deppenwiese
But there will be definitely people showing like metrics and showing information about, let's say hardware vendor names which are better or let's say better in terms of security, which are faster, close security bugs and stuff like that.
[24:47]
Philipp Deppenwiese
If these metrics comes out then I mean this is just research from our side and also from other sites as well.
[24:53]
Philipp Deppenwiese
It's not only us.
[24:54]
Philipp Deppenwiese
And then you can see what kind of hardware you should buy from what kind of render, right?
[24:59]
Viktor Petersson
Yeah.
[24:59]
Philipp Deppenwiese
And I think the change cannot be introduced by publicly shaming them.
[25:03]
Philipp Deppenwiese
It would be super nice to be honest.
[25:05]
Philipp Deppenwiese
But the thing is like it's an ethical question.
[25:09]
Philipp Deppenwiese
And also, I mean the best thing is if the customer of these hardware vendors, if they have transparency into their basically firmware, they get from their customer, from the hardware vendor, they can see how good it is and they can see next time if it's bad, for example, or it gets worse or whatever, they can say, oh yeah, then I get a 20% discount, right.
[25:33]
Philipp Deppenwiese
And then like things end up not being so nice on the hardware vendor side anymore.
[25:38]
Philipp Deppenwiese
I don't believe, personally, I don't believe you can force hardware vendors without business incentives.
[25:43]
Philipp Deppenwiese
So what you need to do is to have all these enterprise companies basically giving them transparency to show like also they can save a lot of money, but they can also basically understand what kind of hardware is running.
[25:56]
Philipp Deppenwiese
Imagine for example, you build a product, you are parallel to networks, right.
[26:01]
Philipp Deppenwiese
And this is just an example, right.
[26:03]
Philipp Deppenwiese
But they're basically building like switches, firewalls whatsoever.
[26:07]
Philipp Deppenwiese
Right.
[26:08]
Philipp Deppenwiese
And then you buy hardware or you get hardware from your ODM, from Taiwan, for example.
[26:13]
Philipp Deppenwiese
And the firmware shitty.
[26:14]
Philipp Deppenwiese
But you don't have any tool to look into this firmware stuff, right.
[26:18]
Philipp Deppenwiese
You don't have any transparency you have talked about before.
[26:21]
Philipp Deppenwiese
Then somehow you need to figure out, yeah.
[26:25]
Philipp Deppenwiese
What's in there.
[26:25]
Philipp Deppenwiese
Right.
[26:26]
Philipp Deppenwiese
Because it's a risk that it ends up as a product, let's say security vulnerability on your side.
[26:34]
Philipp Deppenwiese
It's not like the ODM gets blamed for that.
[26:36]
Philipp Deppenwiese
It's your product.
[26:37]
Philipp Deppenwiese
And so you get blamed and you get basically the bad press about it.
[26:43]
Philipp Deppenwiese
And so this will change hopefully in the future when we give those companies the tools for it.
[26:49]
Viktor Petersson
Do you see the, so the EU has been pretty proactive about legislating, well, I guess trying to fix security in a very, european way, in which some of it is good, some of it is not.
[27:06]
Viktor Petersson
But do you think there is a.
[27:08]
Viktor Petersson
I'm actually rather bullish on that.
[27:11]
Viktor Petersson
The only way security will be improved is probably through legal means, because I don't think there's enough business incentive to actually solve, properly solve security.
[27:23]
Viktor Petersson
That's my, particularly in the Iot world and the embedded world, I think it's gonna be hard to solve it without legal tools to force vendors, like you said, like to force them to actually care.
[27:38]
Viktor Petersson
What are your thoughts around that?
[27:39]
Philipp Deppenwiese
Yeah, so I mean, there's one option, basically, nice phrase like that.
[27:43]
Philipp Deppenwiese
But let's say there's one thing you can do.
[27:46]
Philipp Deppenwiese
You can for sure give the, like the enterprises the opportunity to look into this binaries and, or black boxes.
[27:52]
Philipp Deppenwiese
And when they figure out vulnerabilities, they can basically like reduce the amount of money they need to pay to these odms.
[27:58]
Philipp Deppenwiese
And this is bad thing for odms.
[28:00]
Philipp Deppenwiese
So money is the driving thing in the ODM market.
[28:03]
Philipp Deppenwiese
It's one option.
[28:05]
Philipp Deppenwiese
The other thing is compliance for sure.
[28:07]
Philipp Deppenwiese
So, I mean compliance for regulations.
[28:09]
Philipp Deppenwiese
So you cannot, for example, buy from hardware vendors who don't comply with specific regulations.
[28:14]
Philipp Deppenwiese
Right.
[28:15]
Philipp Deppenwiese
This is another driver, I would say, and this one is, I think, only, let's say it's only helpful if it either enforces you not to buy hardware stuff there or there are hefty penalties.
[28:32]
Philipp Deppenwiese
And I think with the latest regulations from european unions, the Cyber Resilience act, which come into place, I think, what was it?
[28:39]
Philipp Deppenwiese
2025, mid of 2025.
[28:41]
Viktor Petersson
I don't know what it's coming to, into play, actually, but it's, yeah, but it's a pretty comprehensive piece of framework.
[28:46]
Philipp Deppenwiese
Let's say until summer 2026, every country needs to have it in place or a bit later.
[28:51]
Philipp Deppenwiese
And the thing is with that, it's already making a lot of infrastructure, critical infrastructure.
[28:56]
Philipp Deppenwiese
That's the first thing what this thing does.
[28:58]
Philipp Deppenwiese
And then critical infrastructure has regulations like they need to fix security vulnerabilities.
[29:04]
Philipp Deppenwiese
That's for example important.
[29:05]
Philipp Deppenwiese
And if we can detect vulnerabilities in the third party components or in this like black boxes because we don't need the source code because they don't like they try to obfuscate all the stuff.
[29:15]
Philipp Deppenwiese
But if we are able to show that, then they need to fix these bugs.
[29:18]
Philipp Deppenwiese
And this will also drive basically or force them through compliance, I think.
[29:23]
Philipp Deppenwiese
And also s bomb is for example, an important role, I think, for the Cyber Resilience act.
[29:28]
Philipp Deppenwiese
And this will come as well.
[29:30]
Philipp Deppenwiese
So it will just take a while until adoption.
[29:33]
Philipp Deppenwiese
But it's already like making this OEM hardware business much more difficult because years of like not doing anything at all and shipping just stuff and then you are trustworthy because the company have a contract with you since ten years.
[29:50]
Philipp Deppenwiese
That doesn't make sense, right?
[29:51]
Philipp Deppenwiese
I mean, if you go to the supermarket and you buy seafood, right?
[29:55]
Philipp Deppenwiese
You go to supermarket, look at the seafood and then you probably smell or look at the eyes of the fish to figure out.
[30:01]
Philipp Deppenwiese
And it's like basically the, like, you know what I mean?
[30:05]
Philipp Deppenwiese
Like the english word the gills.
[30:07]
Philipp Deppenwiese
Yeah, yeah.
[30:08]
Philipp Deppenwiese
For example, to see if it's fresh fish, right?
[30:11]
Philipp Deppenwiese
And this is what you do as normal person, like in order to figure out if it will poison you, right?
[30:18]
Philipp Deppenwiese
And if you eat it, food poisoning, basically the same goes with like all this hardware render stuff.
[30:24]
Philipp Deppenwiese
We cannot just trust the hardware ecosystem blindly and just hope that's secure.
[30:29]
Philipp Deppenwiese
I mean, how do you do that when most of the stuff is coming from China, right?
[30:34]
Philipp Deppenwiese
I mean, well that's, that was.
[30:35]
Viktor Petersson
Kind of my initial reaction as well is basically like, well, when we, when I got into the hardware world and started building hardware and it's just like, I have no idea what's actually inside of this.
[30:45]
Viktor Petersson
There could be backdoors.
[30:47]
Viktor Petersson
There could be like, I mean, I get a bias.
[30:48]
Viktor Petersson
Binary.
[30:50]
Viktor Petersson
Cool.
[30:51]
Viktor Petersson
What's in there?
[30:52]
Viktor Petersson
I can't build it.
[30:53]
Viktor Petersson
Certainly not reproducible build.
[30:55]
Viktor Petersson
Like I, I can't do anything with it, right.
[30:58]
Viktor Petersson
You can't even unlike you have, unless you have tools like you guys have to actually extract that and actually figure out what's in it, right?
[31:06]
Viktor Petersson
It's.
[31:06]
Viktor Petersson
And it certainly is a world of black boxes.
[31:09]
Viktor Petersson
I think these kind of toolings will force a change in that ecosystem hopefully because yeah, as long as it's treated as a black box and nobody can actually look into it and see what's in there, like nothing's going to change.
[31:24]
Viktor Petersson
And that is a good strike.
[31:25]
Philipp Deppenwiese
Yeah and this is a problem, I mean a big problem.
[31:29]
Philipp Deppenwiese
And I tell you like for example if you don't have tooling this basically is, you cannot do it, right.
[31:36]
Philipp Deppenwiese
But now we got tooling, it's basically possible to look at it.
[31:39]
Philipp Deppenwiese
And then people also start to think about the, I mean they can, there are multiple reasons why you want to do that.
[31:47]
Philipp Deppenwiese
But let's say there's also a big example.
[31:49]
Philipp Deppenwiese
We worked once with meta like for data center, for the data center area and did there for them some kind of security stuff.
[31:57]
Philipp Deppenwiese
And the interesting part was they were already looking on the hardware level, not even firmware like that.
[32:02]
Philipp Deppenwiese
They already looked at firmware as well, but they also looked at their hardware level in order to find basically backdoors inside hardware.
[32:10]
Philipp Deppenwiese
Just imagine that you take a full server like you get the server shipped right on the ship in a cargo and then it comes to your country and then you put it on some kind of transportation unit, whatever is it arrives at the data center and then the first thing you do you pull out like one or two servers and then put them in some kind of screening like you know basically x raying the entire chips and the PCB's and everything.
[32:34]
Philipp Deppenwiese
So the motherboards and looking, if there's any type of let's say component which is probably malicious which shouldn't be there.
[32:44]
Philipp Deppenwiese
Right.
[32:45]
Philipp Deppenwiese
And this is super crazy and this is already have been done for these cloud service providers basically.
[32:51]
Viktor Petersson
Right.
[32:51]
Philipp Deppenwiese
So like Microsoft Azure, Amazon with AWS and stuff like that, they already look at that kind of basically impact from the supply chain.
[33:03]
Viktor Petersson
Yeah that was, I'm not sure you remember that story about super micro servers being backdoored, right?
[33:09]
Viktor Petersson
A few years ago, I'm not sure that was a, I'm not sure that got stored, got pulled in the end if it was actually bogus or not.
[33:14]
Viktor Petersson
But I remember a conference talk about that a few years ago.
[33:18]
Viktor Petersson
Right.
[33:19]
Philipp Deppenwiese
I mean it was probably bogus.
[33:20]
Philipp Deppenwiese
I mean if you want to go into super micro hardware nowadays I would personally say I did some research on my null elements side during the time all the systems are unprotected.
[33:31]
Philipp Deppenwiese
You can just put a flasher on their firmware chip, refresh the firmware with something else no one figures out.
[33:37]
Philipp Deppenwiese
You can just like you don't even need to attack the hardware level, you can just do it on the software level by just flipping something on the hardware chip.
[33:44]
Philipp Deppenwiese
So I mean this is like completely insane, right?
[33:46]
Philipp Deppenwiese
And so in order to improve this ecosystem and also tell people it's really hard because most people are not that technical, right?
[33:54]
Philipp Deppenwiese
And you need to explain them, not even the vendors, whatever, even like the enterprise, like customers who buy that hardware, they are not really well trained on this kind of thread, right?
[34:04]
Philipp Deppenwiese
And they say, yeah, why is it important now?
[34:06]
Philipp Deppenwiese
It hasn't been a problem since the last 30 years, right?
[34:10]
Philipp Deppenwiese
So most people will tell you that, but the reality is like the firmware grows.
[34:14]
Philipp Deppenwiese
I talked about that a lot in my time of being at nine elements that the firmware, for example, in 1999 you probably had like, I don't know, 100 kilobyte of firmware components on your system.
[34:26]
Philipp Deppenwiese
In 2000 you had probably 1 firmware components in your system.
[34:30]
Philipp Deppenwiese
In 2010 it has been around 60 megabyte of firmware components in the system.
[34:36]
Philipp Deppenwiese
And nowadays we are around 32 to 64 megabytes of a normal laptop.
[34:41]
Viktor Petersson
I'm talking about, yeah, I ordered a new motherboard a few months ago and I was like 32 megs for bias.
[34:47]
Philipp Deppenwiese
That's crazy.
[34:49]
Philipp Deppenwiese
You know, you can put a complete Linux kernel inside there.
[34:51]
Philipp Deppenwiese
We did this with Linux boot just for data center applications.
[34:54]
Philipp Deppenwiese
We run Linux inside the firmware.
[34:56]
Philipp Deppenwiese
And then it just costs you eight megabytes.
[34:59]
Philipp Deppenwiese
And I mean, people can imagine what kind of impact it has.
[35:02]
Philipp Deppenwiese
And also the problem is like in the next ten years, I guarantee you we will end up to 1gb on the firmware level.
[35:08]
Philipp Deppenwiese
And the reason for that is all those chips getting smaller, get more features integrated, their bus systems like all the buses, like high speed memory, high speed transfer for USBC, whatever kind of new technology arrives after that.
[35:22]
Philipp Deppenwiese
Everything which needs high speed requires a huge amount of source code to control and also to manage.
[35:30]
Philipp Deppenwiese
And that means like we will get more and more chips with more and more firmware, which is more like Linux operating systems or real time operating systems.
[35:39]
Philipp Deppenwiese
And it becomes less, not only one chip, it becomes ten on the main board already.
[35:44]
Philipp Deppenwiese
And so this is super insane.
[35:46]
Philipp Deppenwiese
It means the firmware growth is massive and in the next ten years it will grow more.
[35:51]
Philipp Deppenwiese
And on some point we just have all these integrated chips communicating with a lot of operating systems.
[35:57]
Viktor Petersson
Yeah, that brings to our topic that you can hint at before, which is, I mean, you said if you buy a super microbox, just swap the firmware on.
[36:07]
Viktor Petersson
And that brings me to attestation, right, and source of truth and validation.
[36:12]
Viktor Petersson
And that's an entire cattle fish, right?
[36:16]
Viktor Petersson
I mean maybe before we dive in that because PK fail, which is one, was one of the latest disclosures from you guys, kind of goes in that territory.
[36:27]
Viktor Petersson
So talk me through PK fail and then I want to segue to have to stash at large.
[36:32]
Philipp Deppenwiese
Yeah, so let's say PKFA was basically the thing that we found in test key.
[36:39]
Philipp Deppenwiese
So there are multiple test keys by the way, not only one, I think it was 22 or something like that.
[36:44]
Philipp Deppenwiese
You can find that on our like outline on the blog post with some materials in the PDF, all the information.
[36:50]
Philipp Deppenwiese
So let's say the interesting part is that this test key have been used by a lot of companies, especially oems.
[36:59]
Philipp Deppenwiese
So for example gigabyte was one of them and supermic and some other, not all of them used it, but like most of them use this test key.
[37:07]
Philipp Deppenwiese
And normally those test keys shouldn't be used.
[37:09]
Philipp Deppenwiese
But the first thing is the test like keys, key materials in that ecosystem should be protected by hsms.
[37:17]
Philipp Deppenwiese
That's the first mistake, right?
[37:19]
Philipp Deppenwiese
They didn't do that.
[37:19]
Philipp Deppenwiese
Most of them just stored on the hard drive unencrypted and then just built from some kind of build PC or whatever.
[37:26]
Philipp Deppenwiese
Build this kind of, let's say, image.
[37:29]
Philipp Deppenwiese
The problem is also a lot of these, you can check that.
[37:32]
Philipp Deppenwiese
If you just check out film images downloaded from the homepage of the vendor, you just run strings command on the Linux, you will figure out the build directories are mostly Windows operating system.
[37:42]
Viktor Petersson
Build directories Terrence, many reasons.
[37:45]
Philipp Deppenwiese
Yeah, the reason is like this ODM, or let's say hardware business is highly driven by Microsoft operating system.
[37:52]
Philipp Deppenwiese
And so it's just how it is.
[37:55]
Philipp Deppenwiese
They are not Linux people.
[37:57]
Philipp Deppenwiese
And this is the main reason why all this stuff is also not well integrated.
[38:00]
Philipp Deppenwiese
You cannot do good pipelining stuff under Microsoft Windows operating system.
[38:05]
Philipp Deppenwiese
There's no, this is why basically Microsoft Windows integrated Windows for Linux, right?
[38:10]
Philipp Deppenwiese
So basically like the Windows WSL environment for like working under Linux with Microsoft Windows, right?
[38:19]
Philipp Deppenwiese
And this is one reason.
[38:21]
Philipp Deppenwiese
And so this is a big problem.
[38:23]
Philipp Deppenwiese
And so they don't have this kind of OpSeC, they don't have this kind of automation.
[38:28]
Philipp Deppenwiese
And then they end up building manually releases and then ship them over some kind of distribution channel.
[38:33]
Philipp Deppenwiese
So that's the first thing, which is the big problem with these keys.
[38:37]
Philipp Deppenwiese
And the second thing is those people are not really skilled on security at all to understand what they embed are test keys and they shouldn't be used.
[38:44]
Philipp Deppenwiese
And the biggest problem was the test key was reused multiple times.
[38:48]
Philipp Deppenwiese
So it's not only like you generate, every time they got an SDK from AMI was a test key, basically they don't generate a new one.
[38:56]
Philipp Deppenwiese
They just ship the same test key for all devices, like for all SDK portions of.
[39:02]
Philipp Deppenwiese
And then you have like probably you have multiple different kind of, on some part you have different kind of SDK keys, but like 22 for example.
[39:11]
Philipp Deppenwiese
But with one you can already like own, let's say three form vendors and like 200 product lines or something like.
[39:19]
Philipp Deppenwiese
That's super crazy.
[39:20]
Philipp Deppenwiese
And this key basically, so for people who don't know what it is, it's basically the PK key, which is the highest key in the secure boot hierarchy.
[39:29]
Philipp Deppenwiese
That means there's UFI secure boot.
[39:31]
Philipp Deppenwiese
Normally this is used to protect your bootloader.
[39:34]
Philipp Deppenwiese
And there's kind of key hierarchy for basically doing security stuff is super complex.
[39:41]
Philipp Deppenwiese
So you have the highest level keys.
[39:43]
Philipp Deppenwiese
A PK, this was basically owned.
[39:45]
Philipp Deppenwiese
Then there can be multiple kecks, basically keys.
[39:51]
Philipp Deppenwiese
And then you have a whitelist database.
[39:54]
Philipp Deppenwiese
And a blacklist database is for blacklisting, for example hashes of bootloaders or certificates.
[39:58]
Philipp Deppenwiese
We have been compromised.
[40:00]
Philipp Deppenwiese
And you also have a whitelist basically allowing things.
[40:02]
Philipp Deppenwiese
And if you have this PK, you can change everything.
[40:05]
Philipp Deppenwiese
You can just generate a new keg under it.
[40:08]
Philipp Deppenwiese
You can do whatever you want.
[40:09]
Philipp Deppenwiese
You have the full basically options to do.
[40:12]
Philipp Deppenwiese
You can block people supplying bootloaders.
[40:14]
Philipp Deppenwiese
For example, you could say the Microsoft bootloader is forbidden, right?
[40:18]
Philipp Deppenwiese
So you can also not fix it.
[40:19]
Philipp Deppenwiese
And only my malicious bootloader, who looks like a Microsoft bootloader works then, right.
[40:24]
Philipp Deppenwiese
Kind of strange.
[40:25]
Philipp Deppenwiese
And then this has already been broken with that.
[40:28]
Philipp Deppenwiese
And so the important point here is you can just do that from the operating system side.
[40:34]
Philipp Deppenwiese
As soon you have root access to system and the system has this kind of vulnerability.
[40:39]
Philipp Deppenwiese
You can just get this type of private key from GitHub.
[40:43]
Philipp Deppenwiese
So it was leaked on GitHub on other probably multiple places.
[40:47]
Philipp Deppenwiese
So since it was used a lot of like by a lot of companies for sure, there have been leaks of these SDKs and you could just download this key and use it.
[40:57]
Philipp Deppenwiese
And then you can basically just sign another CAC integrated into the system.
[41:01]
Philipp Deppenwiese
The BIOS accept it and you have full, basically full features to do whatever you want on the system in terms of moot process security.
[41:10]
Viktor Petersson
But there was a caveat, I think they were all like encrypted, was they were encrypted with like a four numerical, four character numerical key.
[41:19]
Philipp Deppenwiese
It was protected by basically four, I think numbers or I don't remember what it was, but it was super easy to break.
[41:27]
Philipp Deppenwiese
But you don't even need that.
[41:28]
Philipp Deppenwiese
So I already saw this SDKs, I mean, I saw the SDKs already out there with a test key unencrypted.
[41:36]
Philipp Deppenwiese
So it's not like you need to really decrypt it.
[41:40]
Philipp Deppenwiese
Right.
[41:40]
Philipp Deppenwiese
So this is like a brute force it, because most of these vendors just put stuff everywhere.
[41:45]
Philipp Deppenwiese
And this is the main problem, I told you first is how do you handle keys, especially even if they are test keys, right?
[41:52]
Viktor Petersson
Yeah, yeah, that's the same.
[41:54]
Viktor Petersson
So that brings us to attestation.
[41:56]
Philipp Deppenwiese
Right?
[41:56]
Viktor Petersson
So because this is a complicated topic, attestation is not easy in any way.
[42:03]
Viktor Petersson
You deploy it, right.
[42:03]
Viktor Petersson
And it becomes, it's complicated in the cloud, but it's a lot more complicated when you're dealing with devices that you.
[42:12]
Viktor Petersson
Well, that might be a different country that when they were initially doing their initial bootstrapping, you don't have control over it.
[42:19]
Viktor Petersson
So, yeah, I mean, you have a background this, like what?
[42:22]
Viktor Petersson
Yeah, maybe like do a.
[42:25]
Viktor Petersson
Explain to me, like I'm five, what is attestation?
[42:28]
Viktor Petersson
How does that work in the world of firmware?
[42:31]
Philipp Deppenwiese
Yeah.
[42:31]
Philipp Deppenwiese
So it's not only word of firmware, you can pull it over the entire boot like boot up of a system, let's say.
[42:38]
Philipp Deppenwiese
So, but attestation is basically the idea that you can do local attestation and or remote attestation.
[42:46]
Philipp Deppenwiese
And local attestation means you can locally prove something and remote attestation, you can remotely prove something about the system.
[42:53]
Philipp Deppenwiese
Some assumption, let's say, and this assumption normally, like in the most practical way seen, it's what kind of code has been executed or data have been executed on the platform, because you can also read data from the code, pass it and execute to another stage.
[43:10]
Philipp Deppenwiese
And the interesting part is nowadays attestation is super easy to do thanks to the improvements of the TPM.
[43:17]
Philipp Deppenwiese
So there's a trusted platform module, for example, on the most devices where attestation is possible to do with, there's no confidential computing.
[43:25]
Philipp Deppenwiese
You probably heard the name of it.
[43:27]
Philipp Deppenwiese
It's like some kind of technology to run something in the confidential vm.
[43:31]
Philipp Deppenwiese
So nothing goes in, nothing goes out, and you can do computation on the cloud stack securely.
[43:38]
Philipp Deppenwiese
This uses attestation as well.
[43:39]
Philipp Deppenwiese
And for these use cases, basically it's like if you use this with a TPM, for example, you can check up your entire, let's say, what have been executed in the boot flow from the firmware level, from the BIOS level down to the operating system kernel.
[43:55]
Philipp Deppenwiese
And this is one of the most important technologies upcoming in the next years.
[44:00]
Philipp Deppenwiese
Nowadays it's not really often used, and this was also when we had started our startup that we try to have this kind of like technology back there and have it as part of the back end.
[44:13]
Philipp Deppenwiese
But it was like the most people don't understand how valuable it is, because nowadays, I give you an example, people just run things, right?
[44:20]
Philipp Deppenwiese
This is similar with firmware, they just run it, they don't look into it, they have no tools for transparency.
[44:25]
Philipp Deppenwiese
The same goes with when you're running operating system, if you're running Linux in some kind of data center, how do you know no one exchange the code on the Linux system, right?
[44:35]
Philipp Deppenwiese
How do you do that?
[44:35]
Viktor Petersson
Yeah, secure boot is still painful in the Linux world, right?
[44:40]
Viktor Petersson
It's somewhat better in the Windows world, but it's painful in the Linux world even to this day, except for maybe Ubuntu, but beyond red hat.
[44:49]
Viktor Petersson
But beyond that it's painful.
[44:51]
Philipp Deppenwiese
That's completely true.
[44:52]
Philipp Deppenwiese
But also keep in mind, secure Boot just tells you this one has been signed, but you don't know if it's a previous version or it's the newest version, right?
[45:01]
Philipp Deppenwiese
You don't know what kind of version has been executed, you just guess.
[45:04]
Philipp Deppenwiese
Like someone could for example, have an exploit for a specific version which has been signed as a part of secure boot.
[45:10]
Philipp Deppenwiese
And you want to forbid that, but this is hard to do.
[45:13]
Philipp Deppenwiese
Then you have to update, as you say it, like you have to update the blacklist to probably forbid this kind of piece to be loaded inside in this secure boot chain.
[45:22]
Philipp Deppenwiese
But it's also complicated because you cannot control it.
[45:24]
Philipp Deppenwiese
The full chain like secure boot is only the transition to the first bootloader, and then the other secure boot like, or similar secure boot or signature verification parts are done by the Linux world.
[45:35]
Philipp Deppenwiese
For example, if you use Linux on the system, and then you have to, like, it's really complicated to do that.
[45:39]
Philipp Deppenwiese
There's also like kind of shim bootloader who does it, and it's really, we looked into that a lot, and so I believe also it's super enforcing.
[45:48]
Philipp Deppenwiese
If you have systems which have high ability or need to be available, and someone wants really to make your life hard, they just like load something which is unsigned and then the system stops working, right?
[46:00]
Philipp Deppenwiese
With attestations.
[46:01]
Philipp Deppenwiese
This is not the case, the system always works, but you can prove if the system is in the state you expect or not.
[46:07]
Philipp Deppenwiese
And this has been, this goes in a different way.
[46:11]
Philipp Deppenwiese
We don't use signatures for that.
[46:13]
Philipp Deppenwiese
We use hashes.
[46:13]
Philipp Deppenwiese
So every part is hash before it's been executed.
[46:17]
Philipp Deppenwiese
And then the next part, before it executes the next thing is also going to hash it and puts all these hashes into the TPM.
[46:24]
Philipp Deppenwiese
And the TPM can only reset this hashes to the original value by basically restart of the system.
[46:30]
Philipp Deppenwiese
That means you have this kind, you collect just an append only lock with all these hashes get xored and basically combined together, so you cannot basically calculate them back.
[46:40]
Philipp Deppenwiese
And so you have basically a list of hashes which giving you the state of the system.
[46:45]
Philipp Deppenwiese
And this is attestation is used for.
[46:47]
Philipp Deppenwiese
And this is super useful for basically securing systems and integrity protecting them.
[46:53]
Philipp Deppenwiese
But most of these companies nowadays don't use it.
[46:55]
Philipp Deppenwiese
I know some companies make use of it, like Cisco use it in some areas.
[47:00]
Philipp Deppenwiese
There's other companies like in Germany have been using stuff like that.
[47:06]
Philipp Deppenwiese
And for sure, like people start to look more into attestation nowadays, especially with confidential computing, they're doing the same on that level.
[47:12]
Philipp Deppenwiese
Just is a VM.
[47:14]
Philipp Deppenwiese
They want to prove what's running in there, right?
[47:16]
Philipp Deppenwiese
Because they want to make sure nothing like, nothing wrong has been executed before you ship your credentials to this VM, whatever you do there, right?
[47:25]
Philipp Deppenwiese
And this is important part of it.
[47:28]
Philipp Deppenwiese
And so it also got lately integrated in systemd.
[47:31]
Philipp Deppenwiese
So if you want nowadays to encrypt your hard drive, for example, under Linux, and you use cryptid app and you want to do local attestation, for example, to bind all of this hard drive encryption to the firmware, to the bootloader and every component you utilize until basically the system, the init ramfs, spins up for decryption and asks you for the password.
[47:55]
Philipp Deppenwiese
You can do that.
[47:56]
Philipp Deppenwiese
It's now integrated.
[47:57]
Philipp Deppenwiese
Leonard did some great work there in that area.
[48:00]
Viktor Petersson
Interesting.
[48:00]
Philipp Deppenwiese
So if you wanted to try it out by yourself, I think it's still a bit experiment material, but it's going to be integrated in a lot of public distributions like Ubuntu is going to make use of that fedora.
[48:13]
Philipp Deppenwiese
So things are changing lately, but it took a while to adopt remote station localization.
[48:20]
Viktor Petersson
Yeah, just a side rant on that.
[48:23]
Viktor Petersson
I'm still, I find it oddly painful still that you can't use proper, like TPM backed disk encryption on most distros without like bending over backwards five times.
[48:35]
Philipp Deppenwiese
Yeah, I mean, it's still not easy.
[48:37]
Philipp Deppenwiese
I mean, remote itself, attestation is less impactful because it doesn't that you can integrate in theory if you do having company and you're doing some kind of nice product and you just want to secure your product.
[48:48]
Philipp Deppenwiese
You can use attestation quite well because it's remote one is easier than the local one.
[48:53]
Philipp Deppenwiese
Local one is enforcing through encryption, remote one is just verifying it.
[48:57]
Philipp Deppenwiese
So I think maybe there's also remote installation already on the roadmap of system D.
[49:03]
Philipp Deppenwiese
So maybe there will be an easier way to utilize that.
[49:06]
Philipp Deppenwiese
Not by using this encryption or like combining it with it, I'm not sure yet, but the future will definitely look more into this integrity protected launch of operating systems.
[49:18]
Philipp Deppenwiese
But what you need for that is reproducibles.
[49:21]
Philipp Deppenwiese
So most of the software companies should be built reproducible and there are reasons for that.
[49:26]
Philipp Deppenwiese
If you later want to compare really what you have been executed, because otherwise you have to deal with so many different, let's say artifacts.
[49:33]
Philipp Deppenwiese
You need to compare and see because it can be built differently and it doesn't work across all things, right.
[49:39]
Philipp Deppenwiese
If the build server is doing things wrong and stuff like that, then you might end up locking up your system and things like that.
[49:46]
Viktor Petersson
So yeah, I mean that's whole different topic.
[49:50]
Viktor Petersson
That is a very complicated topic at large to cover as well because secure builds and I mean GitHub is doing some amazing work around that to make that actually accessible for a lot of people.
[50:00]
Viktor Petersson
But doing sign builds and yeah, cryptographically sound is not a trivial feat to do.
[50:08]
Viktor Petersson
Let's talk a bit more about other interesting findings that you guys have been revealing.
[50:13]
Viktor Petersson
One of them is the lighty or lighthdpd vulnerabilities.
[50:18]
Viktor Petersson
Do you want to talk a bit about that finding?
[50:20]
Viktor Petersson
Because I thought that was the first time I've heard of Lite in a long time, but well, clearly still in use and has some serious impact.
[50:27]
Philipp Deppenwiese
I would love to, but unfortunately this one, I didn't look into it.
[50:31]
Philipp Deppenwiese
I mean I just started five weeks ago.
[50:33]
Philipp Deppenwiese
So to be honest, that one is.
[50:36]
Philipp Deppenwiese
That one is still like unknown.
[50:38]
Philipp Deppenwiese
I just checked up the recently ones, but I think from this, let's say the next episode, should Alex ask more about the details there because he's quite specialized on that.
[50:50]
Viktor Petersson
I'll make sure to dive in that.
[50:51]
Viktor Petersson
So then one more thing I would like to cover is I'm not sure you caught this, but apparently the root certificate for secure boot, probably not familiar with it.
[51:04]
Viktor Petersson
Microsoft are the one issuing like the root certificates for secure boot across any operating system.
[51:08]
Viktor Petersson
Right.
[51:09]
Viktor Petersson
And I'm not sure you caught this, but apparently the root keys that are signing off secure boot expires in like 2028 and that's gonna have an interesting bombshell for the entire world.
[51:24]
Viktor Petersson
Really.
[51:24]
Viktor Petersson
Like have you, did you catch that one at all or is this news.
[51:28]
Philipp Deppenwiese
I catch it already.
[51:29]
Philipp Deppenwiese
So the thing is like you need to know, in the firmware world there's no time.
[51:34]
Philipp Deppenwiese
This is the first thing you need to understand if you are in the UI five firmware world or whatever.
[51:38]
Philipp Deppenwiese
Most of these like embedded firmwares have no time.
[51:42]
Philipp Deppenwiese
And sure you can spin up network and I mean this is dangerous, right?
[51:47]
Philipp Deppenwiese
If you have this kind of super like highly, let's say high permission software piece, right?
[51:55]
Philipp Deppenwiese
And it can do everything and then giving it network capabilities is probably not the thing you want.
[52:00]
Philipp Deppenwiese
So NTP and stuff like, or secure NTP and stuff like that is not a great thing to do.
[52:06]
Philipp Deppenwiese
And then you getting network, like, let's say you're getting network time issues, so you don't really know what the time is.
[52:13]
Philipp Deppenwiese
You can sure store the time at the cmos or some kind of vram variable and sync it, but to be honest you cannot really believe it because anyone in the operating system can change it value and maybe of a skew implementation.
[52:26]
Philipp Deppenwiese
But this can also lead to glitches in the time.
[52:29]
Philipp Deppenwiese
So not always.
[52:30]
Philipp Deppenwiese
It's guaranteed that you have the correct time on the firmware level, so time doesn't matter so much there.
[52:36]
Philipp Deppenwiese
So even the certificate expires, you could still use it for a longer time because the verification of secure boot is not checking time fields inside the certificates.
[52:46]
Philipp Deppenwiese
There's no joke.
[52:47]
Philipp Deppenwiese
So you can basically change, as far as I know, scary.
[52:52]
Philipp Deppenwiese
Yeah, but, yeah that's scary.
[52:54]
Philipp Deppenwiese
But the thing is like if you don't have time to use to verify against it a secure time source, how do you do that?
[53:03]
Philipp Deppenwiese
Maybe some firmware implementations do that and they have some kind of figure out some way and try to single whatsoever.
[53:10]
Philipp Deppenwiese
But to be honest, using time and certificates is like certificate in general inside embedded firmware was not a great idea.
[53:17]
Philipp Deppenwiese
Probably just use keys and you can never basically invalidate these kind of keys, except when you exchange entire PKI on the side of secure boot.
[53:29]
Philipp Deppenwiese
But this is something you should always do.
[53:31]
Philipp Deppenwiese
So if you listen now to me, always try to customize your secure boot.
[53:36]
Philipp Deppenwiese
Never use the predefined keys in there because you don't know what kind of keys are in there.
[53:41]
Philipp Deppenwiese
We already figured out, for example, there are keys from odms and oems you never heard the name of, and they putting all their stuff in there for some kind of product they sell or whatsoever.
[53:52]
Philipp Deppenwiese
So whenever this kind of keys leaked, it can be used basically for malware, right?
[53:56]
Philipp Deppenwiese
So you should, the first thing you should do, get rid of the factory keys and replace it with your own myriad.
[54:03]
Viktor Petersson
But then that, I guess that's very scary for a lot of vendors because now you're on the hook for maintaining keys and unless you really know what you're doing, you might cost yourself a world of pain, right?
[54:16]
Viktor Petersson
Because PKI management is nothing trivial.
[54:19]
Philipp Deppenwiese
It's not trivial, but there's an easy way.
[54:21]
Philipp Deppenwiese
So if you want to have an easy way to do that for your Linux operating system, I can give you some recommendation.
[54:27]
Philipp Deppenwiese
There's SBCTL from Martin Lindgrund, he's one of the arch maintainers.
[54:32]
Philipp Deppenwiese
This tool is basically super easy to use to maintain secure boot on your system.
[54:37]
Philipp Deppenwiese
You just need to log into your BIOS, set a BIOS password, say factory, reset the keys, disable secure boot, and then you start in the system and use this tool for provisioning and rekeying is super easy.
[54:49]
Philipp Deppenwiese
You can exchange your keys on a daily basis if you want, or monthly or yearly, whatever.
[54:54]
Philipp Deppenwiese
But this tool makes secure root provision so easy.
[54:57]
Philipp Deppenwiese
I can really recommend it and it's available on GitHub.
[55:01]
Philipp Deppenwiese
You can just download it or just as part of your distribution might be already packaged.
[55:05]
Viktor Petersson
I meant more for manufacturing, more so than for your laptop at home.
[55:09]
Viktor Petersson
That's when it gets tricky.
[55:10]
Philipp Deppenwiese
Okay, so yeah, I mean that's probably the different story, but even there you could use some kind of like really like bind to.
[55:18]
Philipp Deppenwiese
For example, if you build a product right, you also want to make sure this kind of product is safe and sane.
[55:25]
Philipp Deppenwiese
And so maybe you have some kind of remote management inside this product.
[55:29]
Philipp Deppenwiese
So what you could do, you can customize your like with all keys and also do raking stuff.
[55:33]
Philipp Deppenwiese
For example for Windows operating systems for sure is the problem because they require their keys to be there.
[55:39]
Philipp Deppenwiese
But what you can do is also you can use your custom key to sign the Windows bootloader.
[55:44]
Philipp Deppenwiese
So be smart with it.
[55:46]
Philipp Deppenwiese
So it's also possible to do that.
[55:48]
Philipp Deppenwiese
So what you can do is then basically like build up your own secure boot chain over the Windows bootloader and everything from there.
[55:55]
Philipp Deppenwiese
You might still need to include some certificate from Microsoft because of other reasons.
[56:01]
Philipp Deppenwiese
That's not super nice, but at least it reduces the attack surface.
[56:07]
Philipp Deppenwiese
Basically crazy because these vendor certificates, what the vendors ship on the PK and the CAC, you should check that out.
[56:15]
Philipp Deppenwiese
There are tools reading out all these keys or showing them basically in your Linux system.
[56:19]
Philipp Deppenwiese
We'll figure out they have tons of other certificates in there and it's not sure what is used for.
[56:25]
Philipp Deppenwiese
Sometimes it's used for drivers, sometimes use for extra hardware, sometimes it's used for their own products they want to ship on your systems.
[56:32]
Philipp Deppenwiese
So I can always recommend to do that, even if it's not so easy to customize your secure boot, but better to be safe.
[56:41]
Viktor Petersson
That's a good catch up.
[56:43]
Viktor Petersson
I would definitely have to take a look at that in detail and make sure what our device will shipping with any firmware, any keys that we are not familiar with.
[56:51]
Viktor Petersson
I don't think so, but let's double check that.
[56:54]
Viktor Petersson
That's a good thing to take a look at for sure.
[56:58]
Viktor Petersson
I think we covered a lot of ground today and I think we've had great conversations.
[57:03]
Viktor Petersson
I'm looking forward to going further into this in the future episodes and maybe have you and Alex together on an episode in the future and just go further into all these craziness.
[57:13]
Viktor Petersson
But I think I am super excited what Binary has been doing over the last few years, and I'm super excited about keeping all these oems, odms and bias vendors on their toes to make sure we can raise the tide in terms of security for the hardware world because there is a lot of work to be done there.
[57:34]
Philipp Deppenwiese
Definitely.
[57:35]
Philipp Deppenwiese
So I mean, I would also love to be included in the next basically podcast, but I would say in general, if you're also interested to speak about that more, you should visit the open source firmware conference because a lot of people coming from big companies and big enterprises there, and probably also maybe you can find some other people doing a podcasts about it and see their view on the things.
[58:00]
Philipp Deppenwiese
So this ecosystem is similar to the baseband ecosystem, you know, baseband also super, let's say niche, closed source, NDA driven ecosystem and also super proprietary.
[58:12]
Philipp Deppenwiese
And the firmware level is basically the same, less worse probably, but because we fight for a lot of things there and we made it, but it's still like a lot of things to do.
[58:23]
Philipp Deppenwiese
And I think we can learn from each other what kind of views we have and how we can improve on that, because it's not always so easy.
[58:29]
Philipp Deppenwiese
And I said I think a lot of issues there is person like people culture in that way that these people are not really well educated enough, not because they can be really great in terms of hardware and fixing problems there and whatever, but they're not well educated in terms of like what is currently the world going into one kind of direction, right?
[58:49]
Philipp Deppenwiese
Even Microsoft had to admit that Linux is a major operating system for a lot of devices.
[58:55]
Philipp Deppenwiese
They started to use it on Azure.
[58:57]
Philipp Deppenwiese
So I was at all systems go last year with Leonard Pottering and all these folks and they told me like the biggest part of Azure is running on Lambda Linux.
[59:06]
Philipp Deppenwiese
There's no Windows operating like some stuff running still on our Windows.
[59:10]
Philipp Deppenwiese
But even Microsoft is endorsing Linux a lot on system and I think we need not only Linux is important I think, but the open source ecosystem which comes with it will help us to improve a lot and also it will help us to be more flexible, customize things and find security vulnerabilities and even like have responsible disclosure much easier for sure.
[59:32]
Philipp Deppenwiese
It also brings risk as you could see with the exact fail issue right where someone tried to embed some kind of backdoor into it and we just figured it out by accident.
[59:42]
Philipp Deppenwiese
But let's say there's a lot of potential and I think it's always better to work together with a lot of companies and with a big ecosystem instead of doing your own thing.
[59:52]
Philipp Deppenwiese
And this is what you can see in the automotive sector.
[59:55]
Philipp Deppenwiese
Automotive is trying to move away from their closed source ecosystem and try to go more into mainline with Linux operating systems.
[01:00:03]
Viktor Petersson
Yeah, no, and I think that's transparency and responsible development is a good note to end on.
[01:00:10]
Viktor Petersson
So thank you so much for coming on the show Philip, and talk to you soon.
[01:00:15]
Philipp Deppenwiese
Cheers.