Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Inside the Yocto Project's Evolving Tooling: SBOMs, SPDX 3.0, and Secure Embedded Systems
Play On
07 MAY • 2025
50 mins
Share:
In this deep-dive conversation, Viktor sits down with Joshua Watt (Garmin) and Ross Burton (ARM) to explore how the Yocto Project is redefining how we build, secure, and maintain embedded Linux systems at scale.
Drawing on their years of experience contributing to Yocto’s technical steering and core infrastructure, Joshua and Ross walk through why build-time Software Bill of Materials (SBOMs) offer unmatched reliability, how SPDX 3.0 enables deeper license and supply chain visibility, and what it really takes to keep embedded products secure across decade-long lifecycles.
We get into the practical and architectural challenges behind modern embedded development:
- Why shipping Raspberry Pi OS at scale is a ticking time bomb (and what to do instead)
- What Yocto actually is, and why it’s not just another Linux distribution
- How build-time SBOMs outperform post-hoc scanners for audit and incident response
- What VEX metadata is, and how it improves real-world vulnerability triage
- How SPDX 3.0 enables granular license tracking and nested component relationships
- Why OTA updates should be a design-time decision, not an afterthought
Joshua also shares what it took to build Yocto’s native SBOM tooling (create-spdx), and how the project is adapting to new pressures from regulation, such as the Cyber Resilience Act (CRA), and increasingly security-aware supply chains.
We also cover:
- Managing SBOMs across multi-layer BSP stacks and opaque vendor binaries
- Using BitBake’s hashserver to enforce reproducible, compliant builds
- The philosophical shift SBOMs are triggering in the embedded world
- What’s next for SPDX, Yocto security tooling, and vulnerability automation
The episode wraps with advice for embedded developers trying to get started with SBOMs in Yocto, from common misconceptions to integration tips and where to plug into the community.
Whether you’re a firmware engineer, security lead, or building connected products that need to survive for 10+ years, this episode is packed with battle-tested insight from engineers shaping the tools behind secure embedded Linux.
Transcript
Show/Hide Transcript
All right, so the reason why I wanted to do an episode on Yocto was that in the last five or so years, we've seen a lot of accidental IoT engineers, people who kind of prototype something on the Raspberry PI and they managed to get some success.
Maybe they had a Kickstarter that really took off.
But what a lot of these people don't know is how you manage devices at scale.
If you just take Raspbian or Raspberry PI os, package that, and ship that out to your customers, you're down for a world of pain.
Trust me, I've been down there with screenly.
It's not pretty.
You end up writing a lot of logic and we ended up writing a ton of logic for trying to do device management properly.
Eventually we moved to Ubuntu Core.
Yocto is a very good alternative to that.
What Yocto gives you is a framework for building disk images, but then it has a wide ecosystem that can take care of device updates using things like a B partitioning scheme, which is considered best practice.
So it has a really good, mature ecosystem for using how to build and scale the IoT devices.
So this is why I wanted to do Episode of yocto, to save a lot of people from time and agony and what we wasted on trying to solve this ourselves.
So if you're looking to do some kind of project, look at Yocto.
If you want a hosted version of this, there are plenty of companies that provide services around yocto.
Balena is one of them, which I've had an app on the episode before on.
So do take a look at that and onto the show.
Welcome back to another episode of Nerding up with Victor.
Today I'm joined by Josh and Joshua and Ross and we got to do a talk all about yocto.
So I guess if we can do a quick round of introduction to maybe start.
Joshua, if you want to start giving listeners kind of context before we dive into the deep world of yocto, about who you are backstory and how you got onto yocto.
Maybe.
Yeah.
So my name is Joshua Watt and I'm a software engineer for Garmin, primarily working on embedded electronics for boats.
And I've been involved in Yocto since about 2016 and I've been on the OpenEmbedded Technical Steering Committee and the OCTO Technical Steering Committee along the way.
And yeah, so that's my background with the Octo project.
Amazing.
Thank you.
Joshua and Ross, you have wealth experience in this world as well.
Yeah.
So I'm Ross Burton, I'm a software engineer at ARM.
I'm in the Distros team.
So we work making sure that ARM stuff works on distributions.
So I'm one of the Yocto guys.
I've been working on Yocto for off and on 20 years now, which is kind of terrifying.
I just looked on my first commitment, it was in 2005.
There were some diversions along the way, but yeah, I've got like commit number 11 or something in Pocketree.
Impressive, Impressive.
I want to start the conversation off with like giving perhaps people who are not even familiar with yocto.
If you, if you lived in embedded or in, in the IOT world, I'm sure you've at least heard the name Yocto and come across it.
But for the people who are unfamiliar with Yocto, what is Yocto like, what does it solve for and why does it exist in the first place?
I'll leave that up to you to want to pick that to start it off.
Well, I suppose the usual answer is Yocto is not a Linux distribution.
It lets you make your own Linux distribution so very useful for embedded or automotive or industrial situations where, you know, Red Hat may not be suitable.
So you can take Yocto and customize it and it's designed to be super flexible so you can trim it down and make exactly what you need to do.
Yeah, it's interesting.
It's definitely got a lot of interesting features which a lot of mainstream distributions don't have specifically for people that are productizing whatever they're making at the end of the day.
Right.
So just to recap that it's rarely used for the context of.
I got my desktop, I want to run Linux on it.
It's, it's more for.
I have some kind of PCB with some kind of architecture on and I want to create a custom distribution of Linux that runs my software essentially.
Yeah, yeah, you see it a lot in like consumer electronics or like deeply embedded things like I'm sure Garmin Josh can talk about some of the things that they make like our TV runs, yocto printers, soundbars.
But also you find it in like infrastructure.
So you find a lot of telecom backends would be running on YOKTO hardware or like big data centers might have VMCs which are built with Yocto.
So yeah, the full stream, you know, micro, all the way up to, you know, data center scale.
But we have, I guess there is a.
It's not really for super low end devices or low resource devices.
Slightly more.
It's not for single board or.
Sorry, single board is not the right word.
It's not for like the lower end of the spectrum where you have like RTOs and that.
So it's a tier up from that where you're running.
So x86 arm and so on is largely where.
Or I guess RISC V these days is where a lot of the boards are running on.
Right?
Yeah.
And PowerPC and MIPS.
All right, yes.
We still have people using that hardware because once you ship something, like a lot of the deeply embedded vendors, they may make a product 15 years ago, they still support it.
So they're still using yocto and they're still shipping this hardware, which a lot of people may think is.
To a lot of us it's dead hardware.
But it's got a lifespan of two decades.
In particular, if you talk about more specialized hardware or special appliances, I guess they have much longer life cycles than your consumer hardware.
Of course.
Yes, that brings interesting problems.
Yes, interesting.
So I guess let's unpack some terminology in the Yoko world.
At least for me when I first got involved.
I mean I.
I've.
I've touched YOCTO quite a few times over the last like 10 years, but I never really got super deep down the rabbit hole.
But I've built images using YOCTO a few times and I had to definitely.
I wrote a blog post a few weeks ago that kind of caught up and about how to use SBoM, which is why I got re engaged in the YOCTO world.
And that's how I got in touch with Yoshu, actually about.
Because he was helping us in one of our CISA working groups about produce, like how we actually do S boms in that world.
But when you start exploring YOCTO for the very first time, it's a pretty steep learning curve.
I don't think it's a.
I think it'll be a lie to say that it's just plug and play and you just got to figure out in 15 minutes.
There are a lot of things to figure out.
I mean, where should we even start?
Like maybe like open embedded pokey bsps recipes?
I mean there are.
Yeah.
Like these are not intuitive concepts, I guess until you get pretty deep down the rabbit hole.
How do you guys best explain the fundamental core concept, if that makes sense in yocto?
All right, so we can skip over the entire YOCTO versus open embedded thing because that's partly historical.
And most people don't use that.
You can use the terms interchangeably, even though they do identify entirely different things.
Okay, so BSP is a board support package, which is a.
I consider it a historical term.
It doesn't really make any sense these days, but.
But it's the configuration and the port for a particular target device, be it a KEMU machine running on your laptop or a particular piece of hardware.
On my desk I've got a TI Beagle Play.
So there's a BSP that targets that, then there's distros.
So we have poky, which is the reference distribution, which is an example, which you should not use in production.
It turns everything on for testing purposes, including stuff like you can log in without a password, which is useful when you're testing stuff, but less useful in shipping hardware.
So it's a good starting point, but it should be customized at least, or ideally rewritten to match your use case.
Right.
What other main concepts are there?
Individual recipes.
So there's a recipe for the kernel and a recipe for the compilers, and your applications may be built out of multiple recipes.
They build packages which are then combined to make an image, which is a thing you normally put on your target.
Right.
So that would be a disk image or something.
And I think it's important to note here that everything is built from source, so you're not pulling in packages.
When you say compiler, you basically mean we're building everything from scratch, like fully.
Well, I guess you are bootstrapping using the compiler on the host, so you're not building the compiler for the compiler, but you're basically starting with everything from source and building everything from nothing, essentially.
Build the cross compiler for you and then use it to build all the stuff that goes on the target.
So the host dependencies that are required to get started are pretty minimal.
It's, you know, gcc, binutils, a couple other utilities, but you don't need any cross compilers or anything installed because we're going to build those for you.
Right.
And I guess it goes without saying, starting from scratch out of cache takes quite a while.
In particular if you're cross compiling from a different one architecture, another architecture.
Right.
But even within singular architecture, it takes quite a while to build everything from scratch.
Right?
So poki, or poki, whatever, how you guys pronounce it, Pokeypoki is the reference.
If you look at documentations online, I think a lot of people would just use and ship that for a lot of distributions.
I guess one of the things I'm curious About there is kind of like horror stories.
What you guys have seen in the world of just like, I'm just gonna repackage this and zip it out on a device completely insecure.
Like, why have you guys seen.
I'm sure you must have seen some terrifying things in that domain.
I don't know if I've seen anything like personally, but I have heard of people mentioning that they still ship with Paki and it's like, well, that's terrifying, right?
You know, I don't know if I've actually seen anything directly though.
There's a fairly regular presenter at one of the yogto conferences called Marta who does security work and she has some horror slides of like unnamed devices, but they accidentally left on.
But you can log in without a password.
Oh, wow.
I was.
The customization thing is kind of very important.
I was actually looking through a license manifest for an unnamed vendor this morning trying to see does this thing, I think, use yocto.
Use yocto.
So the usual trick is find the GPL compliance document and look for particular things like, oh, that's one of our package names.
So yes, this thing runs yocto going down it.
Now, this is a data center infrastructure piece and it's shipping a 4x server and they're looking through the list.
It's like your.
Your security vulnerabilities are magnified because you're shipping everything.
Instead of saying, this is a tiny piece of computer that doesn't need to have a display, let's not ship an X server.
So you do see silly things like that occasionally.
But yeah, the entire oops, I accidentally used Poky as my production device is something which you've been aware people do and shouldn't be doing.
So the latest release puts up a very big message every time you log in saying, this is for testing and development purposes.
Do not ship me.
Right.
So hopefully we won't ever see that in production.
That.
That's okay.
Fair enough.
So when people get started with yocto, what are the, some most common, I guess, misconceptions in.
In like using yocto and like, what are the biggest maybe mental hurdles to get like your head around yoc'd as a concept?
I think one of the things that tends to be confusing early on is the relationship between recipes and packages and your final root file system, which is that like a recipe's purpose is to pull in your source code and compile it into packages.
Like, and these are.
We've got a bunch of different packaging formats, but it is like the same concept as like a DEB or an RPM or whatever type of package, binary package that then gets installed into your root file system.
I think that can be very confusing at first because it's kind of.
It feels like it should be more direct.
Like you take a recipe and that's what ends up on your root file system.
But there's actually an intermediate step of packaging there and not necessarily all the packages that a recipe produces are going to end up on your root file system.
So I think that can be confusing early on.
Yeah, definitely.
You see a lot of people confused about how they built a recipe and it doesn't appear on their target because you just built it, you didn't put it in the image.
Or conversely trying to figure out how to make things smaller.
It's like just stop installing the things and it won't be there.
It does seem a bit complicated.
Yeah.
Yeah.
I guess one of the things that I found confusing was, I mean, Joshua, you kind of alluded to it that there are different packagings.
Like in the config file you can say like, oh, I want rpm, I want DEB files, or there are some.
Few other ones.
I think Alpine packaging is support as well.
I think some other ones.
Right.
I by that kind of inferred that it becomes like Red Hat E or Debbie and E.
But it's completely unrelated that it's purely just how do I get this package into my system more so than anything inferred into the actual system?
I think that was for me at least conceptually confusing.
Yeah.
And I don't know the history of why we have all the different packaging formats, but we do.
And there's definitely been talk about adding more like APKs, for example.
There's been some discussion on the mailing list about that, but yeah, that can be confusing because it's like I'm producing an rpm.
Why can't I just point it at the Fedora RPM repos and install stuff?
And that's not the intent.
They are RPMs, but they are in no way compatible with anyone else's RPMs.
That's the format that you want to distribute your packages in.
And that's fine.
Yeah, for me, I incorrectly assume that you're using like that is basically that base image you're using, essentially derived based on the packaging format to use different base images, but completely has nothing to do with that.
So that at least threw me off quite significantly when I started to play with it.
So you've now built your YOCTO image.
You managed to kind of create something that runs.
Right.
Let's assume you have not used Pokey and you have actually tighten the ship a bit and you actually use it on its production grade.
The next thing that I seen a lot of, I guess issues around YOCTO is around ota.
Right?
Because once you ship a device into production, great, but that's just the start of the journey.
That's not the final piece.
Right.
You need to like, if there is a CVE out there, you need to be able to say, oh, I'm going to make sure that these devices are updated for it.
Particularly going back to the previous point about oh, this might be a 15 year life cycle on this device.
Right.
Like how do I update these devices in the wild or at least provide a mechanism for that can be done manually over that duration.
So let's switch topic to like ota, maybe not even ota, but like updating mechanisms a little bit in yocto.
And how does that work?
Well, go ahead.
Yeah, so basically this is another one of the places where YOCTO says this is a complicated topic.
There are lots of variables, you decide, right?
So by default we produce packages and you can run a feed server and just run APT update if you want.
But in, you know, food like your random laptop, that's fine.
But for something you've deployed to a million houses less ideal understatement.
Yeah.
So there's a number of options for proper battle tested updaters and I can think of like four.
There's Menda, that's the OG one, right?
Yeah.
SW Update.
Rauk.
There's at least two more.
I can't remember the names off the top of my head and they all are fairly solidly integrated into yocto.
A lot of them have companies behind them doing like commercial support.
So it basically is a choice of sitting down and saying what features do I actually want in my updater?
And then finding the one that fits because they do have different feature sets.
Yeah, yeah, It's a tricky problem, but it needs to be approached carefully because yeah, you don't want to actually brick a fleet of devices, nor do you.
Want to ignore that problem and try to kind of shoehorn that into a product after go live either.
Right?
Yeah, it's something which several years ago were thinking, do we need to actually provide integrated into the core an OTA updater?
But it didn't take a huge amount of research to realize this is a.
It's a very broad problem and I don't know if it's feasible to say there is one answer, because that's fair enough.
And I think it does succinctly highlight one of the great advantages of the project, which is that there are all these third party layers that do this very well and they're very easy to integrate or relatively easy to integrate.
And then you have the choice to use what you, what works best for your product without, you know, too much hassle or, you know, people are able to support these things independently, you know, without much hassle.
So that's a great strength that we have with that.
Yeah, being agnostic is definitely a strength in some ways, I would say.
But I think at least if you're old school, you've been around embedded for a long time, OTA and these things, they are kind of like table stakes.
Like we understand that's like a hardware, like a hard requirement before you even like put together a spec for a project, but at least over like, I guess with the rise of the Raspberry PI in particular, like you have a lot of people like or became like accidental IOT people, if that makes sense.
And I'm one of them.
Right.
With our, with screenly, like we would become like an accidental IoT project.
And, and much to your point earlier, Ross, about trying to do DEB package updates in the wild, this is the path we took on Raspbian back in the days.
Try to script that and do that at scale and it's a world of pain.
The amount of fallback and logic you need to do to wrap around that is just insane, particularly when you factor in that the power might be cut any moment because these are devices that you have no control over.
It's not a data center.
Right.
So I guess highlighting the need for OTA is something that would save a lot of people a lot of pain if it's really properly, I guess, instructed up front, really that, oh, do not even contemplate, go live without a strategy for ota.
And, and I guess that's kind of a natural segue over to one of the other things I want to talk about, which is supply chain security and security at large in the embedded world.
And I mean with the rise of CRA particular in Europe, it's no longer an option to think about how does the supply chain life cycle for these devices look like it's called like it can't be an afterthought, like a gun.
Like your OTA needs to be the part of your day zero strategy.
Right.
And I'm Curious about what you guys have seen from the community with regards to CRA or ee.
Cyber Resilience act for short.
Really, for those who's not familiar with it, what how that has kind of shaped how people see yocto and how people use YOCTO differently.
Like, I'm really curious about how obviously you guys have seen that.
Well, I think people have been more interested in their supply chain from the octo side of things.
I think one of another thing that's always been really good is we've always had a really good supply chain just because of the way the project works.
We start with a very minimal set of dependencies.
The recipes are pretty exhaustive in knowing how to build them and having all the metadata information about all the stuff being built.
And then we use that to build your cross compilers, like the tools you need to build the stuff you need to build and then we follow that through all the way to the final image that you're generating is also described that same way with that same level of detail.
And so intrinsically we've always had a pretty strong supply chain.
And so I think a lot of people that I've seen that are using the project as far as like CRA and stuff, they're not too worried about being able to justify their supply chain in way more detail than the CRA is asking for anyway.
So that hasn't been too much of a concern.
It's just been like, how do we get that information out of the project and into something that's digestible by, you know, people, machines or something like that.
Yeah, you definitely have a strength by doing everything of source because there is, you have very few things you need to trust right outside of that, which is great.
But I guess that really again highlights the need for OTA and being able to like, if you like now you know what version you're running and what license has.
That's, that's a fantastic starting point, but it's the start of a journey, not the end goal.
Right.
so on that, I mean, kind of jumping a bit here, but like we talked about building a bit.
And how far is YOCTO to fully deterministic builds today?
Like is it possible to do four deterministic builds on the system today?
I think for, if you're talking, I mean the thing to keep in mind is when there's lots of different layers, right, with varying levels of, you know, third party layers we don't necessarily have as big a control over.
So it does depend on what software you bring in.
Yeah, but as far as like the core stuff, I believe we're fully reproducible or at least try to be fully reproducible as much as we can.
There sometimes are points in time where we're not just for reasons.
Some, some package upgrade did something and, but usually we're working on fixing it.
And we do track that.
There's a, a page on the website that tracks the reproducibility status of the core layers and that's supposed to be binary reproducible.
So you build it today, you know, you get a binary output, you build it in two years.
As long as you start from the same place, you should get the same binary output.
And kind of the way that's structured then in order to sort of allow end users to cover third party stuff, is the way that those tests are written is designed such that you can run them against your own stuff and not just core.
So what I always tell people is like, we've made sure that core can be reproducible, but if you care about reproducibility, you need to be running that test against your stuff and make sure that your stuff is reproducible because we can't guarantee your thing is reproducible, but we can't guarantee that the core stuff is.
Of course that's kind of how that works.
As far as reproducible builds goes.
Okay, that's very cool because obviously that's a big upside with regards to supply chain security as well, like infinity appreciable build.
So that's a big one.
Let's dive into kind of like the second big meaty piece I wanted to talk about in today's call, which is basically how Joshua and I first met, which is around SBOMs really.
And Joshua, you have written much if not most of the SBOM parsing inside of yocto.
So maybe let's start before we dive into the nitty gritty details of why and how this done, I guess we have the philosophical debate of SPDX versus CycloneDx.
How did you guys land on SPDX?
What was the debate internally?
I'm curious.
Well, there is the Linux foundation connection with spdx, which factored into it fairly heavily.
And I mean that's most of it.
Like I looked at both of them and SPDX kind of made sense.
And then the other thing that was really helpful was we knew that.
So when we started SPDX was on version 2.2, but we knew that they were working on 3.0 and so the plan from the beginning was always we're going to do SPDX 2.2 and we know our supply chain is very complicated and there's a good chance that we're going to want some more things added to the SPDX standard.
And so being able to do SPDX 2.2 and then be involved in the design and the development of SPDX3 was extremely beneficial because were able to help guide SPDX3 to really understand deep build supply chains, which is what we have.
And then also were able to help make sure that whatever came out of SPDX3 would align with what we wanted to say, you know, about our SBoM.
So that worked out really well just for those reasons.
So, yeah.
And what's the status on SPDx3 support in Yocto at the moment?
Yeah, so SPDx3 was added in, let me see here, the 5.1 release, just codename Styhead, and that was released in October 2024 and it's the default now.
So the next LTS release of Yocto will be SPDx3 by default.
You can still use SPDx2, you can turn it on.
But yeah, SPDx3 describes our system much better than SPDx2 does.
And so we very much prefer that people use that just because of how much better it works for our use cases, because were involved so early in the design process of it.
Right.
The thing that I thought was really fascinating when we kind of did a deep dive in how you build SBOMs in the first place, it's like, it's very different than how you build sboms normally, by virtue of you being kind of like a package manager or package builder, I guess is a better phrase for it.
So maybe let's take a path down that road and just talk about how do you build sboms and like, what allows you to build sboms, I guess in a far more complete way than almost anybody else I've seen out there, because you can make assumptions that almost nobody else can make.
Right.
So yeah, take a deep dive down path down that road.
So because we're building everything from source, you know, our recipe files and the build system they have to know everything about how to build it from source.
You need to know the build time dependencies, you need to know the runtime dependencies of things, of the packages that you're producing.
And that needs to go all the way back to the cross compiler.
We've always been very good about tracking license information and things like that, because that's something our users care about, have cared about since before we had SBoMS.
And so the breadth of information that we have is very deep.
We can check some, all the source files that went into a build, because one of the other things that we've just pretty much always done is we've had very good build hygiene, making sure that every recipe that we're building is in its own little build silo, and so it's not leaking out and pulling in random stuff that it shouldn't be from other places as best we can.
We try really hard to do this, and so we have a very complete picture of what it takes to build stuff.
And so the SBOM support is primarily an exercise in taking the information we already know and putting it into a different file format.
It's more complicated than that.
I don't want to trivialize it, but it's not a ton of Python code.
Right, right.
It's not that bad.
So because we already started from such a good place, it made it pretty easy for us to end up with these very high quality and very complete SBOMs.
As I say a lot, it's quite easy to have an sbom that's 10 times the size of your final root file system.
If you turn on every single feature, like I want to check some every file and every build and every file that ends up on the root file system, you can get a very large SBOM order of magnitude larger than the actual binary you produced at the end of the day.
So very complete, if that's what you're going for.
We do have knobs to adjust the levels because not everyone wants that.
Right.
And it's can be a lot.
And how do.
Yeah, so you mentioned you're capturing hashes, you're capturing build dependencies, runtime dependencies.
Like that graph is very complex.
Right.
So like, I guess, is there anything missing at all?
Or like, do you literally capture everything that you capture in build time in the SBoM, more or less from your artifact?
It's pretty complete.
I'm trying to think if there's anything in particular that's missing.
I can't think of anything off the top of my head that we have.
So with our SBOM support, one of the choices that we made was we do what's called, or what I like to call, like we capture like first principles, things, which is stuff that we are authoritative on because we're the ones doing the build.
So we try to avoid heuristics and things like that.
Like, you might see another, like other SBoM tools might do like a scan of a binary and say this is probably in there and that's fine, there's certainly nothing wrong with that.
But just because of where we are at in the supply chain, we kind of just take this first principles approach or reporting on the things that we know about because it's in the recipe or whatever, like we did.
We know.
And then you can layer on top of that scanning tools that do other stuff.
Right.
But it's not really worth it for us to maintain a bunch of heuristic tools in a project.
And I can't think of anything.
It's generally, again, it's generally pretty easy to map the information we have from the recipe or the build system into spdx.
As long as the property here, as long as there is a way to express it in spdx, it's generally not that complicated to do.
It's just a matter of manipulating the data around and restructuring it into the right format.
And with regard to.
Yeah, one thing we can do, which I think a lot of people may miss or other systems can't do, is because we build all the software from scratch, we can pick up statically linked libraries as well.
So if your binary statically links to some compression library, it won't be obvious externally or in the package dependencies, but because the SVDX aggregation happens during the build, we can look at the debug symbols and go, oh look, it's got a statically linked copy of Zlib and that can go into the spdx, which will be hopefully very useful next time there's some tragic CVE involving a really low level library that has been statically linked everywhere.
Right, right.
And you also know the version because it's sandboxed inside of the build version.
So like you really know what it's statically linking as well.
Yeah.
The only thing that I guess is missing, like I'm just putting the working group with it cisaw hat on is it's like the supplier information.
Right.
Is there a way to infer that?
And if you are day job, can you implement that as part of the Okta toolchain, say this yocta image was built by Garmin or ARM or whatever and can you supply all those information as well as part of that build cycle to get to a full like augmented S POM as well?
Or is that outside of scope of the tooling?
Yeah, so if you're particularly specifically talking about the like, there's a field in SPDX called like supplied by.
Yeah, the top level.
Yep, yeah, yep.
So you can set that.
There's a variable, a bitbake variable you can set in your local conf usually is where you'd set it.
Right.
And you can set the supplier for your packages and it'll just show up in everything you build then.
Right.
So that's definitely possible.
You can also do.
If you're talking about like who did a build.
Right.
Which might be different than the supplier.
Right.
So you can capture that information too.
I believe it's off by default, but again you can turn it on and that's what I'm talking about.
With all the knobs that are in the stuff, there's a lot of options to turn on more output than are on by default.
The defaults tend to try to walk the line between.
The more stuff you turn on, the slower it's going to go, right?
Sure.
So they try to walk that line between build speed and then also the default that we have is to do reproducible S BOMs because there's definitely features you can turn on that will make them not reproducible, like precise build timestamps and stuff like that.
But again, that field exists, that supplied by field exists in spdx and we do have a way that you can set that.
I believe it's just off by default.
Yeah.
No, I guess what I'm hinting at is the NTIA minima element compliance element of it.
Right.
Because if you're supplying particular embedded hardware to the US government, for instance, you probably do need to meet these guidelines.
Right.
So that's where those pieces become more important.
I guess the next thing I kind of want to cover, which is on kind of tangent this, but very much related, which is the VEX side of things.
And I know Ross, you've been doing some work on the VEX side of things.
A little bit touching there.
So I guess and you both of it obviously are involved indirectly or directly with.
With the work around this.
So maybe how that's the whole cause of.
Well, maybe if you can explain to the listeners like what's a vex file?
And how does that come into play with SBoM?
Maybe we can start there.
So as we start with a little bit of history, in the previous releases we had a tool called CPUCheck and that would download the NVD database and just try and compare your recipes with the database and go, oh, here are these.
Some CVEs you should probably know about.
These have been fixed, fees have been.
These are still unmitigated visa invalid.
And that produced an Output in machine readable text and then JSON, which is fine, but it was all very bespoke YoC date Pacific tooling.
But there's lots of limitations there.
It's a one time report, right.
So yeah, it's a snapshot of the state of the CVEs at the time that you did the build, which is not very useful two months later.
Right.
So we've been looking at improving that situation for some time and there's tangential drama with the CVE community, which probably should gloss over today, the entire episode on its own.
But we are now, thanks to some epic work by Josh, we can dump the full set of CVEs that we know about into vspdx directly.
So we don't generate a separate openvex file.
Because vspdx is slowly becoming the one source of truth about your build.
It's not just your SBoM, it's also every package contains this file list and these files are of these sizes.
So you've got all the actual per file metadata.
And also now, and These are the CVEs that at the time of the build we knew about.
So we can say, oh, this issue, we fixed it with a patch.
So then in a year's time you can use the tool to process the CVE data that you have in the SVDX and then you do an up to date scan.
This is at the moment work in progress.
There are bits of tooling floating around.
There's a YOCTO VEX check tool I think it's called, which mostly works, but I don't think it scans the latest files that we've got.
So that still needs a bit of work.
But yeah, it's coming together and it's looks like in the next release we'll have all this all sorted out, which will improve dramatically.
I just want to run a CV scan across my software store.
Right.
And it's really tracking that over time.
Right.
Like SBOM is a great tool for that.
But then, yeah, it's a truth in time, not like a absolute truth forever.
Like I have zero cvs, right.
Or whatever they would be called in the future now that CVEs are semi gone or.
Yes, yeah, let's avoid that.
But so the idea then is you can generate this dump basically as an artifact of your work Joshua from the SBOMs.
And then what's the relationship with upstream source and YOCTO source?
Right, because you're pulling in upstream.
But I presume there are sets of patches involved.
You're not Just pulling master from upstream repo.
You're applying a set of patches for insert reason.
Right.
How does that look in particular from a vantage point of vulnerability, patching and like and even I guess backporting patches Because I presume that's something that will kind of fall on the OCTA project as well.
Right?
Not sure who of you feel closer to that cat of worm.
We do apply patches for sure and there's a.
The way the patches are applied, there's a tag you can put in the patch that says this fixes a CVE and then the build system will see that and report it in the CVE statuses for the things and then that ends up in the SPDX output.
So it's pretty easy from a mechanical perspective.
As long as the patch exists, it's pretty easy to apply that and get that to show up in your build output and then get marked as fixed basically.
Yeah, so that's kind of how that works.
But do you guys run situation where you have to backport as well or how does that.
Because I would imagine that's a scenario that is very real.
For particular if you're talking about life cycles of 15 years of a device.
What's the LTS window by the way, for Yocto?
Like what's the support window for an LTS release?
LTSs have a four year cycle.
Four year?
Yeah, so that's four years from the YOCTO project itself.
Obviously if you're producing hardware which you want to last 15 years, then either you need to maintain it yourself, but there are plenty of OSCs who will give you a supported YOCTO derivative for like 10 years or more.
Yeah, we do backport a lot patches.
As Josh said, if the stable releases tend to only get security fixes or like pretty serious bug fixes, so when patches do get taken back, they all need to have a little CPE tag in the patch identifying what CVEs were fixed.
Yeah, that just bubbles through.
Okay, okay, interesting.
One thing I really like about this whole approach in general with SBOMs and how you build things is that you.
I kind of hinted this before, but you're solving for the problem of unknowns, which is very real in the world of SBOMs because if you point any of these modern tools out there, like Sneak Trivia or any of these tools to generate SBoM, they're just basically, if you talk about an image, for instance, they basically is like reading the package database and then building an S SPOM based off that.
It's not very difficult to circumvent that by either modifying the database or just chucking a binary in there that is not even picked up by these toolings.
So that's one of the things I really enjoy with the approach for YOCTO and how you guys are doing that because it feels like one of the few tools out there that are like proper complete with regards to SBOM generation.
And I guess Zephyr is another project that is doing well with regards to that from a slightly different angle and maybe that warrants another episode of going down the rabbit hole of Zephyr, which is completely different.
Guys, this has been super interesting.
I think I've covered most of the things I want to cover in today's episode and I think there are a lot of really interesting nuggets in what you guys have done.
And for people who want to get started with yocto, people want to learn more, what do you guys recommend to avoid again, avoid the compet force and kind of get started the right way?
Well, I'd start by going to Yoktoproject.org also the documentation from there is good if you start with the Quick Start guide and don't click on like full manual because you will get blown away.
But the full manual is immense but the Quick Start guide is a few pages and that will get you started and just treat the rest of the full manual as a reference, not something you go to bed with to read.
Also we could say we're an old school community so if you want to chat with people then we're still on IRC but there's an active IRC channel on Libra.
Net it's Hashokto so yeah, come there have a chat if you've got any questions.
We're friendly, very cool.
And what's next on improving the security posture or in YOCTO in general?
And what's on your guys?
What are you guys excited about on the roadmap going forward for the next year?
Not sure if excited is the one I do, but the CVE situation is something that well I committed to working out the CVE story for the next milestone which is like the next three months about a week before the drama all kicked off.
So regretting my life choices there but hopefully we get some stability and we understand what the future is going to involve so that needs to be improved and the new tooling written to just have everything in the SPDX and generate reports from that is where we're going to be going shortly.
So anyone who's been using the CVE check tool will probably be aware of its limitations, and hopefully the new tools will just replace that really nicely.
And that's before you rewrite everything in Rust.
Yeah.
No, don't get me started on Rust.
Josh, what's on your agenda?
What are you excited about for the.
Roadmap as far as the SBOM support?
We're definitely looking at doing the VEX reporting that Ross is talking about.
I was trying to think if there was anything else major.
We're still heavily involved in the.
I'm still heavily involved in the SPDX community also.
They're working on their 3.1 release.
So there's a couple things that we're looking at there to just make it better in general.
I can't remember any specifics off the top of my head, but there's been a few things that's been like, we could do this a little better and so we should do that.
So we're planning on.
We'll probably be an early adapter of SPDX 3.1.
Well, there are not that many tools out there that even do 3.0 yet, so I think you're very early on in that regard.
Yes.
Good.
Well, thank you guys so much for coming on the show.
I very much appreciate it and I hope people learn quite a few things about both YOCTO and about, well, SPX and as well as using SBOMs in the embedded world.
So, again, thank you so much, guys.
Have a good one.
Thanks.
Thank you.
Cheers.
Found an error or typo? File PR against this file or the transcript.