Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Dustin Kirkland on Chainguard, Zero-CVE Containers, and Supply Chain Security
Play On
14 FEB • 2025
59 mins
Share:
In this engaging conversation with Dustin Kirkland, we explored the fascinating world of container security and supply chain integrity through the lens of Chainguard’s innovative approach. Dustin, with his extensive background spanning IBM, Canonical, Google, and now Chainguard, brings a wealth of experience in open source, security, and enterprise software development.
We began by discussing Chainguard’s unique position in the container security landscape. Unlike traditional Linux distributions, Chainguard takes a different approach by focusing on building and maintaining “zero-CVE” containers - container images that are continuously updated to eliminate known vulnerabilities. This involves an impressive automated pipeline that can detect and patch vulnerabilities within hours of upstream fixes becoming available.
A significant portion of our discussion centered around the technical architecture behind Chainguard’s container security approach:
- The build system that pulls directly from source repositories rather than using pre-built binaries
- Their unique approach to package management using APK format
- The automated vulnerability remediation process that exceeds FedRAMP standards
- The implementation of software bill of materials (SBOM) for complete transparency
- The use of Sigstore and Cosign for container signing and verification
We also delved into the challenges of maintaining rolling releases versus static releases, particularly in enterprise environments. Dustin shared valuable insights about how Chainguard manages to provide both security and stability through their approach of continuous updates rather than periodic major upgrades.
The conversation took an interesting turn when we discussed open source sustainability and the evolution of business models around open source software. Dustin shared his perspective on how the industry has matured from the early days of Linux to today’s cloud-native landscape.
Some key takeaways from our discussion:
- The importance of proactive security measures in container environments
- How automated tooling can dramatically reduce the time to patch vulnerabilities
- The role of SBOMs in modern software supply chain security
- The balance between security, stability, and usability in container images
- The evolution of open source business models and sustainability
For developers and platform engineers interested in learning more about Chainguard’s approach, Dustin recommended visiting images.chainguard.dev where you can explore their public container registry and documentation.
This episode provides valuable insights for anyone interested in container security, supply chain integrity, or the evolution of secure software delivery in the cloud-native era.
Transcript
Show/Hide Transcript
Welcome back to another episode of Nerding up with Victor.
Today I'm joined by Dustin Kirkland.
Welcome, Dustin.
Hey, Victor.
Thank you for having me.
So today's episode is going to be around Chainguard, largely.
And I guess before we dive into what Chainguard is and all the juicy details on how you do security, maybe first let's talk about you.
Like, your resume reads like a who's who of tech.
You're from Austin, proud Austin, as far as I'm aware.
And let's just maybe give the audience a kind of a view of who you are and your backstory.
Yeah, you bet.
I've lived in Austin, Texas for 25 years.
I graduated from Texas A and M University, computer engineering degree.
Moved to Austin for a very specific purpose, which was to work at IBM on Linux.
In the very early days, late 90s, early 2000s, IBM was one of the first companies that would pay people to work on open source software.
And I love that and certainly built an early part of my career around Linux and the Linux Technology center, worked on the security team.
I spent eight years at IBM.
In fact, one of those years I was on site at Red hat in Boston 2005, long before IBM and Red Hat became one thing.
After eight good years at IBM, I joined Canonical, a very small, scrappy Canonical that was just building its first server, Ubuntu.
The, the server edition that was 2008.
Time frame, which was a 704ish, was the first server.
804 was the really the first time we had a, a real, you know, server experience.
Maybe.
Maybe, yeah.
804, I think Hardy Heron was, I think, our first real server and went straight to the cloud, you know, so like we skipped hardware.
It was kind of interesting.
The strategy then was Amazon was, AWS was just spinning up.
So yeah, I actually left Canonical and joined a startup that had built itself around an open source project that I had co authored called ecryptfs.
It was in the security space, an encrypted file system that we used in Ubuntu to encrypt home directories.
In the very early days of Ubuntu, I would say notably were encrypting home directories in Ubuntu before disk encryption in either Windows or Mac was a thing or default.
And now I think it's pretty standard to expect to encrypt your data.
But I think we did something pretty remarkable there in the early days of the Ubuntu desktop.
Anyway, this little company called Gazing was built around an encryption module that encrypted first MySQL databases.
The founders actually raised a round of venture capital funding and then hired a CEO and hired myself as cto and we had a really interesting fun run encrypting basically healthcare data in the 20111213 time frame, right around when the HIPAA compliance was going into effect.
Remind me to circle back to that when we get to Chain Guard because there's an important tie in there.
That company actually was acquired by Cloudera.
I did not go to Cloudera.
I went actually back to Canonical and led product for a number of years.
I did two stints at Canonical, four years the first time and then six years the second time around and you know, 10 years total.
Really proud of everything, you know, that we build at Ubuntu.
I still, you know, celebrate the Ubuntu Schweig and fun memorabilia.
After Ubuntu, I joined Google and worked on Kubernetes.
I was a product manager bringing basically Kubernetes on prem.
So I worked on the operating system and the Kubernetes worker nodes, which was, you know, important part of running Kubernetes on Prem for Google prior to the Anthos product.
But basically that became the on prem part of Anthos.
After Google, I spent a couple of years in financial services.
It was interesting.
I really saw a wave of change coming in the financial services industry around just modernizing infrastructure and tech.
Both Canonical and Google had a lot of customers that were looking at Kubernetes and open source and cloud and scale.
So I first joined a private startup that provided clearing and custody services as an API called Apex, and then joined one of our customers, which was Goldman Sachs, who had built one of Apex's customers rather, who had built basically a consumer bank on top of that.
Took some time off after that and got in touch with some old friends, the founders of Chainguard, and they recruited me to join early on and help build out the engineering team.
So I think I'd wrap that with saying of 25 years of work, I've spent half of that in three of the world's biggest companies, Google, IBM and Goldman Sachs.
The other half in growth mode startups.
I've also spent half of that in engineering and half of that in product management.
So I've kind of had a foot in both engineering and product camps and both startups and big public company.
Awesome, awesome.
That's a lot to unpack there.
And I guess that's a natural segue.
Over to Chainguard, which is your current employer.
And I guess for those not familiar with chainguard, what's the elevator pitch?
What's the story like?
How do you describe it?
Briefly?
Yeah, we call ourselves the safe source for open Source.
So we've bootstrapped our own everything.
We build everything straight from source compilers, libraries, tool chains.
You could kind of think of it as a distro, but an undistro or a distro less distro.
We have, we don't take the typical distro approach of snapshot the whole world.
We're much more of a rolling distro rebuild all the time and constantly our product today that we sell and we've got customers, many customers running in production have basically purchased from chain guard hardened container images.
So we build all of this from source so that we can produce a docker image or that'll run in any OCI compliant docker runtime kubernetes and so forth.
And we keep those docker images@0cve.
CVEs are common vulnerabilities and exposures.
Those are named numbered vulnerabilities.
And basically we have an SLA with our for our customers that say that we will remediate vulnerabilities criticals within seven days, highs, mediums and lows within 14 days of there being a fix available for that CVE.
In doing so we have exceeded the Fedramp standards which are required by the US government in order to remediate CVEs.
We exceed the FedRAMP mandates which is 30 days for critical, 60 for high, 90 for medium and 120 for low.
So our customers often when they're buying chainguard, what they're buying is peace of mind that they will get their vulnerabilities remediated well before the Fedramp requirements require.
And in many cases they've got some work they need to do to rework, rebuild their services, redeploy those services.
So we bought them basically time.
Okay, so it's a really well hardened docker image.
So rolling releases, obviously there must be some friction with old school enterprise when you say rolling releases, because that would probably make a lot of CTOs for legacy firms quite uncomfortable.
Right, so how do you actually think of that?
Yeah, sometimes Victor, but I think for every one of those there's a dozen developers saying yeah, but I want this latest and greatest software, this version of this library that doesn't exist in another distro and won't exist in another distro until the next LTS or major release.
And so there are plenty of developers and platform engineers clamoring for a different way or for something new and different.
The other thing that we do is enrolling forward.
We're constantly absorbing all of those security fixes and what we're shipping is exactly vanilla upstream source code built into binaries rather than taking that code and transmuting it with merged patches or ported patches.
We don't backport patches, that's generally not part of the chainguard strategy.
So we don't snapshot something and then maintain it for 12 or 15 or 10 years or whatever the, you know, the current super long term support is.
We don't go through that process of having to decide is this patch back portable if we have back ported it, is the software still any good or does it work?
Yeah, in order to backport this is going to take all of this other work that we have to also move forward.
We just, we don't do that.
Instead we roll everything forward and in doing so we're shipping a much more, a much closer to the upstream maintainers vision of what they have just released.
And we often ship that within minutes to hours of upstream releasing.
And if we can pass, if all of our automation builds, tests, qualifies, signs, we can publish updated images typically within hours of an upstream release that fixes a CVE.
So those seven and 14 days on the SLA, that's the worst case by far.
Right.
Okay, there's a lot to unpack there because I think the first thing to unpack is if you dive into the world distros in particular, if you look at the amount of patches that applied to any piece of software like you mentioned, backports.
Right.
That's a huge part of the overhead that all the distros more or less will need to maintain.
Right.
And that's.
So you basically say we're not going to do any of that, we're just going to move forward.
Right?
Yeah, very minimally.
I mean we have the ability to patch.
There may be a few dozen patches that by the way we look to drop typically within hours or days of upstream actually releasing that fake.
So if there is a super critical thing that is in an upstream repository but hasn't been in a cut release yet, we can make a call which says all right, cherry pick that fix.
We're only backboarding it, you know, a micro version or two.
We're not backporting it to something that was released in, you know, 20, 15 or something.
You know, it, this is, it's A it's typically very small and we keep it for a very short duration of time and then drop it as soon as we can and we can get onto a newer released version which again passes upstream testing, passes our cicd, passes all of our integration testing of all these containers have to work together.
We drop that patch and we don't carry the burden.
The maintenance of, you know, continuing to need to port that forward.
How do you deal with, I mean, backwards compatibility must be coming up quite a lot, right?
In particular, I mean what springs to mind is Python 2.7 to Python 3, right?
Massive breaking changes, right?
If you're always rolling on latest, you will have obviously the latest and greatest, but you also will have some cutoff period where there is a breaking change, right?
And how do you kind of deal with that for your customers?
Because that's.
That must be painful for some of the customers, right?
Yeah.
So a couple of things there.
First of all, I oversimplified it a little bit with latest and greatest.
That is true.
We do that.
We actually ship all versions and rebuild automatically all versions of any open source software that is still maintained by upstream.
So we only end up life something when the upstream maintainers end of life.
So if Python still has an LTS or still has a security supported version of 3.13 and 3.12 and 3.10 and if hypothetically there was a 2.7 version, we would continue building all of those.
And when a customer buys the Python image from us, they get access to all Python images that are simultaneously supported.
Ditto for Postgres or MySQL or any of the other thousands of open source packages we have.
Now the second thing is back to compatibility.
Here's something that's I think really interesting about the modern paradigm, the current paradigm, and that's the container boundary, Victor.
Like within a container you can ship a whole bunch of things that you know, need to work together and that's that application.
And that application still requires and a down level version of, you know, Python, Ruby, Java, some library or something, but you don't have to move all the rest of your infrastructure that might be one of your 2,000 applications that you're maintaining inside of this, you know, giant financial services or health care company.
And there's an exception made for this one application which sits behind the firewall and it can continue running an older version of Python, but all the rest of the things have access to, you know, newer, later, greater software and then it's really risk management at that point.
Are you Willing tolerate some number of CVs and unpatched vulnerabilities in this end of life or, you know, less maintained software than the stuff that's truly latest and greatest.
Yeah, because you have, I mean, by moving it to the container level, your blast rate is a lot smaller.
Right.
So you don't have to think on.
Yeah, that makes a lot of sense.
Let's talk a bit more about the open source versus the closed source.
I wouldn't say closed source, but the subscription service.
How you want to define that.
I'm not sure what terminology you guys are using, but how do you kind of split this too?
Right, because obviously you need to make money.
So obviously providing free disk images is not a great way to just provide revenue streams.
Right, so tell me a bit more like what's your strategy around that?
How do they differ from the ones that anybody can use from your equivalent to Docker Up?
Yeah, yeah, good question.
So first of all, chainguard exists to solve a customer problem.
We're a commercially oriented company, we're venture funded.
You know, we're building a business around producing a great product that solves a particular customer need.
For the most part, our customers are largely either trying to meet a specific compliance framework.
Obligations under a compliance framework could be FedRAMP, HIPAA, PCI, any one of a number of those.
And in doing so, you know, we charge money for the value that we provide.
In doing that, we're often, you know, sometimes the business value analysis that we'll do, the BVA that we'll do for our customers will show 18 to 20x return on investment.
So what they're paying for Chain Guard is saving them 18 to 20 times what they would be paying to try to do this themselves, to manually patch this themselves or manually address these with other distros.
I would say, you know, from an open source Linux distro.
You know, we don't consider ourselves a distro as much as we consider ourselves a secure software supply chain.
You know, just in terms of what we're building.
It's a way to build software and it's everything that a customer needs to build their software.
We don't really, you know, look at ourselves necessarily as a Linux distro in terms of like free Linux distros.
I mean, Debian's fantastic, Ubuntu's great.
There's Arch, there's Gen 2, there's Nix.
I'm running one or more of those in the various machines I have spread around my desk.
Right Here the world is Fedora.
I forgot to mention Fedora.
Right?
The world is awash in awesome ways to, you know, do it yourself, build yourself, you know, and run a free Linux distro.
Chainguard is not really designed for that.
Chain Guard's design for the platform engineering team that needs to deploy thousands of container images at tremendous scale while mitigating vulnerabilities and risk so that they can go about running their business, which might be an airline or a hospital or a financial institution.
Those types of customers of ours, they are not in the need to be in the business of mitigating CVEs.
And so we take that headache off of their plate for a fee, effectively.
And you know, for the most part it's a really, you know, it's a really, it's a good value prop.
Right?
But you can still, even if you're not a customer, you can pull down all your latest images.
Right, if they are publicly exposed.
Right.
There's a.
We have a subset, it's 40 or 50.
Yeah, we have 40 or 50 Docker images that are available.
You can, you can docker run cgr.dev or images.chainguard.dev is our image repository.
You can clearly see which ones are available, which ones are freely available.
Docker run Docker run Chain Guard slash Python.
We've got some in Docker Hub as well.
That's another good place to pull it from.
So if you're pulling from, from Docker Hub, you could run Docker run chainguard slash Python and a handful of others.
We basically kept our 40 most heavily used images.
The latest version is free.
All of the other, you know, subsequently supported.
Also supported versions are available, you know, as part of the commercial license.
We have FIPS versions of many of those images which, you know, utilize FIP certified cryptography.
And then there's a catalog.
We have about 1200 distinct images that we [email protected] about 40 of which, you know, are available for free.
And the for free is really for proof of concept.
It's for, you know, try before you buy is for test it out.
It's for comparison.
Go scan Chain Guards Python image compared to another one and you know, let your sneak or wiz or a trivia gripe or aquasec.
Let it tell you know, which image has more vulnerabilities and which one has fewer.
Okay, all right, interesting.
But, but there's nothing that prevents me from using them for like say, coming outside of Your target audience, so speak like a smaller startup, they can just go in and run it and try it out.
Or smaller projects.
Right?
Yeah, absolutely.
I mean our.
We've got a wide swath of customers, you know, many of which are governments, government agencies, some of which are public companies protecting their data, their users.
Some are public companies who are selling services to government agencies and so forth.
Some are startups, AI startups.
We have a number of AI startups that start with Chainguard because we have images for many of the.
We keep talking about Python.
Python's hot right now in the AI space.
We have Python images, chainguard, Python images for many of the AI tools that don't exist as a package in another distro.
You can't apt get, install or yum install the vast majority of the software and yet we've got Chainguard images that you can just run in your kubernetes.
Right, okay, cool.
So let's talk a bit about govtech in particular.
Well, not govtech but like compliance, I guess, for governments, obviously the executive order, for instance, drove a lot of SBOM adoption.
Right?
That was a big tipping point for sbombs.
You guys kind of slapped into that kind of wave of new, kind of forced, I guess, forcing the old brigade to kind of like step up the game for security.
Right.
So tell me a little bit more about that.
Like what you mentioned, Fedramp is I would imagine is one of the big drivers for you guys.
Tell me more about like it's a big responsibility, right?
And being.
You must have like automated a lot of that build process in order to meet those compliance requirements, right?
Yeah, indeed.
So a couple of things.
First of all, let's talk like briefly sboms, especially for, you know, someone who doesn't know.
SBOM is a software bill of materials.
I like to think of it as the nutrition facts on the side of the cereal box.
The SBOM tells you what's in your software and there's a very standard, I think your last episode, at least that I listened to, talked a bit about the SPDX file format and SBOMs.
It's a byproduct of what we do at Chainguard.
So we produce an SBOM for every single image that we produce.
Along with that SBOM or within that SBoM, we've got every package, every version of the package is identified clearly in that SBoM.
The open source license, the pointer to the source code and the commit, the exact commit ID where we built this from and the exact tool chain that we used to build that.
So Going down to all of the linked libraries or static libraries that got included in all of the software.
And so it's, you know, think about it as a much more machine readable and much more detailed version of, you know, hey, your, your cereal included flaked oats and wheat and, you know, random other flavors and some high fructose corn syrup.
It's like that.
But for the thousands of pieces of software that went into building that one image, which runs Redis or something, I mean, that's.
I, I have a few questions around that, right?
Because obviously building SBOMs, people never built SBOMs.
They think it's just like, oh, I run a command and here is my sbom.
Great.
That's how people who've never actually done sboms in real life tend to view it, right?
You guys obviously have and particular building SBOMs in an environment like you are.
So if you're building everything from Source, generate the SBoMS for C code and I would imagine it's like a large set of different languages that goes into that toolchain.
How do you derive this?
Do you do it?
I know Yocto and some of the other projects, they do it on the compiler level.
That's how they derive all the response from artifacts from the compiler, essentially.
How do you guys solve that?
Yeah, it's sort of a sum of different, a couple of different pieces.
You know, we've got, at our core we've got a build system that is driven by GitHub Actions and then runs build jobs in Google Cloud.
Basically we start with building packages.
Our package format is apk, which is the Alpine package format.
We're not a derivative of Alpine, but we use the APK package format.
It's just a tarball.
I mean, it's literally tar.
You can untar an apk.
Out of that comes the files that get laid down on the file system, plus a little bit of metadata.
The, the, basically the S BOM and A dot packages.
That YAML file that describes the entire tool chain that was used to build that package at that moment in time.
And we've actually got a patch set in a PR in flight right now where we're adding in the case where we carry any patches, you know, I told you earlier, we don't carry patches very often.
But in the case where we carry patches, we will put that into the metadata as well.
And we'll do that regardless of the license.
You know, GPL and GPL software requires that you do it.
Apache, BSD don't.
The patches are not precious to us on that.
So we're going to.
We'll ship that as part of that metadata that's part of the package as well.
Those packages then go into.
We use a form of terraform internally to define the images.
Images for us are those container images.
The images then are built from a set of packages that get installed, plus some other things we've got to do to fix up that image.
One of the.
I don't know how deep you are into packaging up packaged RPMs and DEBs and now APKs.
APKs are much simpler than DEVS and RPMs.
Our APKs don't, for instance, create users.
You install Postfix on Debian Ubuntu.
The Postfix patch package creates the Postfix user.
We don't do that inside of the APK itself.
The APK is just bytes that get written to disk.
More similar to Slackware back in the days.
Yeah, exactly.
But we do need a Postfix user if we're going to run Postfix to send mail or something.
So instead we do that at the image layer.
So that image will include, add the Postfix user, set its uid, create its home directory, run whatever scripts we need to do to configure.
Configure Postfix.
And then some of that allows for Docker style, environment variables and injection of metadata and data that happens at that image level.
The image itself comes with its master SBOM of everything that's in that image, which is sort of the packages have their sboms of what's in the package and then the image is.
Includes an sbom of all the packages that are in there as well.
So you have a hierarchy of pointers essentially inside the SBoM.
External reference whatnot.
Yeah, yeah, exactly.
And then each of those will point very directly back to the typically git repository or whatever the source repository is, the exact hash ID of the commit that we use to build that thing.
So, you know, the goal is totally reproducible, whether it's us, we need to rebuild it sometime later, or you the customer, or someone else downstream of us needs it sometime later.
Right.
You touched on something that's highly hot topic right now in Espo, which is license compliance.
Right.
Like we touched on that in last episode with Gary and Kate.
Doing that audit is non trivial work.
In particular if you build the everything on source.
Right.
So how are you guys going ahead and do that?
Do you literally audit every single package?
Like, ooh, Somebody modified this GPL2 license in this file.
Or how Stringent.
Have you guys been in that audit process?
Yeah, it's ironic you're asking me that.
Just a couple of hours ago we had our prioritization for this quarter's worth of work and we've got an important piece of work we're going to do here, which is ongoing license auditing.
So essentially what we do right now is the first person who packages something, the first person, engineer who builds something, does the license research and then sets that license in the YAML metadata about the package.
And we've got, you know, some audits and checks and balances and another developer has to sign off on it.
But it's generally assumed that the license doesn't change over time.
And if it does, then it's a manual if someone would need to manually notice that and go and fix that.
And yeah, there are some license changes which, you know, make hacker news.
And we all hear about and read about usually when something's going from a open source or permissive license to a BUSL or something like that.
And it's, I mean, it's not a problem when we know that we can go and fix it.
However, there's something that we've been, I've been increasingly concerned about, which is just in keeping with that chain guard ethos of tracking everything it took to build a thing and scanning.
We also scan, you know, all these packages for vulnerabilities.
Why, why don't we also scan those for licenses and yeah, both check that what we say the license is that it matches that and then really throw up the flag if somehow, some way, something, a package changed its license.
So yeah, just a couple hours ago, we signed off on a bit of work for a couple of our engineers to lead around ongoing license CICD basically for those packages.
And then, you know, failing a build, for instance, if the build says a package is GPL v2 and instead it's MIT, or maybe it was GPL, GPL and then one day became MIT.
Then, yeah, hey, let's take a look and make sure that, you know, nothing's amiss here.
It's important to us, it's important to our customers.
You know, so far it's usually not the ospo, the open source office driving, you know, the chain guard purchase.
It's usually more the platform or security engineering, maybe the CISO or cto.
But usually to buy chainguard, it goes through their OSPO for a review.
And you know, we've been able to answer their questions with our SBoMS and SPDX files and demonstration of that.
But this one is trying to go above and beyond what our ongoing obligations are.
Right.
So and because you do this manual process for every package, I guess as part of that you could do an audit and making sure that you are adhering to the SPDX license files properly and all that.
Right, because that's a mess as well.
Like if you just generate nest bomb for something like you will most likely not have a complete correct, you will have some, that's Apache 2, some of the attributes space 2.0 and like it will be a mess, right?
Yeah, that's right.
I mean you could get down like literally the line of code and it also gets really complicated when you start talking about like statically compiled software.
In fact, some of the most sophisticated licensed checkers have come out of the Golang community, especially Google, in fact, GitHub Google go to GitHub Google.
There's I don't know, five or six projects built around license checking, all of which are written in Go and most of which have to do with ensuring that, that like stack of libraries that get statically jammed into a statically compiled Golang single binary that those licenses are compatible.
You know, there are incompatible open source licenses, two perfectly acceptable open source licenses that cannot, you know, you can't cross those streams.
Yeah, that brings me back to.
I had a conversation with Alan Jude on why ZFS not in Linux kernel.
Right.
That's a perfect examp.
Right.
So but the goal is then that you want basically to deliver NTIA minimal element compliant SBOMs to your customer essentially on their behalf.
Like how do you, are you comfortable with that output?
Right now they are fully meeting and exceeding those output, those standards.
What was the standard you mentioned on.
All the NTIA minimum elements, which is kind of the gold standard for S pumps, right?
Yeah, I'm going to plead ignorance on that exact standard.
I don't know.
I can double check that one or let you know in the comments.
Fair enough.
We produce S bombs, you know, for every image, for every package, you know, and I, I think we're.
So far this has not become a blocker for any opportunities we've had.
Fair enough.
Yeah.
No, one of the contrast you have around there is, that's becoming more of a controversy is like stating who the producers of a particular package is.
Right.
Because that's hard.
How do you even do that in open source world?
Like, oh, I have this GitHub user who did this commit what country Is this person from who knows?
Right, right.
And that becoming more of, more a topic in the SBOM world, as I've seen lately at least.
All right, cool.
So this is super interesting and what do I turn my attention to now is you kind of already hint at this, but I really want to understand this better in terms of like attestation, signing how this, like from zero to like how this image is created and like, I guess the attestation all the way, like how you can really trust that all the way through.
Like talk me through that process from zero.
Like how do you like sign everything from the very first like thing you pull in?
Yeah.
So it really starts with.
We typically pull.
I don't know what the percentage is.
It's 90 plus percent now.
The vast majority of the time we are pulling straight from a source repository.
We're not grabbing a tarball and that's important.
So, you know, you kind of, you look attacks like the XZ attack and the binary artifact is very vulnerable to an injection of some malicious code.
It doesn't even take that much, Victor.
I mean, it's.
Yeah, it's literally bytes of base 64 encoded code that gets inserted somewhere, somehow that can open a door or break some other security that allows for a more sophisticated attack later.
So we did a full audit.
XE was very specifically targeting Debian and Ubuntu.
We pulled the XC.
We had that XE code available.
It wasn't affecting APKs, but was targeted at devs.
We pulled that code within hours, replaced it, rolled it back, waited for a fix, and then republished that.
But the other thing we did is we took a full audit of our source repos and went and looked at the few places were still pulling tarballs instead of zip files and tarballs instead of source repos and replace pretty much any of them, all of them that we could.
What we do now instead is when we build a package, we use git ostensibly, if not git, then whichever other distributed source repo is being used and checksums, basically look for hashes and digest and make sure that matches.
If that doesn't match, we don't attempt the build and we flag that and someone has to go and double check what got out of sync.
But we do have this automatically part of our back end build system, which is proprietary and we hold precious.
That's part of the chain guard Secret sauce is watching thousands of open source projects simultaneously.
Multiple, you know, multiple streams of that, multiple, you know, versions.
And anytime a New version gets tagged by the upstream maintainer.
We've got asynchronous jobs that watch those.
And the moment a new release is tagged, a bit of chainguard automation will fork a job that will go and pull that source code.
Check that, and this is the part that you've asked about.
Check that those checksums match.
We'll apply our build rules.
We'll increment the version number to match what upstream has released.
We'll rebuild that package and then build a dependency graph, a directed dependency graph of everything that depends on that build depends and runtime depends.
When there's a build dependency, there's a big flurry of activity that happens where we will rebuild everything that build depends on this thing that's bumped and changed recursively all the way till we get to the solution that solves all of the nodes at the end of the tree.
When those packages are done, then we build another graph of which images do we need to go and update and rebuild, because that package has actually changed.
And so that kicks off another set of jobs to rebuild those images.
Then we resign, and then we republish to the Chain Guard registry.
And then our customers, most of our customers run their own registry and they can configure it in either a pull or a push.
They get.
Our customers have access to their list of images that they have their license.
So we've got a, basically a master list for each customer which images have they bought from Chain Guard.
And then either in a push or a pull, we sync to their repositories or they pull from our repositories all the updated images that have changed.
And this is happening right now as we speak, Victor, without anyone doing anything for, you know, 80, 90% of the time, this is just automation flowing through.
And then we've got engineers that are catching our.
We've got, you know, sustaining engineers, delivery engineers who are catching the ones that fall out of that automation and we'll go and address it.
And some of those are manual things we've just got to go and fix.
Occasionally we'll see a pattern of failures that we can go and solve at the automation level so that we never see that class of build failures again.
Right, Talk to me a bit more about how you actually do this.
Right, so in Toto cosine, those are tools that are pretty widely used in Sigstor in this ecosystem.
Right.
So talk to me about that, how that fits into the whole chainguard system and how people can verify data that's.
Pretty close to the chainguard origin story.
Two of our founders, our CEO Dan and our CPO Kim, were two of the original creators of Git, Cosign and six Store.
In fact, chainguard.
Our first product when were a very new startup, before we started building these hardened images, was built around Sig Store.
Actually, we ended up sunsetting that product and focusing on the guarded containers once we saw tremendous product market fit and, you know, just the value of the problem that we're solving for those customers.
That said, we produced Sig Store artifacts signatures all verified through CoSign for every single chain guard chainguard image that we produce.
In fact, I would direct you to images.chainguard.dev where you can grab any particular chainguard image and you can actually see the provenance of that image and get the commands you need to run Cosine Verify.
All right, here we go.
you can see down here view all 1177 images and you can filter on FIPS or AI or GPU.
I'm going to grab one of these and you can search.
I'm going to grab one of these at random.
I'm going to grab the Envoy Image, Last Change 5 hours ago.
And you can see how frequently these things are getting rebuilt.
Envoy last built five hours ago.
I'm going to click on it.
Envoy is not one of the free images, so you'll get the pull URL as Contact us, contact us.
I'll pull Python up in a second afterward.
You were just asking about the provenance, being able to really track and understand where this code is coming from.
If I click on the Provenance tab, you can verify those Envoy images with this Cosine Verify going and checking Sig Store piping this through jq.
You can do that for the production or the developer images.
We've been talking sboms.
I'll click on this SBOM tab and you can see everything that goes into the chainguard Envoy image.
And it's these two dozen packages.
There's a busybox and some certificates.
You need glibc, you need gcrypt.
And then you know, the main package is of course, Envoy.
And then this configuration and this entry point.
And by the way, you can see the licenses were talking about.
Those licenses, you know, gpl, Apache.
Look, this one's MPL and mit.
And then the big tab is the vulnerabilities tab.
So every couple of hours we rescan all of our images as of 5 hours ago.
And that Changes, you know, because vulnerabilities get published all the time.
Right.
But as of five hours ago, this envoy image was free of CVEs.
It was last updated, last built five hours ago and scanned here.
So built and then scanned.
Now just seeing a bunch of zero CVE images is actually kind of hard to understand what's going on.
If you click on the Advisories tab, you'll get a timeline of the things that were fixed and when they were fixed.
So there was a glibc vulnerability that was patched on January 23, which is less than a week ago.
And you can see some of these.
We do some research on and we decide, hey, this busy box doesn't actually affect Chain Guard because, you know, we disable networking and busy box or something like that.
So yeah, that's a pretty good look at, you know, an image.
I'll.
We've talked about Python a lot here.
I'll just grab the Python image as a one more quick point here you can see the Python latest image is freely available cgr.dev chainguard python.
We mirror those to get to Docker Hub as well.
Remarkably, I think you'll notice that python say 21 megs.
This is absolutely tiny.
And the reason is because you go to the SBoM and you see what's in our Python image.
It's a minimal Python.
You go and pull the Docker Hub image for python and it's 300 megabytes of image magic and like hundreds of modules that you may not need in your Python image.
Well, once you've done a PIP install, you will have up to 300 megs already very quickly.
That's true if you're adding, you know, additional stuff on top of there.
So I'm grabbing the vulnerabilities tab again.
This Python image was scanned five hours ago.
Completely vulnerability free.
This is really cool.
I saw Wolfie in one of these ones.
I'm not sure is that one of the repositories or.
I wasn't quite sure what that was.
I saw that Wolfie is kind of our.
It's sort of a code name, It's a beloved code name for essentially the open source core that is core to Chain Guard.
It's not exactly like.
But you could kind of think of Wolfie like Rawhide or maybe Fedora is to rel or something like that.
Wolfie is actually the species of octopus.
That's our little logo right here.
It's the world's smallest octopus size of your thumbnail.
It's absolutely tiny, and that's a testament to how small.
We try to build images.
We really shoot for that minimal nature for our images.
It's no doubt impressive to build a Python image in such a small footprint.
All right, so we talked a bit about how you guys are building things, how you're signing things.
Those are super interesting in terms of where you see things are going forward and particularly around open source supply chain security.
It's been in the news a lot lately and finally feels like last year was a big awakening for a lot of people.
Like, oh, we actually need to stop paying attention to this stuff.
And there were actually some legal compliance coming through that actually, I mean, for me at least, I think security is one of those things that we actually do need legal push for because otherwise it's just being brushed under the rug.
Right.
With the rise of cra, all these things like, how do you see the landscape shaping out in the future of supply chain security?
Really?
Yeah, I mean, of course there's a balance between how much regulation is too much regulation.
I think it's clear that no regulation is the wrong answer, you know, especially when it's your data and my data and your healthcare data.
My healthcare data and financial records and stuff like that.
It's not a matter of, well, don't worry about it.
If a company is irresponsible with your data, they'll just go out of business.
That doesn't work when we're talking about like too big to fail type institutions, you know, And I don't know, I'm pretty exhausted by it.
Right.
I mean, it feels like once a month I get another free credit report because someone's leaked my data.
Like, yeah, I don't need more free credit reports.
What I need you to do is take care of my damn data.
Right.
So, like, there's just, there's no value in another, you know, another free credit report.
And that's all that we get basically when there's a, a data breach or it's a get.
Of j free card.
Right?
Yeah.
Or go cash this check for $11.27, you know.
Right.
Like, oh my God, that's going to cover, you know, my privacy being violated.
Right?
Yeah, I think there's, I think we need something here.
You know, the small piece of the space that we've carved out in chain guard is patch the vulnerabilities that have fixes.
Like there is no excuse.
I, you know, would posit there's no Excuse for a major company, especially a public company, a company that has public shareholders, trusting them with the data.
There's no excuse for having unpatched vulnerabilities beyond a small amount of time.
And that's kind of what FedRamp says.
30 days for critical.
And like I said, we can beat that.
And I think you should beat that.
But putting some number out there that.
Look, you can't just have criticals, highs, mediums, vulnerabilities that have fixes.
That's the thing, Victoria, we're not talking about.
We're not asking, you know, your airline company to go and write the code that fixes a vulnerability.
They're depending on us, who's depending on Upstream to fix it.
And until such time as it's fixed, it's still a zero day.
And so none of these say anything about zero days.
There's going to be zero days where I just find there's like almost blatant irresponsibility is not applying fixes that exist out there in the world just because we can't go that fast or it might break compatibility or, you know, excuse, excuse.
I think, I think we got to move past that.
Yeah, 100%.
I mean it does open the Pandora's box of like.
I mean, you're saying we talk about zero days, but like Open Source is, no, is obviously here to stay.
There's.
I mean, I think you have to be very naive to say that Open Source is dead.
Right.
So.
But it does raise the problem with this, which I kind of covered a bit with my episode with Mark, which is like Open Source funding, Right?
Like we need to figure out that.
And how do you see that?
Obviously you've been in the Open Source and Linux world for a long time.
Open Source funding hasn't quite been solved is probably a very big understatement.
Yeah, I mean, I'm, you know, going, I'm in the middle of my third decade in and around Open Source and I would say that it's nice to say that Open Source is here to stay.
It certainly wasn't that way in the late 90s, early 2000s, you know, were putting this thing together, you know, called Linux and Open Source.
And were up against Microsoft at the time, right?
Not this Microsoft, but you know, Gates and Ballmer era Microsoft, were up against Unix for that matter, Solaris, even in IBM where were, you know, loud and proud.
Linux, were the revolutionaries.
AIX was the, you know, was the Darling System 390 mainframe, was the Darling, that was the cash cow.
There were, you know, a few dozen of us and then a couple of hundred of us working on Linux at IBM in the, in that, you know, 2000, 1, 2, 3 time frame.
Thankfully I think we got past that.
Actually you know what, I'll even go back to the startup that I was at gazing which you know, we built a brilliant product.
This encryption encrypted file system and key manager that I think was pretty prescient.
It predated Hashicorp Vault, but worked a lot like Vault works in terms of serving keys.
This was 11, 20, 11, 12, 13.
We struggled to fundraise, built around an open source project.
I mean we raised a little bit of money but certainly not the kind of money that chain guards raised to date or other companies, you know, like us.
I think that's a testament to how far we've come.
Yes, I do think back to like funding I do believe, and this is maybe a different Dustin speaking than you would have gotten 10 years ago or 20 years ago.
You got to have a business model around it.
I don't, I don't think certainly if you're in the, in the startup world, the idea of build a thing, give it away for free and then one day rug pull, change the license and now expect everyone, your, you know, beloved users to just hand over money to you.
It's a little disingenuous.
I want to know that going, you know, I want to know that from the start, you know.
Yeah, I agree with that and my head is more around the open cells of the world, right, where there are like five people the planet who knows that this stuff works and you can't quite monetize that.
Right, the libraries, right?
Yeah.
I mean look, there is a, I think there's a huge place for the Apache foundation, for the Linux Foundation.
I think, you know, the nonprofits, for the common goods is definitely, I think that's the right place for it.
I'd love to see more opportunities for independent developers to have a, you know, a full time or a part time home there.
You know, Linux foundation is very famous, fully famously made a home for you know, Lina Storval's, Greco Hartman, you know, others like that and lots of little foundations that have spun out of that.
Apache is a collection of similar foundations.
The Wikimedia Group is a collection of different but similar foundations.
I think that's a, I think that's a model that maybe scales a bit better than an independent developer who writes one essential library or tool and you know, either doesn't.
It's hard to monetize, man.
It is.
Without a, without, you know, some stroke of brilliance, without a, an mba, without like a co founder or a teammate who knows how to take your brilliant innovation and turn it into money.
That's not something that independent developers who can write some amazing code can necessarily do over and over again.
But I do think the frameworks around some of the foundations could maybe help if there was residencies available for things like, you know, Open ssl.
Open ssl I think got the kind of help it needed once someone shined a light on the fact that, you know, the entire world's infrastructure depends one person working either figuratively or literally in their basement.
I don't know.
That is an important thing to shine a light on.
Yeah, I mean it's the old XKCD cartoon, right?
It's like this one piece of everything depends on.
Right, yeah, that's exactly what I was referencing and thinking.
Yeah, I mean it's interesting.
Like I.
Yeah, I think, yeah, maybe that is the path.
I know GitHub tried their funding thing with like subscription donations, but I don't know how.
I have no data on how well that has actually worked out.
But yeah, it's obviously it's an unsolved problem, I would say at this point in time at least.
Good.
So this has been super interesting.
I think we covered a lot of grounds here.
Is there anything you want to add to the conversation before we wrap up for today?
What do you hear from your listeners, your viewers?
What are they worried about from a software supply chain?
Are they worried and if not, is that a problem?
I think the people that are listening to this tend to be very tech savvy.
So I think they are the enlightened ones that they want to know the juice, the secret sauce that goes into the chain card.
I think that's really what people want to know about and that's why I wanted to focus on.
I think on the other hand, there is a massive part that is not my audience and I think that CRA in particular in Europe, EU Cyber Resilience act will really force a lot of companies to have a heart attack in the next six months when they realize that oh, holy shit, this piece of software that we haven't maintained for the last seven years, all of a sudden we actually need to do something about that and we're talking about rolling releases versus static releases.
Try to jump from seven years with the software and be compliant.
That's not a pretty one.
And I think that's.
Yeah, that's what I think.
That's what I think is going to change this year with regards to software supply chain security, at least on the European side of things.
Yeah, well, I'll take a crack at that first one like demystifying what, you know, what's.
How does Chain Guard work and what makes it special.
We're about 300 people.
Many of the technologists here in the company have come from other distro backgrounds like myself, you know, Canonical Red Hat, others, many of us came from Kubernetes and Docker and open source container back or container backgrounds.
Almost all of us have open source in our veins, in our blood.
We are maniacally focused on solving real customer problems, real world customer problem.
And that's a little different than the completely noble mission of putting out the perfect Linux distro for the world.
Ubiquity for the world.
We're not focused on ubiquity.
I don't think it'll, I don't think any of us lose any sleep at night over whether Chain Guard's gonna take over, you know, the world.
What we do lose sleep about over, you know, any given night is if we don't meet our SLA for our customers who are expecting Chain Guard to meet or beat or exceed those CVE remediation and so we've got, you know, alerts every alert you can imagine in place to ensure that's not happening.
Secret sauce, how it's working.
You know, it starts with a build system that I think is absolutely world class.
I think if someone wanted to try to reproduce Chain Guard, that's the part that would take them years to build and tens if not hundreds of millions of dollars in funding to reproduce.
We stay as close to open source releases as possible and the patches are microscopic and fleeting.
We get rid of them as quickly as we can.
The concept you mentioned about trying to upgrade from one version to another and there being like six or seven years, this is where I think there's actually a falsehood built in there about stability.
When you talk about the stability of a thing, and I only use Red Hat or something because it's stable until it's not.
It's stable until you need to upgrade from one to another and then you've accumulated years and years, if not a decade of tech debt and changes that then need to be accounted for.
And so instead what you trade when you Take that like rolling release disturb approach that always updating all the time.
If you're never more than a week or two out of update, if you have a 30 day, everything has to be patch or remediate.
You got to reboot every 30, you're never more than a month away.
And so the idea of like git bisecting your code that depended on something in Python that changed between 311 and 312.
I mean, Victor, that it's a pain, but it's like stubbing your toe as opposed to breaking your leg.
You know, upgrading distro versions.
Even from an Ubuntu 22.04 to 24.04, it's like it'll take nine months of rehab to, you know, recover from a broken leg.
It will take nine months to prepare for, you know, an upgrade of a fleet of ubuntu systems from 22 to 2404.
Yeah, I think you, I mean, you're absolutely correct about that and I completely agree with your viewpoint there.
But I think there is a big sentiment in the tech industry particular people have been around longer that it is.
You want to make microscopic changes so that you don't break things.
Right.
But that is a good way of accumulating all tech debt.
And then you basically have to pay that price at some point.
And do you have.
I agree with your philosophy.
Right.
Like, do you want to do it incrementally or do you want to do it in a big bag?
Right, right.
You will have to face the music at some point.
Yeah, no doubt.
I think I've seen it both ways.
Certainly when I was in financial services, it was much more in the big bang.
We're going to upgrade every couple of years, but we're going to spend an entire year preparing for an OS upgrade.
I never want to go back to that again as an operator.
Many of the developers that we're serving, they're not in tune with that anymore.
I think maybe the thing that I would love to hear from your audience is, hey, how can we help you communicate to your boss's boss that there's actually.
I know what you need us to do.
We the developers, we the platform engineers, we need to deploy secure code, maintain secure code.
I'd love to hear from them.
How can we chainguard help communicate to their boss's boss that there is a better way that, you know, we can help them get them what they need in order to deploy that, you know, latest and greatest AI app in order to build that code.
That depends on a Python or a Java library that doesn't exist in, you know, in another distro right now.
Yeah, no, I think there is an opportunity to do this at this point in time.
Right.
Because now if you're already in the cloud native mindset.
Right.
Where kind of the Docker is your runtime and you don't really like Kubernetes is kind of commodity and Docker is the operating system that.
Well, Docker is the deployment fabric, really.
Right.
For your application, it's a lot easier to swap out the container runtime versus anything else.
Yeah.
I would not want to go back to the days of patching VMs.
That's a lot more complicated than upgrading a Docker image.
Right.
Replace those in place as opposed to the incremental upgrades, you know.
Yeah, absolutely.
Perfect.
Dustin, I've really enjoyed this has been a lot of fun.
I appreciate you for coming on the show and.
Yeah, good stuff.
Have a good one.
Thank you so much.
Thank you for your time.
Really appreciate the thoughtful questions.
Thank you.
Thank you.
Cheers.
Found an error or typo? File PR against this file or the transcript.