Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Past, Present, and Future of Computing with Bryan Cantrill, CTO of Oxide Computer Company
Play On
14 JUL • 2024
1 hour 23 mins
Share:
In this episode, I’m joined by Bryan Cantrill, CTO and co-founder of Oxide Computer Company, for a deep dive into the evolution of computing technology. Bryan’s journey from developing DTrace to reimagining cloud infrastructure offers fascinating insights into where our industry has been and where it’s heading.
We start with Bryan’s groundbreaking work on DTrace at Sun Microsystems. What really caught my attention was his frustration with system observability limitations and how it drove him to develop a solution that could instrument running systems without modifying them. This approach to problem-solving - focusing on observation rather than modification - has influenced system design ever since.
The conversation gets particularly interesting when Bryan shares his experiences running a public cloud on commodity hardware at Joyent. His insights into the practical challenges they faced, from OS-based virtualization with zones to adopting OpenSolaris, reveal the real complexities of building cloud infrastructure. These experiences clearly shaped his current work at Oxide.
I was fascinated by Bryan’s explanation of Oxide’s integrated approach to hardware and software design. We explore their innovative choices, from larger fans for better cooling to developing custom service processors and operating systems. Bryan’s perspective on the economic implications of the Moore’s Law slowdown, particularly regarding infrastructure ownership versus rental, offers valuable insights for anyone planning their computing strategy.
If you’re interested in system design, cloud infrastructure, or the future of computing, you’ll find plenty of practical insights here. Bryan brings decades of experience and a unique perspective on how integrated hardware-software solutions could reshape enterprise computing.
Transcript
Show/Hide Transcript
Welcome back to Nerding out with Victor.
Today I have a very special guest, a guest that has come up in multiple conversations around previous episodes.
It's Brian Cantrell and he is the CTO and co founder of Oxide Computing Company.
Welcome to the show, Brian.
Well, thanks for having me.
It's great to be here.
Very excited.
I don't actually want to start off with Oxide because I want to gradually make my way there because I think you have a very fascinating backstory and I want to take a stroll down memory lane back.
Back at your Sundays, actually.
That's where I really want to start.
And your, I guess your first initial claim to fame is you're involved with D, right?
Does your initial, like exposure to the world.
World really learned about you.
Really?
And I guess maybe we should start there because I think there is an interesting parallel with DTrace and modern monitoring.
And maybe you can just for the, I guess, younger audience in the crowd, explain what Detroit is and why was such a kind of groundbreaking project.
Yeah, I mean, so D Trade is a facility that allows you to dynamically instrument the running system.
The frustration that I had is that, you know, these systems are entirely synthetic, that we have the software systems and these are not biological systems.
Sometimes it feels like we've got that kind of complexity.
But at least for anyone who is contented to think that we've got biological complexity, I was like, you know, it's just, go bone up on the biology a little bit and remind yourself how exceedingly complicated biological systems are and how difficult it is.
You know, all of the lurches we've had in biology have been where we've been able to observe the system.
It's really hard to observe a biological system.
And by contrast, we in software systems, these are entirely synthetic.
We should be able to answer any question we might have about the runtime execution of a system, but we historically couldn't.
And we would have these systems that are executing millions, tens of millions, hundreds of millions, billions of instructions per second, every one of those instructions written by a human.
Again, entirely synthetic.
And yet if I wanted to answer really basic questions like what the hell is the system doing?
I couldn't do it.
And the reason that we couldn't do it is because when we kind of deploy these systems, they are not designed to answer those questions.
And, you know, they would kind of emit some, maybe some pre structured data around kind of what they're doing.
But if I know I don't want to Answer the question of what you think you're doing.
I want to answer the question of what you're actually doing.
And you know, again, at the most basic level I see the system is creating thousands of processes per second.
Who's creating them?
What are they?
It's like, no, we can't do that.
You know, we, if I.
What system calls are being executed on the system?
You know, we can't do that.
And at the time, all of the kind of the infrastructure around that was not focused on the system, it was focused on particular processes.
So if I wanted to answer the question of what system calls is this process executing?
I had, I had infrastructure to do that.
Truss on what was then Solaris, now Lumos Strace on Linux, for example, using the slash proc file system to control a process.
But those things are super invasive.
So when you're running Strace on Linux, for example, to answer the question of what system calls are being executed, you are stopping that process.
On every system call, you're stopping the process.
You're going back to Strace.
Strace is printing something out.
It's resuming the process.
You're actually acting as an automated debugger.
And that does not allow you to answer.
I mean, I think that ends up being very crude.
It's very invasive.
And it doesn't allow you to answer any questions about the system as a whole.
I want to know what system calls are being executed on the system.
I had really no way of doing that.
And kind of the question I had is like, why don't we dynamically change what the system is doing so we can answer that question?
And that was kind of the nugget of the idea in D trace and then kind of separating out to the way the system is instrumented from the facility that consumes that instrumentation.
So D trace, we have this notion of providers, and those providers, some of them can dynamically modify program text, some of them can dynamically modify, for example, the system call table.
Some of them can act on asynchronous events.
Some of them can dynamically modify user processes.
Some of them know to go to known instrumentation points, but all of them know how to dynamically change the system to feed data into D trace that can then process that.
And all with the objective of instrumenting production systems.
So I was really, you know, my big belief is that the systems that are in production are the ones that are actually interesting and right.
If you can't actually use this infrastructure in production, then it.
There's a bunch of Questions that you simply can't answer.
And what you are forcing people to do is take problems that they have in production and reproduce them in other environments.
And when you force people to do that, you're reproducing symptoms, not necessarily reproducing underlying pathologies.
So that's what we did with these race very much, you know, had some guiding principles that I still think are very important.
The kind of, the.
The most important principle in D Trace is absolute safety.
So it was very important to me that errant use of D trace could not actually damage the system that you were observing.
And this is a principle that unfortunately, other systems that have kind of followed on from D Trace haven't observed, haven't really adhered to, which is kind of unfortunate.
I mean, I love.
And I mean, I think you have to look at something like EBPF as a very different kind of system.
Very different kind of history there, and very much designed with the idea of like, let's go modify the system to, let's change its behavior.
And with D Trace, it's like, no, we don't want to change it.
We actually want to understand what the system is.
Right?
The whole focus on D Trace is I want to observe the system, I want to understand it, not that I want to modify it.
And we, of course, when you observe the system, you do modify it at some level, you modify its dynamic behavior with respect to timing.
But we don't want to do more than that.
We want to minimize that probe effect.
And that's been really important, and that continues to be really important.
And, you know, D trace is something that we, you know, continue to use on a daily basis and continue to use it to understand what the system is doing.
So I think we, you know, it was.
It was kind of early on that you realize, you know, whenever you're on something bigger than yourself, there's that kind of moment where you have this kind of, wow, this is actually.
This is an even bigger deal than I thought it was.
Like, I thought this was going to be pretty important.
And I think this is actually much bigger than that.
And that certainly happened very early on with D Trace, when were able to take a light to this part of the system that had not had a light on it in terms of inside the kernel, being able to instrument every function, entry and return of the kernel, being able to dynamically observe this kind of program flow that we couldn't see before.
And it was pretty quick where were using D trace to solve problems.
And I didn't know how I would solve without D trace.
And that has kind of happened over and over and over again.
And it's been exciting to introduce that to new generations of software engineers and have them be able to like, wow, okay, I've been able.
I don't know how I would have solved this without D trace.
It's something I've heard over and over again in my career.
And that's super gratifying, you know, to be able to deliver a technology that has that property is a really.
It's a rare opportunity.
It involves no small measure of luck in terms of having the right problem.
But really gratifying to kind of see it in the world.
Yeah.
Stood the test of time, I guess.
It's really like, it's a tool that still.
Yeah, that's really impressive.
And just talk me back to those days.
So you work with Brendan Greggs, who was obviously a big voice and a big man in the ebpf, which kind of, as you alluded to, is kind of like the next generation of system monitoring.
I'm curious about those early days at sun because it just feels like the gift that keeps giving.
Even today we're looking about as back at zfs and it's like, wow, this is still really cool, but it's 20 years old and kind of just now making into the Linux mainline.
And it's like those days just like, walk me back into those days at sun, prior to Oracle.
So I think that sun.
And there's a reason for that because in the kind of mid-90s, everybody.
Computer companies, Computer companies historically had their own operating system and the.
But in the, by the mid-90s, more and mid to late 90s, more or less, all of them had been persuaded to give up their own operating systems efforts and to embrace Windows as the future, believe it or not.
So there was this very dark period of time where Windows was everybody's future.
And you know, I wanted to do OS development and I came out of school in 96 and everybody was telling me that, you know, OS development is kind of dead, it's not really being done anymore, it's not important.
And I believed it was really important.
And I believe that there were, you know, when I was talking to some folks about the early ideas around D Trace and kind of asking like, why hasn't this been done?
And one of the responses I got from someone who was very knowledgeable about the industry was that if this had been possible, it would have been done by now.
And which is really kind of gutting to think that, you know, that long ago people thought problems had all been solved.
And, you know, it's kind of like there's this story of the patent office believing that everything that had, you know, could be patented had been patented in the late 19th century and how absurd that was.
But, you know, having seen that with my own eyes, I can understand how people, you know, as they, especially as they age as technologists, they kind of fail to see the possibilities for the future.
And a bunch of these companies kind of gave up on system software and just they ceded their future to Microsoft.
And at the time, you know, I.
It was a very proprietary system and not a very impressive one.
Honestly, I thought that there was a.
And really it was only Sun.
And this is really before.
I mean, Linux exists in 1996, but.
But just barely as a hobbyist project, right?
There's kind of the famous mail from Torvald saying, you know, this is a hobbyist thing that I expect to go anywhere.
And that was not that.
That was, I think, in 92 or 93.
I mean, that was not that long before.
And so Linux exists, but not really.
And the BSDs exist, but they're hanging under the shadow of this lawsuit from AT&T.
And the proprietary unices are kind of all going.
Are all going away.
They're all being replaced with Windows.
And only sun believed in the operating system, right?
And the, you know, as a result, sun attracted a lot of young technologists that believed in the operating system.
And, you know, we spent a bunch of time in the late 90s, I think, which is very heady.com boom and sun, you know, was in the kind of the right place at the right time.
And we spent the kind of the late 90s really making the operating system, fleshing out the abstractions that had existed since, you know, Bell Labs and making them work at scale and so on, and where that kind of landed us, ironically, post.combust so.combust happens in 2000, 2001.
Sun, you know, @ the time, of course, sun is going through layoff after layoff.
But meanwhile, we in the kernel group had these ideas that had been germinating and now had the opportunity to really go execute on them.
And they were a bunch of different ideas.
So they were Dtrace, which we started in 2001, which was born out of our experiences trying to develop debug production systems.
ZFS was born out of Jeff and Matt's frustration with existing file systems, the service management Facility was SMF was born out of our frustration trying to deliver a service on a system zones OS based virtualization.
Born out of the need to have true multi tenancy and wanting to do that with os.
Vert the fire engine which is a network performance project that was really important in realizing that like the ability these nics had the ability to process so many packets that we had to think much more intelligently about how we kind of use the nic, the fault management architecture FMA born out of our very traumatic experiences with faulty hardware kind of how we thought about that.
So there were all of these things that were kind of happening at the same time.
And I remember thinking like as, you know, in the late 90s, as were kind of attracting folks to the company, I remember being in Bonwick's office and telling Jeff Bonwick and telling him that like, you know, I really think we've got the potential to have a Xerox PARC like environment for OS development.
And that ended up being prophetic maybe in more ways than one because on the one hand we did build that environment.
On the other hand, I think, you know, sun itself obviously no longer exists and which I think I'm kind of, you know, perversely there is a level of gratitude that I have for that the, you know, sun obviously acquired by Oracle in 2010 and I think sun ended so abruptly and it was so clear that it's that we all really needed to go elsewhere.
The environment that sun had really encouraging innovators and innovation was so antithetical to Oracle.
Yeah, I can imagine there was kind of a clarity.
You know, a part of me is actually because, you know, IBM and HPE were both then HP were both trying to buy sun and it was Oracle that kind of came in.
And as these two are bickering with one another and Oracle ends up sweeping in and buying the company.
If IBM had bought the company, there's a part of me that's concerned that I would still be there.
I'm not, you know.
And so I actually appreciate in a very strange way Oracle really kicking me out of the Sun Microsystems nest and forcing me to go elsewhere in the world.
That's an.
But that's an interesting parallel universe where sun would have been folded into IBM's mainframe business and you would have this kind of like alternative server universe.
I guess totally.
There are some really interesting alternative universes to play out and that's definitely one of them.
And that was very close to happening.
Right.
I mean the.
And this all had to be a publicly documented because sun was a public company at the time.
So there's an SEC document that talks about company A, company B and company C and the acquisition of sun and company A is IBM, company B is HP and company C is Oracle.
And so, you know, all of this kind of intrigue is documented.
Right.
If the, you know, IBM and HP were both each wanted to buy the company, but each also wanted to sabotage the other and they also wanted to kind of nickel and dime it and Oracle again swept in and bought the company.
Right.
But it very easily could have been an inside of IBM.
And it's a different, you know, I don't know what would have happened.
It would have been a different history.
Like Linux would probably never have had the footing that we'd ended up having if that would have been a scenario.
Potential hard to know.
I think that's the alternate history where you write where the operating system is open Source not in 2005, but like 1997, which would have been interesting.
I, you know, it's hard to know.
I really, I think that, you know, it's tempting to think that, but I also think that one of the important attributes, it was always going to be true.
I feel that open source infrastructure.
Open source infrastructure is always going to win.
I think, I think that the economics are such.
We would hope and I think it's very hard to have open source that is entirely dominated by a single vendor and yet still becomes a kind of ubiquitous platform.
I mean, and Linux has the property of not being dominated by a vendor.
So it's hard to know.
Yeah, but it would have been different.
It would have been different.
And again I, in my kind of darkest moments, I'm afraid that I'd still be there.
So I guess I'm grateful for Larry Ellison for giving me total clarity of thought in terms of what was next, namely not Oracle.
Yeah.
But what I'm curious about more of is like what was the team dynamics at Sun?
Because I think there might be some lessons there for team building for future aspirations.
And the thing, I mean there are a couple of lessons that I think we've certainly tried to apply at Oxide.
One is that you individuals innovate.
Right.
Technologists innovate, innovators innovate and you've got to give and they innovate as a group.
Not to imply that there's not a bunch of lone wolves, but in contrast to CEOs, do not innovate.
And what the leadership of a company creates an environment where Individuals and teams coming together can actually innovate to solve real problems.
And you need to trust people to innovate.
The innovation is a risky act.
And risk, in order to take risk, you have to have trust.
And I think that sun did really an admirable job of trusting its technologists.
And an amazing number of companies don't do a good job of trusting it, trusting their technologists.
And I think you really need to create an environment that creates that trust and fosters it and encourages people to build what's in their heart.
And then you want to have people that you add to a company where what's in their heart is consistent with what the company needs to go build.
Right.
So that's kind of the trick because, you know, you can't just kind of have a Montessori school where everyone does whatever they want whenever they want.
Right.
But you also want to.
When you've got people who've got, like, you know, I've got.
Got a chip on my shoulder.
I've got something that is burning within me that I want to get out.
That's D Trace.
That's cfs.
That's.
That's fma.
That's a lot of these things.
And you want to have an organization that really fosters that.
And sun did.
Sun had an organization that fostered largely because no one was in charge.
I mean, I think that there was a lot that were.
I think sun could have actually been much more deliberate about that, but it ended up inadvertently creating that environment.
And that's the environment that gave us Java, that gave us Spark, that gave us.
And they gave us these innovations because the technologists themselves were ultimately trusted to go create them.
And now in.
In Java's case, and in Spark's case, it's because the technologists really needed to create a secluded environment to go.
Then when we did Fishworks inside of sun very much took the Java and Spark model of getting shelter from the rest of the larger company so we could go build something that was disruptive and innovative.
Right.
But the fact that sun allowed those things to happen, I think was the company at its finest.
And I think that's something that companies can take inspiration from.
Organizations can take inspiration from that.
You know, how can we empower innovators to go solve these important problems?
And what does that mean?
I think that sun was good at it and offer something for companies to learn from.
But I think there is probably a ceiling into scalability for that.
That model will not scale indefinitely, I would imagine.
Yeah, that's a Good question.
I don't know.
I mean, I think that the challenge we put in front of us with Oxide is like if you were much more deliberate about that model and much more directed about it, you actually need fewer people.
Bluntly.
Yeah.
And you know you can, if you're very deliberate about the way you formulate a team.
And you know, we, our hiring process at Oxide is designed to be very deliberate.
Much more deliberate than a Sun or Joy, which is where I worked after Sun.
If you're very deliberate about that team formation, how much can you do with how few people?
In other words?
I think the scaling issues, I think it's not really well answered to me because I think so many of these companies, and it's a challenge.
When you have a company that has got a great deal of commercial success, they often over hire just bluntly and over hiring creates its own problems.
We have all known that like small teams do great work.
So how do you have.
And I think, you know, Amazon was good at this for many years of having collections of small teams.
Two pizza rule.
Did the two pizza rule.
And I think that there's, you know, there's a truth of that.
There are plenty of aspects about the way Amazon ran itself that you know that I would.
I think there's room for improvement.
But I think that the observation about these kind of small teams and how do you empower these small teams and then how do you kind of scale out this idea of small teams?
I think that there's lots of room to advance the state of the art there and I think there are different approaches.
And one thing we definitely do at Oxide that's been very important that I think is very important for the scalability of an engineering organization is how do you kind of convey ideas.
And I'm a big believer in the written word.
I think that to write down their ideas allows a.
I just think it allows people to do more because it allows me to understand what other people are doing much more readily.
It allows people, when they come into the company, they can understand what.
And I, so I, I'm a big believer in.
And I think there are other things you can do that again allow you to scale a company without necessarily scaling the number of people.
How can you do more with fewer people?
Absolutely, I agree with that.
Okay, so let's move on a bit further down the line.
So obviously you left sun after they were acquired by Oracle.
You moved on to Joyent, which was I guess a semi spin out from sun in some ways.
There were some Core.
I guess it feels that way.
It.
No it's not.
Joy was not a spin out.
Joyent was a at all.
Joyent was an independent company.
The and team Constellation perspective.
I meant more.
Yeah, I think that is definitely true.
So there are the.
And in part because Joyent had made a couple of big interesting bets that were consistent with some of the bets that sun had made.
So in particular Joyent made a big bet on OS based virtualization on zones initially with FreeBSD jails and then when the operating system was open sourced they lunged at what was then open Solaris and became Lumos and deployed that into production.
And in many ways we at sun were looking at Joyent.
It's like this is the way this technology should be deployed and this startup understands our technology better than we do at Felt.
And so I think that the and one thing that was definitely eye opening with respect to the fate of Sun.
So Joyent had deployed open Solaris and needed and its business was growing quickly.
And in particular Joyent was in this Facebook developer program to allow Facebook apps to be developed on what was effectively primordial cloud computing.
This is where you know, shared hosting is beginning to become API driven elastic infrastructure cloud computing and Joyce business is growing and they need to expand their hardware footprint and sun could not make hardware show up.
And there's a very famous or infamous blog entry, certainly infamous inside a Sun.
The founder of Joyent, Jason Hoffman, wrote a blog entry the sun does not shine on me.
And he described this kind of journey of deploying sun software and not being able to mechanically buy Sun's hardware.
And I just remember reading that and thinking we as a company are screwed because the, because we had everything was strategically right about why we had open sourced everything.
But we need like at the end of the day when someone needs to go buy a hardware system, like we have to be able to like sell it to them.
We have to be able to make it show up.
And sun in its later years was unfortunately and it guts me to say it was just disinterested in the operational side of the business.
Son was kind of enamored with the strategic side of the business but had was bored by the operational side.
And that was a real tragedy because I think strategically we had made a lot of the right moves but when you're operationally disinterested, when you kind of lost that connection with the customer, then the company is lost.
And so Joyent wanted to buy sun gear and couldn't and called up their Dell rep and Their Dell rep was like, on it.
The Dell rep was like, absolutely.
And shows up with a bunch of Dell gear and Dell wins all of Joyce business running Sun's software, Sun's open source.
Right.
And so when it was time to look beyond Oracle, which is to say as I was, and I went into that acquisition with a very open mind, I'm like, oh, this could be great.
Like, we're gonna take Oracle's business acumen and combine it with Sun's, you know, technical ability to innovate, which of course is like, you'll see naive hindsight when I actually realized like, oh, this is actually just a corporate sociopath.
And I actually, you know, honestly, like, I'm just, I was ashamed of working for Oracle.
That's what it boiled down to.
And you know, Oracle coerces its customers and I, I can't do that.
I don't, it's so important to me to deliver something to customers that they love using.
Like that is real.
Which feels like it shouldn't be controversial, but that is really, really deeply.
That's important to like the meaning of the endeavor.
Right, right.
The fact that, you know, the fact that when the purpose and when technologists tell me like, I can't, I don't know how I would have debug this without D, like that is, that fills you with meaning.
Right.
And that fills you with purpose.
And to, by contrast, to work for a company that, in Oracle that has litigated against its customers, where it's customers, you know, the, I live in California, the state of California, had, was involved in long standing litigation with Oracle.
And it's like that's embarrassing.
And it's like, I, I can't work here, I need to go somewhere else.
And as I was looking around, it's like, you know, that company, Joyent really was using our technologies in a really interesting way to solve this really interesting problem in terms of cloud computing.
And at the time Joyent2 had seen not just the promise of elastic driven infrastructure, API driven infrastructure, but also the promise of these, some of these new languages and in particular Node js.
And you kind of added all this up and I'm like, this is really interesting.
And I was really excited to go to Joyent for all of those kind of technical, like that, strategic positioning.
As it turns out, in many ways, I should say the most important thing, what turned out to be a very important thing about going to Joyent was not just all of that and not just the technologists.
I got to work with at Joyent.
But the Dell rep that had made all of that Dell gear show up, well he got frustrated with Dell because Dell didn't see.
Actually despite the fact he made gear show up at Joyent, Dell itself did not see the relevance of Facebook.
And they kind of view this as like this is a website and he had left Dell and he'd gone to Joyent and ran sales at Joyent.
And that's Steve Tuck, my co founder at Oxide.
So Steve and I worked together for a decade at Joyent and when were looking what's next, what to do next, we knew that we wanted to do something together.
And you know, I think that with Oxide we have really tried to take that best of sun in terms of innovation.
High bar to hit head.
Yeah.
High bar to oh hi.
Bar to hit.
Oh, we're hitting it.
Oh absolutely.
We're hitting it.
Like that's a good.
Segue into actually for those not familiar with Oxide.
What's oxide?
Why does it matter?
Yeah, so Oxide Computer company.
So we are a computer company.
Names on the tin, very deliberate when went to raise like we're not.
We really wanted to make clear to VCs that we are a computer company.
So our big belief is that elastic infrastructure, cloud computing demands hardware and software co design, that these things have to be done together.
And you know, if you do only that hardware layer, you end up with personal computers.
Right.
And that's what we got from Dell and hpe.
If you do only the software, which is actually what we did at Joyent.
At Joyent we did, we developed a public cloud, we use that software could be run on prem by folks running their own cloud computing.
But our software, very sophisticated software ultimately reached limits at the hardware and firmware.
And ultimately were trying to deliver elastic infrastructure on top of personal computers from Dell and HPE and Supermicro.
And there were real limits for reliability, for availability, for performance that we couldn't actually square for efficiency.
And you know, as you look it's like, well, you know, Dell would tell me I'm the only customer seeing this and I, I've heard that so many times from Dell.
Like no other customer is seeing this.
And you know, we're running our own system software.
So the, you know, Dell always wants to ask what operating system are you running on this?
And of course like our answer to that is like an Illumos derivative.
Something that like, you know, people are like well that's your problem, whatever that is.
Like, sorry, what did you just say?
Like, that's your problem.
It's like, you know, like, leave my operating system out of this.
Like, this is my.
My operating system is not the problem.
Problem, pal.
We actually had Dell, we had a perk issue, the perk being the HBA host bus adapter, a perk issue where it would have a parity error on system reset.
And they tried to blame the operating system for that.
So this is like the system can't boot because of a parity error.
It's like, well, what operating system are you running on this?
It's like, that definitely doesn't matter because my operating system is not running yet.
Like, the system can't boot, you jerks.
Right?
Dell would tell you, like, you're the only customer saying this.
You're the only customer saying this.
And again, I have heard that so many times.
And I'm like, how is.
How all these hyperscalers, you know, yes, Joyent is running, you know, thousands or tens of thousands of machines, but you know, you've got like hundreds of thousands and millions of machines that are being run elsewhere.
How are they not seeing it?
And what you realize is like, oh, the hyperscalers are not Dell customers.
Hyperscalers are not HPE customers.
They're not, they're not even super micro customers.
They have built their own machines.
They have built their own machines to run at scale.
So if you look at Google, if you look at aws, if you look at Facebook now, Meta, they have become their own computer company.
And that rack scale machine is.
Or warehouse scale machine in Google's case, is not available for purchase by an enterprise buyer.
So if you want to run cloud computing in your own data center, for all of the reasons that you might need to do that, security or economics or latency or risk management, if you want, and you're cloud educated, you want elastic infrastructure, you want cloud computing, there is no product for you.
You can't actually buy that.
How far?
I mean, obviously Meta made a big splash on their open hardware stuff and they have their open source server infra.
I haven't honestly spent too much time diving into that.
Like, how complete is that?
Or is that a PR spiel?
Or.
I mean, it's okay.
It's not, it's not cynical pr.
So it's better than that with the Open Compute project, I think.
And we had Amir Michael, who was the, who led the Open Compute project inside of what was then Facebook.
We did an interview with him on our, on the Metal podcast.
When we started the company.
And it's a very interesting conversation because I think Amir, as a technologist, saw this divide between what Facebook was building and what was available from Dell, hp, Supermicron, and earnestly wanted to deliver those same.
He saw, like, this is a much better system.
I want to make this available outside of Facebook.
So I think the Open Compute project was born out of a very earnest desire.
The problem is that you can't just throw out kind of specifications of a rack and kind of have things magically hydrated.
And in particular, Open Compute never made any until extremely recently had no software component.
So if you could magically get one of these machines, which was very hard to get by the way, we actually have a Tioga Pass, which is their machine circa 2018, 2019.
We have a Tioga Pass inside of oxide.
It was extraordinarily hard to get that because Facebook buys these in, you know, unit hundreds of thousands, millions of units.
And the ODM that they go through for this, we win.
Is like not used to selling like a unit.
Like, we wanted to buy like one because, you know, we wanted to evaluate it.
You can't do that.
They want to sell you racks and racks.
Even when you get a Tioga Pass and we've got the physical hardware, what do you run on it?
Like, you don't have any software.
And is there a bootloader on that so far?
Or like.
Because I think they run core boot, if I'm not mistaken.
Right.
So they have tried to do a bunch of different things and kind of depending on your, you know, your vintage, they have run core boot, but it still has an A speed BMC on there.
So you've got this proprietary bmc.
So even Facebook was not able to go to a truly open bmc.
You've got, and you definitely don't have that, the control plane.
Like, even if you get the thing booting, it's like, okay, so congratulations, you've booted a host operating system that's not a cloud.
Like, you're a long, long way from a cloud and you need that whole control plane and then that control.
You really want to tightly integrate that, the control plane software with the lower, lowest level system software.
You want to deliver truly integrated hardware and software.
And we just saw that there was this enormous gap where no one was doing this and no one was doing it.
Not because it isn't a good idea, because it definitely is.
It's the right way to build it.
It's very hard to do as a startup and because it requires you to solve a whole bunch of problems, it requires you to solve.
And so we've done our own compute slide design.
We don't have an ASPEED bmc.
We've done our own service processor, that service processor actually when we looked at the system software on that, we came to the actual reluctant conclusion that we needed to do our own operating system.
So we've got an all Rust based.
Operating system which we appropriately, as one does.
What's that?
As one does, we appropriately enough called it hubris, just as a recognition that it was an act of hubris at some level.
The debugger for hubris, by the way, is called humility.
Appropriately, the.
So you know, we did our own service processor, we did our operating system there, our own system software there, we did our own compute slide design, our own board design, we eliminated the bias.
There's no bias in the system or AMD Milan based, but there actually is no bias at all.
There is so there is no core boot.
We have what we call the.
The Pico host bootloader.
Foible.
Foible basically executes the operating system directly and then the operating system actually brings up all of the different elements of the host cpu.
So is this similar to.
I think Red Hat announced something recently, I think about something similar like a bootloader that's not a bootloader or something along those lines.
Yeah, I mean I think that people have tried to do this.
I haven't caught up on what Red Hat has announced, but there have been a couple of attempts of this like Linux boot and so on, where you kind of have a Linux kernel, but ultimately those things are chain loaded where.
And perhaps Red Hat has adopted our model which we call holistic boot, where the system never goes backwards and never resets.
So the.
When we come out of reset we run our host operating system which brings up just enough components to be able to go fetch the rest of itself off of.
Because you ultimately you have to live in Spiron when you're at that kind of the lowest layer.
And historically what you do is you put your bootloader in Spyrom and it has kind of enough to find the kind of the next thing in our case, like the.
It's.
It's not the bootloader, it's the fraction of the operating system needed to find the rest of itself.
And so it doesn't use UEFI boot or anything.
There's no UEFI in the system at all.
And oh, uefi.
It's UEFI Yeah, I know.
Itanium's enduring legacy.
Sadly, UEFI was born on IE 64.
And Uefi is really problematic in part because, you know, every time you Universal and extensible and in the same acronym, it's like, oh, this is not good.
Because in order to implement uefi, you.
I mean, there's some very idiosyncratic implementation decisions, but it requires you to implement an entire operating system, really.
So a UEFI bootloader has an entire operating system that exists in the bias.
And that operating system will needs to actually initialize a bunch of units on the cpu and then it's going to.
But it's only doing that to go boot something else.
So it's like, oh God, I'm going to go boot something else.
I need to.
To like, now I need to like, shut these things down.
And so that the actual host operating system that I'm going to boot can think that it's on the bare metal.
It's just kind of like magically come into being.
And that's a huge problem because it means that one, you've got a bunch of functionality that is OS functionality sitting in this undebugable piece of proprietary software in terms of the bias.
And I feel very strongly that all of that needs to be opened up.
But then you also have this problem that the units on the part are kind of going backwards effectively in that you've actually initialized them once and now the host operating system is going to initialize them again.
And the problem is that it's really easy to have residues of that UEFI bootloader that the host.
And one of those residues has been formalized in smm.
Right?
There's something called system management mode, which the host CPU can enter whenever it wants.
Whatever reason it wants can stay there as long as it wants.
And smm, the actual software that executes when you are in SMM, has been installed by your bootloader effectively.
And when people find out about this, they're like, wait, what?
And like, yeah, sorry, it's.
And you know, the history here is like, well, meaning, I guess, in that, like, oh, like we've got USB mice in an era where we don't have a USB mouse driver, so PS2 modulator for.
Yeah, yeah, exactly.
Yeah.
And it's like, all right, I get it, like you want to make like a functional like laptop or whatever, but this is a very visceral example of this divide between hardware and software, with hardware telling lies to software by having software in It.
So the fact that, like, oh, no, don't worry, it's a PS2 mouse.
It's like, yeah, it's not a PS2 mouse, it's a USB mouse.
Like, why are you lying to me?
It's like, well, you can't handle the truth.
It's like, well, then, like, maybe that operating system doesn't deserve to boot.
You know what I mean?
Like, like, actually, can we just have open hardware that is clearly specified and let system software, like, be system software.
The problem is like, well, that doesn't allow me to sell a laptop in time for Christmas.
Is what that.
In, you know, 1992 or whatever it is, 1995, whatever that was.
Wherever the original sin is for SMM.
And then the problem is that SMM, it's, you know, of course I don't have a mouse on a server.
Like, why do I still have this mode?
It's like, well, once you have this vector for a bootloader to extend its kind of tentacles into the running system, I mean, you really have.
You bit the apple and now you actually have.
You have a system that is very hard to reason about because you have these layers that are more privileged than the privileged software on the machine.
And that's a huge problem.
There's.
There's.
Yeah, there's so much to unpack there.
Right.
Like, I think the most obvious thing to me and this stuff, like, I've been somewhat obsessing over biases over the last, like six or eight months or so.
4.
Purely from a security perspective, really.
Because, like, when I.
It's one of those things where I don't think I'd thought about a bias for a good 10 years because I've been running on, like, Macs.
And like, yeah, sure, they have a bias, but like, you don't really touch it.
But then at Screenly, we started doing our own hardware and I was like, all right, so we buying this piece of hardware from ODM in China that provisions this bias that I don't really knows.
There's no S bomb for that.
It's just like, here's a blob that you install.
Is there a version kind of in the zip file?
And then you're like, cool.
Then you see pixifail and logo fail.
You're like, right.
Or you run like strings on the binary.
And it's like, why are there URLs in here?
Out of curiosity, what's at the other side of that URL?
Why does my bias know about a URL?
Like, what.
What Possible good can come from a bias.
Knowing about the bias should not know about the Internet, let alone the web.
Because you want to have like a fancy logo says new bias update available or whatever it is.
Right.
But then like, who's running that machine?
What is that?
Like the attack vector is so severe.
100% and no, like that is all.
And that's a residue from running personal computers in the data center, which we should not be.
We should be running.
Like, we should be running servers and servers.
I feel very strongly that should be open.
These things aren't.
But I also feel strongly they shouldn't even exist.
And so I talk at the Open Source Firmware Conference a couple years ago.
I've come to bury the bias, not to open it.
Like, I think that layer of abstraction should not exist.
It should be up to the operating system to do that lowest level of initialization.
And the challenge is that is in tension with this kind of idea of like, well, I want to have a commoditized world where, you know, everyone can deliver whatever hardware they want.
And it.
But we, I mean, my view is like, yeah, that's why you need open source system software.
So yeah, if you want to deliver a new laptop.
Yes, you're going to have to do some Linux driver work if you want Linux to run on.
Right, right.
You know, that's a world that we are getting to, I guess, but not quickly enough.
Yeah, it's.
Yeah.
There's so many interesting things like in particular, like, so how do you do like secure boot or the equivalent of.
I presume you have you implemented secure boot on.
On docs and servers or.
So we have a proper root of trust.
Right.
So we have an actual root of trust on secure silic or putatively secure silicon.
The LPC55.
I say putatively secure because we have found two pretty serious vulnerabilities in it.
And the Laura Abbott on the Oxide team has done a.
And she's got some terrific blog entries describing these vulnerabilities that she found on the LPC55.
After you found the second one, people are like, why are you still using this chip?
It's like, well, your silicon is hard and there's value to be had in flying the airliner that just crashed.
The airline that just crashed tends to be on their game a little bit more.
But so we are using OP655 and we run actually hubris on that as well.
The root of trust can then attest to that payload that's running on the Service processor.
Service processor can then attest to the actual payload that's running in the spyrom.
And so you can attest all the way up.
So we know exactly what we're running all the way up and allows you to actually have confidence in the entire stack of system software that you're running, not just kind of one sliver of a component.
I think that's the problem with, with tpms.
I mean, there's, there are great things about tpms and.
But the problem is that they're too easy to circumvent because there are.
If you don't ask, if you don't ask the TPM a question, it's not going to give you answer.
Right.
So right.
We.
With a proper Harvard root trust, that is the root of trust that the.
And if you have something that doesn't attest, you can either choose to not boot it or you can, you know, we like the Chromebook model where it's like you can boot it, but we're going to indicate.
So you can debug it, but we're going to indicate that like this is not trusted.
So this is not from Oxide.
This is not signed by us.
So that's a very important part of the problem you're setting up to solve.
Right.
Okay.
All right.
So I want to talk a bit about the hardware design more because I'm one.
I'm sure I'm not the only one who's in the video of you standing next to the racks of server.
And anybody spent any time in a data center, they know how extremely noisy they are.
You walk in, it's just full of servers and you can, you're wearing earbuds or you're wearing headphones because it's so crazy loud and.
Yeah.
And you can actually have a conversation next to the Rack.
Like that's an impressive feat.
Like, walk me through how you manage to get there.
It's surely not liquid cooling removes, you.
Know, it's super simple and mechanical.
The efficiency of a fan is proportional to the cube of its radius.
And so if you want to move a lot of air with very low power, use a big fan.
The bigger fan you can use.
And the, and what we did and kind of the.
In many ways the first decision that we made was around the geometry of the fans.
So as were, you know, were contemplating Oxide, Trammel Hudson.
And if you know, Trammel is a huge friend of Oxide and an open source firmware pioneer for sure, Trammel's done a ton of interesting work.
And I was talking to Trammell about the initial ideas of oxide and Trammel was like, yeah, no, that's a great idea.
By the way, be sure to look at fan size because you know, the Facebook folks are use these 80 millimeter fans and I think it's a really big win for them.
So which is.
It's kind of like, okay, that's interesting.
Boy, I hadn't really like, tell me more.
I didn't really know about that.
And when you look at the geometry of an oxide sled, it's not this kind of like smashed 1U or 2U that's 19 inches wide.
Like that is a.
That's a bad geometry.
You actually want to be narrower and taller.
And by being taller you get a bigger heat sink on there.
And then importantly, you get big fans.
We've got 80 millimeter fans.
Those 80 millimeter fans can move a lot of air.
And in particular those 80 millimeter fans don't.
In fact, the.
One of the modifications we need to make to the fan is at 0% PWM.
Those stock Sanu Danki fans were at 5000 RPM.
It's like we don't need them to be at 5000 RPM, we need them to be at like 3000 RPM.
And so we worked with Sandy Danke, our fan partner, and they made this modification for us.
We're allowing us to be at 2K RPM and 0% PWM.
And that is in.
You're just drawing a lot less power and you're making a lot less noise.
I mean the acoustics are kind of like the, an artifact of the inefficiency of these one and actually even worse than the.
You know, it's funny because we have these one use in the office that we needed to use to develop the oxide software before we had the hardware.
One of these tricks when you are co designing hardware and software is how do I develop the software before I have the hardware?
And the answer is like at some level you're going to have to simulate.
You're going to have to take existing hardware and simulate what the new world's gonna look like.
So were doing that one U commodity servers with the.
You know, I did feel pretty bad because Josh Kluo, engineer at Oxide, was setting all these 1U servers up with their terrible biases and you know, the bias and the BMC and all of the crap that we.
And it was just like so bad.
And he was hitting like every bias bug you can imagine.
And I'm like, Josh, you know what we need to do is like, we need to start a computer company.
He's like, I know I'm at that computer company.
Like, you know, why am I having to live in the filth of the past?
Like I am here at Oxide so I can live in the future.
Why am I having to endure the past?
But we set up these one U's and they are super loud.
And we.
And actually Josh at this point wants to rip them all out of the Oxide office.
And Steve talked.
My co founder and I are like, no, we have to leave at least one or two of them so we can show people when they are visiting Oxide what the old world looks like.
Because people don't necessarily know they haven't been in a data center.
And if you look at an oxide rack, you're just like, well, of course you design it that way.
It's like, well, no, no.
Let me show you the way the world looks prior to oxide.
And you go into these one use that are loud as hell.
Yep.
With these screaming fans.
And I would.
I had gestured to this so many times as were kind of showing people kind of the story box set.
And one of the.
And I kind of like rubbing my, you know, kind of waving my hand across the back.
And one of the things I kept noticing is like, actually, you know what my hand is hottest is actually the fans on the power supplies.
Not.
And it's because the power supplies, those 1U power supplies have their own fans on them.
Right.
And those fans scream.
Power supplies generate a lot of heat.
Those fans have to work really hard because a power supply is super complicated.
An AC power supply, you're going from AC to DC super complicated.
It is not simple, right.
To do that efficiently is not simple because those fans, one those power supplies, they're trying to make them as dense as possible.
Yeah.
So there's a huge amount of static pressure that those fans have to overcome.
They're trying to pull air through a cocktail straw and trying to actually offload a lot of heat.
Those fans are screaming.
Those fans are also sitting on the hot side.
They are, they are screaming.
Those fans are.
The most likely thing to fail in your server are the fans on your power supply.
So you have two of them, right.
So you've got those redundant AC power supplies with this AC power cords and like those fans.
It's like.
And there is not, to the best of my knowledge, there is not a server power supply, AC power supply on the market that will tell you how much power is going into those fans.
Like that is an unanswerable question.
If you've got Dell HPE Supermicro, you can't even tell how much power is going into the actual fans on the server, but you definitely can't tell me how much power is going into the fans on the power supply.
And the answer is there's a lot of energy being wasted there.
And the, you know, one of the things we did at Oxide is the thing that every hyperscaler has done, which is to have a proper DC bus bar.
You've got a power shelf that is rectifying from AC to DC and then you're running DC up.
And then those power supplies are at just much higher power.
3.6 kilowatt power supplies.
Right?
I mean this is like much bigger.
They, they can be made much more efficient.
And so we've got, you know, six of these rectifiers that are able to provide power for the entire rack.
And so you end up with the kind of.
You add all this up and the geometry, the DC bus bar, and you add all of it up and then you can have a conversation behind the rack.
So it's a bunch of things coming together.
Talk to me about the dimension because that's interesting, right?
Because you, I mean, if you go into any data center, it's all 4219 inch racks, right?
So how do you bring something that doesn't comply to that standard and try to color that?
Right?
Yeah.
So I mean, I think like the 19 inch rack is like that.
You know, it's funny because we're like, God, we're gonna have so many people that are gonna resist this because it's not 19 inches.
It's like, don't care.
People care about 19 inches when they want to put something else in that rack.
If you're gonna give me an entire rack, I need you to fit on a floor tile.
But I actually don't care about that.
I mean, so it's like there we.
That ocp, the OCP width was too wide for us and so we wanted to be narrower than that.
But we felt like, hey, if we can fit on that floor tile, like the 19 inches is not what people care about.
We have also gone a bit higher than the kind of 42U.
If you go to a lot of DCs, there's a lot of height that's being on that's not being utilized.
You have to be careful about that because you've got cages, right?
So you have some head constraints, right?
That's right.
And if you, and you want to use that headroom because what you want to do is, I mean the actual like rack level problem is how do I take power and get the most amount of compute and networking and storage?
Like, that's what I want to go do.
And we put the power budget for the rack at 15 kilowatts, which at the time was like, oh my God, like hyperscalers are like, God, that's anemic.
People in enterprise data centers, like, wow, that's pretty rich.
Like I got.
Normally I'm on like 12 kilowatts per rack now.
Like AI has thrown all of that into the like now 15 kilowatts is like, isn't that like two GPUs now?
I mean it's just like.
But we wanted to, you know, how do you kind of use that 15kW with the greatest density of compute?
And ironically, you get the best density of compute by actually lowering your physical density.
In other words, by putting, by actually using a little more of that vertical space and opening things up a bit.
You open up that geometry, you allow those fans to run at much lower RPM and you consume much less power on fans.
I mean in a, you know, you rack and stack one use, you're going to be spending a quarter of your power on fans, which is, but.
I guess pretty great.
I guess what I'm curious is like if you go to like Equinix or any of these big DC providers, it's hard to rent floor square footage, I guess because you usually rent a rack, right?
That's a 42G rack.
You go in and rent like it's already pre racked.
You go there with your gear, you rack it yourself, right?
Is that, I mean, maybe that's a theoretical problem, but that's at least my.
Exposure to, in part because like we are giving you the entire rack, including the switches, right?
So we did our own switch as well.
That we're just kind of coming into that real estate.
That has not been a problem.
That has not been a the colos.
And as you can imagine, providers like Equinix are really excited to see something like oxide because you know, they have not, you know, they get very frustrated with the kind of the poor state of the art with respect to cabling and power management and everything else.
And so there it's very tantalizing to have a partnership with someone who's actually delivering true rack scale compute so that has not been an obstacle for our customer.
And they could probably reduce their spend on power generation like their diesel generators, their aircon, all that stuff can probably be reduced if this were to scale up.
Right.
So you can also get, I mean I think that in part of what we see that's in front of us that's exciting is you can get the true rack scale power management.
So you, because we, the service processors are on their own dedicated network just as they are when you've got, you know, racket stack servers, you've got the bm.
And part of the reason that you've got, you know, URLs and the bias is because there is a management network that these things are on, which is itself very scary because that management network has to be.
If you are able to root a bmc, you can root the box for sure.
But our service processors are on their own.
We've used some of the unused differential pairs on.
So we've got a cable backplane, right.
So there's no actual operator.
Cabling sleds are like gigantic blades and they blind made into that.
It's like cable porn really.
Like if you look at it really is.
It's, it is great.
It is.
And you, because you go to the back of the oxide rack and you're like, where's the rest of it?
It's like, no, this is the whole thing.
Like this is just like these little, these slim cable bundles as it turns out.
Like you've got a lot of very high speed networking on there and then you are able to have the service processors are networked in a very low latency network.
We've got a, an operating system in hubris that allows us to get real time or near real time response.
And you add all that up and it's like we can actually do holistic power management.
So we can.
One of the things that I'm excited about is the CPUs because they are both client and server parts.
There's actually a lot of really rich power management on the CPUs that we basically don't use really because we don't use because there's no real way of we're missing all the apparatus to connect it.
Well, we actually have all that connected tissue in the oxide rack.
And one of the things I'm excited about and we've got some potential customers now where it's like I need to fit this in 5 kilowatts.
And I'm like, we can do Some really interesting things in 5 kilowatts.
Because these CPUs, yes, they're you know, 225 watts out of the chute.
They can consume a lot less power and still do work.
And so, you know, what can we do to really lower that power profile and to give you observability into where the kind of the current power is going?
There's a whole bunch of things that we can go do and then that's where your kind of colo providers like that's definitely interesting.
Where, you know, could I, you know, can I foresee a world where I can dynamically shift a power cap on Iraq based on the cost of power or based on the availability of power?
I mean I think that the power infrastructure is a huge issue.
This is going to become a, a really pressing issue.
I think AI is forcing it to become a pressing issue because what folks are going to hear is like, hey, I don't have any more power to give by the way, so you guys are going to have to use it more efficiently.
And I think that is going to necessitate the kind of rack level power management.
Invest in data center.
Invest in data center on Iceland, I guess.
Where electricity is cheap, right?
Exactly.
That's right.
Well you could do where electricity is cheap.
It's great.
Like okay, we're going be located next to a dam or whatever and you know, we're gonna.
I think that the, I think it was Google originally.
What at the Dales in.
In Oregon, right Where they are buying up old aluminum smelters which were also.
Aluminum smelting is like a super power intensive process and so you can do all that but ultimately there are kind of there, there are limits there and we got to spend that power budget more efficiently.
And I think that really necessitates that kind of that rack level power management which requires this kind of holistic approach that we've taken.
I mean the like in order to get there you really need to take this kind of hardware software code.
Right?
Okay.
So, so you have your rack, you've deployed it and then I presume you have some kind of control plane management layer that like, is it like a combined PDU kind of IPMI structure to access entire rack or like how does that look like in the world?
Pretty different.
It looks much more like a cloud.
So what you are getting as an operator is not a collection of servers.
You're getting a rack that offers elastic infrastructure.
So Interesting.
Okay, we've done not just the lowest Levels of firmware and service processor host, cpu, firmware host operating system.
But then we have also done the entire control plane that allows you to go provision virtual machines.
We've done our own hypervisor, we've done our own.
And all of this is open source by the way.
Is it KVM send derived or is.
It something completely beehive?
Beehive derived, Right, because you're on the Unix stack.
Right?
Right, yeah.
And the beehive derived.
But we chucked the user level model.
We did our own user level, the machine model in Rust, which we call propolis.
Propolis is B glue.
So there's the origin of the beehive, the kind of the origin there.
And we did that in part because we wanted to reason much better about machine state.
So we could offer live migration as a first class operation.
The ability to live migrate VMs.
And you want to do that for product lifecycle management.
Right.
So when we, you know, in our experience operating a public cloud, the reason you end up with these islands of low utilization is because you got hardware that you really want to power off but you can't because someone is running on there that can't take any downtime or what have you.
And you're like, well we're definitely not going to provision anything new on there.
So like right, we just have this like this one customer and you know, these two small.
And anyone who's done cloud computing or any kind of hosting provider, this is going to be a similar story of whoa.
And now you've got a machine that's 70% utilized or 30% utilized or 20% utilized.
And what you actually need to be able to do is live migrate that vm, be able to move the VM without any downtime so you can pull the sled out.
So we have live migration as kind of a first class operation and then.
But we delivered that entire control plane.
So what you actually get to the interface into the oxide rack are Those API endpoints, CLI, web UI to go, provision VMs and provision virtual storage.
Provision virtual network.
How does I mean.
So you do live migrations, which is really cool.
But the pain in the ass or the pain point with live migration tends to always be storage.
Right.
It's not the CPU side.
It's usually like moving the storage side.
Right.
How does that work?
Yeah, so we the storage comes pre moved.
I guess we have an elastic block store like thing that we call crucible and crucible data is stored in triplicate and there's no affinity with respect to where your VM is running and the only affinity is that it's inside the rack.
So you're talking to three separate.
We've got 10 NVMe drives per sled.
Your storage is sitting on three separate drives elsewhere in the rack effectively.
So we can, you can be migrated to another sled without consequence in there.
Interesting.
And you have to do this with such a low latency, the replication because like it's.
Yes.
Yeah.
And we, and actually we did.
I mean I'm a big fan of podcasts, obviously you are as well.
So on our.
We've got our own Oxide Friends podcast.
Been a lot of fun and get the team talking about things in their own voice.
And so we did an episode recently on Crucible and getting into all of the challenges of making that high performing and low latency and still has to be obviously high avail, highly available and reliable.
And then you need to be able to, you know, when it comes to storage, it's not the sunny day stuff, it's always the cloudy day, it's stormy day and hurricane kind of stuff.
And you know, how do you know, a sled is removed or you.
A spindle has been transiently removed, it needs to be resilvered and there's just a whole bunch of stuff that need, that you need to get right when you're building a storage facility.
And but with.
We've, we've done all that and built it in.
I mean it's very important to us that like that is included.
That's not kind of an add on.
And this is CFS derived.
I don't want to dive in too much in.
This is a.
But it sounds, I would imagine it's some CFS derived.
Yeah, it is.
We are using, I mean using zfs perhaps in a slightly different way than one might imagine.
We are not using, you know, part of the power of ZFS is taking this volume management abstraction and building it into the larger file system.
We are not run.
I mean I have run many times in the past where you're running mirrored or striped across a group of disks.
What.
We're not using ZFS for that.
We are using ZFS on these individual drives to give us the manageability that you get with the file systems.
The unit of administration and management is very powerful.
And then we also get the thing that I'm just not going to go away from is the checksums that we get with zfs.
And in fact those are indirect checksums.
It's like, yeah, I'M just not giving that up.
Sorry.
I so very much built on ZFS at the foundation, but we are and then and have built on things like ZFS snapshots and so on to do our own snapshotting.
The ultimately it's a software system that we've built on top of.
Yeah, cool.
I mean we start.
Need to start wrapping up here, but I kind of want to end on a few kind of industry trend changes that I was curious about your thoughts on.
I caught your episode on the Changelog and I was listening to that and about like this whole like pendulum of cloud versus On Prem.
Right.
That we see every.
Oh, it's not the first time we see the debate.
Like I'm sure you saw the whole like debate or like I guess it's a bit clickbait from dhh.
He did his whole like, oh, we're moving back to On Prem, Cloud is bad, blah.
Your pin is a bit more nuanced and a bit more pragmatic and less clickbaity, I guess.
Like, like how do you see this pendulum swinging a little bit?
Yeah, I mean I think that first of all, I think the importance is that we want elastic infrastructure.
API driven infrastructure is just an advance.
That's what we all want that and you want that in the public cloud sometimes when you want to rent it and there are times that you want that on premises.
And I think that the core belief of Oxide is that cloud computing isn't just for the public cloud and that there are reasons and you know, there are people and you ironically, when we started oxide, you know, VCs thought that like, you know, look, technically we think this is like, we know you can do this.
And I'm like, what technically, I must not be explaining what we're doing because like, technically this is outrageously ambitious.
Almost like foolishly ambitious.
But you know, the VC is like, oh, technically, I don't think there's any risk here.
You don't want to be like, hey, let me stop you right there and explain why.
This is like, this is like loaded with technical risk.
This is like technical risk coming out of every orifice and dripping on the floor.
We're building Apple from scratch, but we're.
Building Apple from scratch, by the way.
So like, I don't know how you get to the.
And by the way, you didn't even ask us if we're doing our own switch because PS we are going to do our own switch.
And it's like.
But like just everywhere you look, it's technical.
Right?
But they're like, oh, technically we think you could do this.
You're like, okay, but we don't think there's a market.
And you're like, oh, it's like, it's a.
No, it's like, first of all, technically, I don't know if I can do this or we can't do this by the way.
So like.
But let's not dwell on that.
There's very much a market.
And I think that people don't see those folks that are on prem because they need to because of economics dictate that they're on prem, or latency dictates that are on premises or security considerations or risk management dictate their on prem.
And you know, one of the things that we've always known is that the, you know, folks that are operating, you know, for the public often that it would say, like governments need to actually own their own infrastructure they can't rely on because you can't have a situation where public infrastructure is turned off.
And as a result, you know, a government entity is turned off.
Right?
The.
And those folks have to run on prem and they are cloud educated, like because they've grown up with cloud computing and they are in a colossal amount of pain.
So, you know, that market, the market very much exists.
And I mean it's VMware essentially, right?
That's right.
And they want and deserve cloud computing.
And modernity should not be reserved for the public cloud.
It should also be allowed to be consumed as a product that you buy.
This should not merely be for rent.
And I think part of the problem is especially at scale.
I mean, we think that like the security considerations, risk management latency, those are the ones that exist today.
Like if you are in an air gap facility, you want cloud computing, like you're real interested in oxide.
The ones that are coming that are starting to emerge and are absolutely going to be coming tomorrow are those who, companies that are born on the public cloud, grow to the on the public cloud, have product market fit and realize like, I can't be renting all this infrastructure.
I've got to economically own some of this.
Like, we can't all be vassals, you know, we, you know, we.
And I think that's part of what DHH is tacking into of like I need to own my own fate.
And you know, there is something that is not, it's not just economic, it is just like it is the, not just the financial security, but the kind of the risk management security of Like, I can control my own fate, I own my infrastructure.
It's not a fit for everybody, but it's a necessity as you get to scale.
And I think that people are beginning to realize there was this myth for a long time that cloud computing was a terrible business that I, I feel that Amazon did such a brilliant job of convincing everybody, like, oh no, no.
Like it's a very low margin business.
It's really terrible.
You don't want to be here.
And then you're like, wait a minute, this is actually a wait, aren't you the same company that says your margin is my opportunity?
And like this is like, actually this is a really good business, like for you.
But now I'm having a hard time building a business on top of it because.
Because my margin is your opportunity and the companies need to take that margin back.
But I mean there's.
Yeah, I mean I think you made a good argument in that podcast about like cloud is great if you're small.
Right.
And the reality is that there is a tipping point at some point where bringing it on prem makes sense.
Right.
But I think a lot of people, particular people who are I guess, not experienced in building and managing and running cloud infrastructure about under indexing on the complexities of actually managing hardware like security.
Like, oh, rack number four, like this unit went down.
Oh shit.
Now it sends remote hands to replace this unit.
Like, oh totally.
It is kind of comical when you talk to people because people who have grown up on the public cloud have just kind of assumed that if I ever need to go on prem, like, I don't know, Dell makes the thing that Amazon offers, so I'll just go buy it from Dell.
And they're like, oh my sweet summer child.
You know, they.
And they begin to realize like, no, like Dell, what you are buying has nothing in common other than like the cpu.
Yeah.
Now two of your drives are broken in your second drive raid.
So now like, well, you one drive short of like a full corruption.
You have five hours.
You five hours.
Yeah.
Good luck.
And the, and resilver that in production.
Right.
And I mean just the software that you're, I mean there's so much that there's such a terrific gulf between what you can buy and what you can rent.
And that's what we're trying, we are trying to offer a product that is able to live within that gulf and allow people who are only cloud experienced to be able to deploy this infrastructure in a way that like makes sense.
It's like, yeah, folks, don't need.
I mean, I mean we, there is no IPMI in our system in general.
Anything that has like intelligent as part of the acronym, you know, it's like compensating for its total lack of intelligence.
But the.
We don't have any of those things and you don't need them.
And so what you get is a system that's like why would it have to be more complicated than this?
It's like, well, historically.
But we have a system that people who have got experience in the cloud can actually operate for themselves and don't need to have ramp up a new team to actually go build out.
And I think that's where the real value lies because I mean we have very short memories in particular when you forgot how it is to do trips to data center to fix things and then you're like, oh right, I don't go into the public cloud.
Very nice to have Google doing those rather than Amazon.
That's it.
And it's like those.
And they've done a, you know, to their credit, Amazon has done.
And then followed by, you know, Google, Microsoft.
But they've really done a really great job of serving their customers and generating a pretty terrific product.
And it's a lot of, I mean to be able to generate that product on premises is really tough.
And it's something for them.
Right.
Because AWS has a thing called AWS Outposts that's you know, kind of out of character for them, is a bit of a wreck and it doesn't really solve the problem that customers have because it is actually really hard to deliver all this stuff on prem.
To deliver a distributed control plane in a form factor that is a product is really hard, presents a bunch of new challenges.
Absolutely.
And I think just to wrap up this conversation because there are a bazillion things I would like to cover about it in the interest of time.
I know we can't go on for much longer, but the last thing is something you made an argument for which is I never thought about from this perspective.
But you mentioned that Moore's Law is slowing down and Capex makes more sense.
Right.
And it's an interesting argument for going on prem again and obviously that Moore's Law might change again in two years.
Who knows?
Like we don't see who knows what happens.
Right?
Well, I mean I can't predict the future.
There might be a groundbreaking breakthrough in a few years where like it kicks off again.
Right.
And we see faster density of these things again.
But I think that was a really Interesting argument of looking at the current landscape of where we are at least right now with the current trajectory.
Right.
Well, I think also the question is too, you know, in a world where Moore's law is slowing down, which I think is somewhat indisputable, how long can you run a cpu?
Like how long does it last?
And the answer is it lasts a really long time.
Yeah, really long time.
I mean this stuff is Solid state and CPUs are like some of the most reliable stuff we've ever made.
We humanity.
Right.
I mean you kind of think about how crazy it is that these things run at hundreds of watts, dissipating hundreds of watts, executing billions of instructions a second and they can do it for years and years and years.
Really, it's remarkable stuff.
Thank you.
Gordon Moore and the pioneers of semiconductors.
Yeah, but these things can run for a really long time.
And you know, the longer these things can run, the less sense it makes to rent them, the longer you're going to run on them, you know, that I'm going to run.
And you know, interestingly, you know, AWS Amazon just extended their own depreciation and the this is like a bit of an accounting trick that led to them having a much better quarter.
Right, right.
But what Amazon is saying like oh by the way, we're not going to throw these things out actually we're going to run these things longer ourselves because they last longer and we don't have to discount that, but we don't discuss that is not.
And I think we are not.
You know there were those all those years where every reinvent was a price cut from Amazon, not really in those years anymore.
And certain things like bandwidth they never cut price on.
But the, we don't really live in that era anymore.
And right now the dividend of the fact that these things are so reliable and run for so long is going to the owners of them in the public cloud.
And we really think that some of that needs to accrue to people that are trying to actually run compute.
They should be able to do that.
I think that's a fantastic note to end on and I think there are plenty of things I would like to explore further and maybe I can have it back on the show in the future, dive deeper into those topics.
But absolutely Nerding out is right in the title.
So I'm happy to nerd out in any of the many, many dimensions of what we built.
So we'll have to do it again.
Amazing.
Thanks Brian.
Have a good day.
Talk soon.
You, Victor.
Take care.
Cheers.
Found an error or typo? File PR against this file or the transcript.