Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Daniel Stenberg on Curl's Journey: From C64 Demos to Internet Transfers
Play On
28 JUL • 2024
1 hour 26 mins
Share:
Welcome to another exciting episode of “Nerding Out with Viktor!” where our host, Viktor Petersson, sits down with Daniel Stenberg, the brilliant mind behind Curl, a powerful command-line tool that has revolutionized internet transfers for developers worldwide. This engaging conversation delves into the fascinating world of open-source software, security, and community building, offering valuable insights and takeaways for tech enthusiasts.
Daniel’s journey in programming began on the Commodore 64 and Amiga, where he honed his skills in the demo scene, creating demos and forming a demo group with friends. This experience laid the foundation for his deep understanding of programming and computers, eventually leading him to Unix systems and IBM, where he developed Curl. With over 26 years of maintaining an open-source project, Daniel shares his expertise on community building, reducing friction for contributors, and creating a welcoming environment for new developers.
One of the most entertaining aspects of their conversation is when Daniel recounts some of the craziest support requests he’s received, showcasing the widespread use of Curl and the amusing misunderstandings users sometimes have about open-source software. His patience and good humor shine through as he shares these anecdotes, offering a glimpse into the real-world applications and challenges of using such a powerful tool.
The conversation also delves into the future roadmap of Curl, focusing on the adoption of new standards and the importance of backward compatibility. Daniel explains the careful consideration that goes into deprecating features and ensuring minimal disruption for users, highlighting the complexities of supporting protocols like HTTP and HTTPS. He also discusses the measures Curl takes to prevent supply chain attacks and ensure the integrity of its code, making security a major theme in their conversation.
Daniel shares his experiences with handling security vulnerabilities in open-source projects, providing insights into the challenges faced by maintainers. His dedication to maintaining the highest standards of security and quality is evident throughout the discussion, emphasizing the importance of corporate support and community contributions in sustaining open-source projects and ensuring their long-term viability.
The episode also touches on the future of open-source funding, with Daniel sharing his thoughts on how projects like Curl can secure financial support. He emphasizes the role of sponsorships and community contributions in sustaining open-source projects, making this conversation a must-listen for anyone interested in the inner workings of tech companies and the innovative ideas driving their growth.
Join Viktor Petersson as he delves into the fascinating world of open-source software with Daniel Stenberg, a true pioneer in the tech industry. Whether you’re a seasoned developer, an open-source enthusiast, or simply curious about the journey of one of the most widely used tools in software development, this episode is for you.
Transcript
Show/Hide Transcript
Welcome back to nerding out with Victor.
Today I have a very special guest, a guest that was on my initial roster of people that I wanted to have on my podcast when I started scoping this out back in December of last year.
It's Daniel Stenberg.
Welcome.
Hello.
Hi.
Thanks.
And for those not familiar with you are the founder of a tool that I would be very surprised if not every single guest on the.
Every.
Sorry, every single viewer of this podcast has used in one capacity or another.
Which is curlae.
Exactly.
Yeah.
And for those, surprisingly, who have never heard of Curl, how would you describe Curl?
I always struggled with how to describe it because Curl is a command line tool that just does Internet transfers, uploads and downloads specified as a URL.
And of course, it supports 28 different URL schemes or protocols.
Well, that's what Curl is, the tool.
Then, of course it has a library to libcurl, which is usually the thing that gets me to cover more installations everywhere because that's a component used by a lot of others.
So it's running in a lot of places doing Internet transfers for the world.
Really?
Yeah.
I would describe Cross as one of the most amazing swiss army knife tools for the command line.
That is a tool that I use religiously myself and I love it.
So, yeah, thank you for creating such a great tooling.
So I want to talk a lot about Curl, but before we do that, I kind of want to talk about your early days because I think there's a lot of similarity with people who have a similar track record as yours that have been amazing engineers over the years.
And your engineering background really started in the Amiga world.
Right.
And then the demo scene.
So maybe, Phil, people about your background, like how you ended up on this.
Track really well, I started with the Commodore 64.
Oh, the C 64.
Oh, right before them, yeah.
So, yeah, the Commodore 64 was really the computer that got me interested and sort of fascinated by programming and computers in general because, well, I had.
I had a friend there in my school, so I sort of got the taste for it and we gathered it at his place and, you know, entered programs from those early computer magazines, you know, with those data things.
You had to type it in from a magazine.
Took forever.
And then it was wrong in the end, but you didn't quite have any linting then.
Well, some of them had some checksum, so you knew that it was wrong, but some of them didn't.
It was just ended up just one of those numbers was wrong somewhere in all of that.
Well, so anyway, that got me sort of into it.
And then me and my little brother, we bought Commodore 64 in 1985 and that.
And then I actually, I was interested in programming pretty much from the beginning, not that I could, but, you know, I.
Back then, they shipped with manual somehow basic was built in.
There were some basic manuals how to do that so you could start making some, you know, guess the number programs or really silly basic things.
But yeah, that got me into that.
And very soon, well, my friend angel who had the computer before me, he and I started, well, look at, there's this thing called demos.
People are doing how, you know, flying things, music and colors and stuff, and how do you do that, actually?
Oh, right, there's something called assembly.
You have to write assembler to do that.
What's that?
And then we just got into that.
And then we got into that demo scene on the Commodore 64, and we made a lot of demos, created a demo group, and worked on that crazy amount of time for several years, actually.
Yeah, I mean, I remember SFL sweet Dreamhack back in the days when Dreamhack was actually mostly demo scene in the early days, and they were filling up arenas with this 64K demos and, I mean, it's amazing.
And the amount of super high caliber engineers coming out of that scene is just fantastic.
Right?
Yeah, yeah.
It was a really good school and a community and everything, so you pushed yourself to sort of try to show off, but you also learned tricks and how to do that, and it was a really good way to get into computers.
I really sort of attribute a lot of what I know and my interest in to those early days when I learned exactly how to, how everything like that worked, how to write assembly.
You know, you get a different sense of what memory is on how computers work when you actually write assembly.
Exactly.
Instruction by instruction and assembly on those early day computers and eight bit things, they were.
I mean, it wasn't easy, maybe, but comparison to modern computers, they were easy.
So learning assembly one of those more stupid devices is actually a pretty good way to learn assembly, because things were easier.
You could, you know, you could actually count how many cycles a program would take because you knew how many cycles each instruction would take and so on.
Yeah.
And I had even Upton on the podcast a few episodes ago, and we talked about, I guess, the lack of people.
I mean, that was the original of the raspberry PI, right?
Is that he found that there were very few people that actually understood that level or that low level of engineering that the C 64 and the like, symbols, computers in the same cohort really fostered.
And so I think it's.
I'm glad to see that there are new tools that kind of addresses that because we had an era.
Well, everything was high level and nobody really understood what actually happened under hood.
Right, right.
So.
And then obviously that led into the BBS era, which I guess you ran a BBS as well in those days, right?
Yes, I did.
Or did together with my brother.
And we wrote our own BBS software there too, because why not?
So, yeah, it was fun.
It was a fun era of early communication and message boards and stuff.
And a little bit like what we would later learn, the Internet could be like communicating and discussing things with people in a greater sense than just the little you and your closest friends.
So, yeah, that was great fun.
And then because I switched from the Commodore 64 and then I started to working on the Amiga and Amiga suddenly was not as fun to do demos on because, I don't know, when I did demos on the Commodore 64, it felt like a small little sandbox.
Everything had some sort of borders.
It was hard because it was small and tiny.
You had to sort of squeeze in things to maximize.
And then it.
It felt like when going to the Amiga, everything was completely.
Well, it felt like that, well, everything is possible on this based, you know, sixty eight k, eight megahertz.
This is what a monster.
So it felt like, okay, and completely different assembly.
It was 68k instead of the 6502.
So it wasn't really that fun to do demos.
So me and my friend, then we started let's do a text editor, because we thought that existing text editors were not as good as they could be.
So why not make text editor?
And at the same point in time, basically I started working at IBM.
And at IBM I got into contact with AIX, the IBM Unix version, on Rs 6000 machines.
And on those we ran emacs.
And that sort of opened up my eyes for, wait a minute, emacs where everything is sort of programmable in lispenness.
I didn't really like Lisp, but I figured, well, this is sort of.
I got a lot of inspiration for our text editor.
We called Freqsad from Emacs at work, then worked on that on Amiga at home.
And then eventually I transitioned into working on Unix directly instead of doing on.
I just dropped the Amiga thing and went into working on Unix machines.
And I think that has probably come in very handy now as the array of supported devices for Curl is insane.
Almost.
The breath is insane.
Yes, it is.
It's super crazy, actually.
What's the strangest architecture you've had to port to curl, I guess Libcurl, more so than curl I would imagine.
Actually.
it's not that, well, there are, they're really not that strange when it comes to Libcurl because Libcurl is very low level.
It basically only runs on top of a TCP IP stack.
And the POSIX style API that was made, I don't know when in this, I don't exactly, I don't know my history there, but it was founded long before I started this.
70 something.
Yeah, exactly.
So mostly all operating systems offer a very POSiX style, like almost at least for doing networking.
So usually it's very easy.
So usually there's just some quirks here and there include files and exceptions and tiny things.
So usually I haven't.
I used to say that because I truly think so, that windows is actually the most quirky operating system to work with because that's the single operating system that is the most different to all the others and yet has a lot of users.
So it's still a relevant one.
So we still have to make it work.
But that's actually the one I think I struggle the most with, possibly also because I'm not really a windows guy, so I don't really know all that sort of.
For me it's always an uphill battle to work on the things there.
So otherwise almost everything else is more streamlined around whole six API and how to do things.
Fair enough, fair enough.
So let's talk a bit about obviously building and maintaining curl because that's obviously, it's a beast of a project in the sense of not only complexity, but also like community.
Because you have obviously maintained this for why is it 15 years or whatever it is, you have maintained it now.
Which is not trivial, for 26 years at least.
Oh wow.
Yeah.
Okay, so I'm fully off of my research here.
Building and maintaining a healthy community is in the open source world, obviously.
I guess GitHub made it easier and harder simultaneously as we moved away from mailing lists and whatnot.
Right.
What have you found?
Like dealing and maintaining a healthy community?
I'm curious about your insights on that because that's.
I would, I think curl is regarded as a very healthy ecosystem compared to what many other projects.
Well, for example, I think, yeah, I worked really hard to make sure that we reduce friction and make it as easy as possible for anyone and everyone to participate.
That's one of the reasons I still argue in favor of GitHub, even though there are reasons to be against it.
But the fact that so many people are already on the platform, they already know how to work on the platform and how to do things on the platform, like how to make a fork and make a pull request.
If you just go with that concept, you lower the bar quite a lot for so many users.
And I want it to be like that.
I want it to be easy and possible for anyone to fix whatever problem you find.
If you find a typo in the documentation, I want it to be simple for someone to just fix that typo and send it to me.
And I don't want it to be any obstacles or bureaucracy or anything in the way to do that or for me to accept those changes.
So that's the goal here, to make sure that everything is streamlined for those who want to participate.
And then of course make it possible for people to do changes again, not just that single type at once.
Right.
But when you've dipped your toe in the water, sort of make it possible for them to, hey, grab the next issue or help answer a question or join in the team.
But also I should call it sort of, we don't have any, there's no steps in the ladder.
It's not like you're joining different levels and then in the end you get to 100 or anything, you're just, you decide how much you want to be there and you can participate a lot if you want to, or just a tiny teeny bit if you want to.
That's up to you.
And of course I want to make sure that anyone can participate to the level they want to.
And if they want to do a lot, I want to help them do that and join in, because we always have a lot of things to do.
So of course we always welcome contributors as any open source project.
How many active contributors do you have?
I guess active is a subjective definition, but however you would define active.
Exactly.
So that's.
I often then struggle with that.
People talk about communities, right?
How big is a community or something?
Because it's nothing.
There's no clear boundary anywhere.
When are you in?
When are you out?
So there's no, you don't step in or out.
You're just, everyone is in and out at the same time.
So I don't really know.
But if we count, for example, number of commit authors per month, I think we're around 25 on average month.
I mean, that's a very healthy ecosystem.
Yeah, I'd say so.
And it's not the same 25 every month either.
So we have actually a pretty steady stream of new committers.
I think we're at around ten per month that are new authors.
So it's quite a lot of people coming in and of course a lot of them just do it once and never show up again.
But at least it tells me that it's possible.
So I know that the door is open, you can enter if you want to, and then if you don't want to stay or if you're happy with that single change.
I can't do much about that.
Sure.
Yeah, absolutely.
Let's talk a bit about the community, because contribution is one thing, but then obviously, at least from my experience, an equally big burden, if not bigger burden, is the actual maintenance of issues and replying to feedback or issues in general.
On GitHub, how do you see that?
Have you maintained that in a healthy way?
I guess because it's extremely difficult to do without a lot of patience in the long run.
I guess it's a polite way to put it.
That is certainly true, yeah.
So it's one of these sort of patience testing and character building exercises to year after year, just.
And of course it's that way that people usually don't say anything when things are going well and everything is fine and dandy.
They just take the code, go, and I never hear from them and they never tell me anything.
That's good.
But the moment something is not working the way the person thinks it should, then they come.
So basically it means.
Sorry.
Yeah, curl, 26 years ago, 26 years of people complaining on things that don't work the way they think they should.
So yeah, it is an exercise in being patient and trying to ignore these slurs or the sort of, I often try to view them as, you know, cultural or language differences.
So, yeah, maybe that insult wasn't intended, it was just a bad way of using the english language.
But of course it's.
Sometimes you just have to bite your tongue and wait a few days until you respond because it's just someone who hasn't understood how to behave in public.
Absolutely.
And I mean, I guess in particular, in the open source world, there are two levels of, I guess entitlement might be the right word for it, but there is the open source entitlement of, I need this feature, build this for me because I want it.
Right.
Not like they want to contribute, but rather you go and build it because I want it for my job.
Exactly.
And really?
Yeah, you gave me this now and it doesn't have, you know, it gave me 99.95% of everything.
But this last little thing doesn't work.
Go fix it now, because my company depends on this and we have a problem.
Please ship it tomorrow at the latest.
We've already proven to our customers, so please go do it.
Exactly.
Otherwise we will be sorry.
So, yeah, please spend your spare time now and fix our problems.
Yes, there's some amount of that.
And how do you.
I mean, I guess that's, I guess as a test of patience again, really, it's just doing that in a way that, because I guess you have to bite your tongue quite a bit in these.
Yes, and sometimes I fail at biting my tongue.
And also since five years ago now I work on curl full time, so I sell support then to business and whoever wants to pay for support.
And so then nowadays I have a better answer also.
Okay, if your commercial user is actually depending on this, you can actually pay someone.
Well, I don't have to say it's me, but to fix this issue, we can probably have it fixed by tomorrow.
But if you're just sitting there unwining and your company is suffering, maybe just asking volunteers to spend spare time is not the way to do it.
Yeah, and I want to dive into that because I think that's a whole much bigger question around open source and funding in general.
So I want to save that.
But a pin in that particular conversation a little bit.
But I think that's interesting for the community.
The other thing that I find, you've been posting these for a while, I think on Twitter and whatever, on social media around these strange support requests you were receiving for users to please uninstall my curl.
Everything from cars running curl.
And they found your email address.
Talk to me about the craziest example you've had of people reaching out to you.
Yeah.
So since curl runs in, well, I estimate I've used the number for several years now.
So I say 20 billion installations, which I think is nowadays a little bit.
I think there are more, but it doesn't matter.
It runs everywhere in.
And the license says that you have to ship it, you have to mention the license somewhere in documentation or in products.
And of course that means that in a lot of products you can't even figure out that it actually runs curl.
But in a lot of products at least they put the license in some kind of about window or, you know, third party license things.
So somewhere you can find the curl license.
And in the ker license there's my email address, which is actually rare among software licenses.
Yes.
And that is important only because if you then go to that weird page with a thousand software licenses, you can scroll for 45 minutes, and then suddenly you find an email address among all of those.
And that's my email address.
So that at least that's my leading theory on how they come to email me when person has a problem with a gps in their car.
I can't figure it out.
How do I do this?
Probably they try everything and they end up on that screen eventually.
And then they find my emailer.
I'm going to email that guy and ask, gps problem.
So out of the blue, I get a question about GPS in a Toyota Corolla.
From often they also.
Then, of course, they don't tell the whole picture.
They just tell me something about, hey, fix my problem here and tell me a model number or something.
I have no idea what they're talking about.
And please go ahead and blah, blah.
Usually that end is more amusing to me than that person because that person is probably most upset or angry even.
That's.
And when I try then to excuse myself and say, well, you know, I have no idea what you're talking about, I only.
It never goes well because that's not the answer they want.
They want me to fix their problems.
Yeah.
Yeah.
So one, in my, from my view, one of my more amusing cases was this woman who emailed me about her Instagram account was hacked, she claimed.
So why do you email me about your Instagram account?
I have no idea what are you talking about.
You can.
You should contact Facebook at the time, this was five, six years ago.
But then she got back to me and said, well, your name is instagram.
Exactly like that.
Right?
In the.
In the about window, it says, you know, curl down, hdmi.
Oh, right.
I had no idea.
Fun, you know?
Cool.
That's like a billion installations.
Suddenly, I thought it was really fun.
She did not think that was very fun at all.
So.
But she insisted.
No, no, I should just talk to my friends at Instagram.
They should help her to fix her account.
And, you know, back and forth.
I tried.
No, no, I have no idea.
I never talked to them, never met them.
They never told me even that they used this.
I had no idea until you told me, you know, I'm just an open source guy, blah, blah.
And then she got silent.
And then I think about a week later or two, she emailed me again.
Oh, you've been lying to me.
Because look at this, your name is also in Spotify.
And then she sent me another screenshot with my name in Spotify on the same phone, right.
That cannot be a coincidence.
My name in both Instagram and Spotify, the same phone.
That's the only evidence you need that?
Of course.
What more do you need?
So now unhack my phone immediately before I tell these companies about you.
Then you just give up, right?
What do you do?
There is no coming back from that, right?
No, but it is amusing, obviously, for us, who's living in this world for a long time, and for us, it's so obvious.
But for the average user, have no idea what open source is.
Nor curl.
Right, right.
No, no.
I've had the fullest sort of sympathies for them because it's a really complicated world.
And for them to understand this, I understand that it's really hard for them to even accept it or buy it when I say something, but.
So nowadays, I usually never even reply when they email me about these weird things because it's so hard to even.
I don't know what to do.
They don't believe me or they don't trust me or they.
Anyway, I can't give them the answer they want anyway.
There's no upside.
I collect them on my site now and I have a collection of my funniest emails nowadays.
Actually, that's where I've seen it.
Yeah, that's.
That must be it.
What?
I mean, what's the amount of emails you get per day for that?
I mean, is that crazy or are they.
Yeah, no, it's not crazy.
So actually, it's not even one a day.
So it's more like one strange email.
Well, most of them are not fun at all.
Most of them are just confused and then I just don't care about them.
Some of them are slightly amusing or scary, but there's more like a few per month maybe.
So it's really not a load of any sort.
All right, that's fair.
All right, let's go back to curl for a bit.
And I was really curious about a few things about this, and one of them is the future roadmap I had.
And in terms in particular, let's focus on, well, the standards moving forward.
So I was chatting with some friends before I hopped on this, and one of the people asked me, like, how do you assess which standards you want to adopt in curl?
Is that driven by user demand or is it driven by you?
Seeing the future of, like, this is really interesting.
I think this has weight, or how.
Do you see that it's driven by all of that and the rolling of a dice, basically.
So it depends.
So yeah, I try to follow, for example, since curl is used for Internet transfers on the Internet, and a lot of curl use is like mimicking what a browser can do.
So instead of doing it with a browser, you can automate it with curl.
So that's one reason to make sure that almost everything you can do with a browser, I want it to be possible to do with curl.
Right.
So if the browser suddenly adopts something new, crazy fun something, then usually we also go that route because then we know that it's going to be the new thing by a lot of users on the Internet or the web.
So typically things that are adopted by all the browsers, we also do some of those things.
You can figure that out pretty early on that, you know, and all the browsers are on the train, they're going in that direction.
Of course we are going to implement that as well.
Like the HTTP versions, right?
All the big ones are going to adopt the new HTTP version, so it's not a bad bet that we are going to do that as well.
And I guess you're constraining that to ones that are official standards.
So, I mean, chrome is kind of playing a bit rough and dirty here and invent standards themselves a bit.
So I guess.
Right.
So I don't want to go sort of on a tangent on doing silly things either.
So, no, I want them to be real standards and with a sort of with a goal or sort of with a future that we think that people will use.
But of course that's hard to know.
And also I kind of enjoy being sort of early on the train as well, so that we can help out to, like, I've been early on HTTP two, I've been fairly early on HP three, so that we get a tool to start also testing out the versions fairly early, perhaps even during the protocol development, so we can get some stuff early on and try it out and see does this actually work in reality?
So that's kind of fun, even if it also means that sometimes it's a lot of effort that we don't have to backpedal and do something differently because standards change and so on.
Yeah, I can imagine.
So it really matters on a personal interest and who wants this.
And if companies show up and say, hey, we want to do this and someone pays for that development, that's also of course a strong motivator to go push for that because some things, if there's no company supporting stuff and it's a huge effort, then maybe we just, yeah, we might want it, but maybe it's too big an effort for us to just do it ourselves.
So we just wait and sell and see.
Yeah, because you're also building tech debt in adopting something potentially.
Right?
Exactly.
And it's really horrible to adopt something and then later realize that everything, everyone is going to just abandon that anyway.
So why did we do all that effort and support it forever after that?
So no, we also want to make sure that we don't do too much of silly things that we have to just carry with us for all eternity.
And that brings me to another question, which is about backward compatibility.
You guys have been very good at that historically, but I guess at some point in the future protocols will be dropped, I would imagine, because either they're not used or is there philosophical stance that nothing ever will be dropped.
How do you see that?
My philosophical stance is more like if no one notices, then when we drop stuff then it's okay.
It's not a behavior break.
If no behavior is actually broken.
So if no one is actually using it then we can actually remove it.
But if you have no telemetry to tell you that.
Exactly.
So that's a sort of.
Yeah, that's a weird answer because I actually don't know.
There's going, whatever we do, there's going to be two people somewhere in that exact, there are using whatever we have done ever.
So we are really careful about removing support for anything.
So we don't remove support for entire protocols.
We can remove support for some stuff, but for example if we have things that you can enable in the tool or in the library ask for this to be used, then we didn't guarantee for it to be used.
Right.
So we can not use it because you can still ask for it.
It'll just never happen and things like that.
So it is possible.
And we also have dropped support for a lot of different third party dependencies and offer new third party dependencies instead, which so from the outside is still the same, but internally we stopped supporting some TL's libraries and stuff like that because they were, they die off.
Okay.
And then do you have a protocol for, I guess deprecation, notifications and stuff like that in the CLI or how do you communicate these?
Yeah.
Yes.
So we have a procedure in the project how we do when we remove things from the project.
So we have a pretty long time so that we warn and alert about when we want to do changes.
So at the minimum, six months ahead, we sort of document exactly what we want to do.
What the outcome is.
And if you have a problem with this, you can always yell about it and we can sort of bring up the discussion and either postpone it or just cancel the precision completely or whatever, because everything is always up for discussion.
So we can always change our minds as long as anyone wants to do.
And it's also open source, so you could always fork it and just keep it the way it is if you so desire.
Right?
Yeah.
And there's always a matter of sure if you show up and you want to be a contributor and help keep the code in shape and everything.
So of course that changes the equation.
Right.
So then of course we can support whatever, if there's someone there who is actually using it, have a use case and make sure that we have test cases and everything is good, then I, then we don't remove it.
But do you still sign off?
I usually don't do that.
Yeah, exactly.
Do you still sign off every PR coming in?
No, I don't.
So I review almost every PR, I think.
But we are several committers and so there are a few bunch of other people who are actually merging quite a few changes.
So I think I'm at 56% of all commits authored.
Okay.
But it's, my share is sort of shrinking over time because we have a lot of other good people who are doing a lot of development.
But you are the only, I guess, paid maintainer of curl.
There are no other paid maintainers to work full time on it.
More.
I'm the only full time paid maintainer, yes.
So I've actually off and on, I hire a few of the others, depending on what kind of assignments I have and what kind of money I have to spend on them.
So I actually pay a few of the others on and off.
Okay.
Okay.
What was the biggest pain in the ass protocol to maintain, or what is, or was, I guessed.
Curl is HTTP and HTTPs.
They are the two big protocols that people use for curl.
And with curl they're also super big and super complicated protocols.
And since those two go hand in hand, right, because it's mostly most used and big complicated protocols, that means that we have most features with them and most of edge cases and most users doing things in crazy ways because there are such big protocols used in so many different ways, I think a fair amount of everything we do is around HTTP and HTTPs.
So even though they are the most used and most tested and most well maintained, they're also the ones that are causing us the most problems because there are so much, they're just very complicated.
They started out easy.
Well, yeah, they were probably easy back in the early nineties, but they have developed into beasts really.
And now with so many different protocol versions too, and different combinations of those and enter proxies and proxies of different versions and then all those different headers and things you can do with the content with different versions in different ways.
Yeah.
It's a never ending sort of challenge to keep all of that together.
I'm curious, obviously in the last five years, Cloudflare has eaten up so much of the traffic and acts as an intermediary for so much traffic.
They do interfere with headers, so they do add their own headers and they do modify how packets flow, I guess.
Have you seen any, I guess, positive or negative change as a client with curl since that change has happened over the last few years?
I think in general, the entire trend with everything, everyone going behind cdns, I think in general that makes things easier for me as a client because it reduces to sort of the plethora of different implementations and different servers.
So we're going into a future with fewer server versions, so less diversity really among server implementations and crazy things that are happening in the server side that we have to adopt to as a client.
So I think in some ways it's easier, but in most ways it doesn't really matter to us.
It's the same thing because they appear as servers for us.
Right.
So we have to work with any server, and if that happens to be a CDN, it's just a server as another server to us.
Yeah, I was just thinking if they sanitize the data in some way, that makes it easier for curl to ask.
I mean, I guess they would have to swallow up those edge cases on their end rather.
Yeah, I guess so.
But yeah, but I guess, you know, if they take 25% of the Internet, so sure they make that 25%, perhaps a little bit more cleaned up, but we still have all the other 75% served by others.
And if there's a server out there that is sending a crazy header somewhere and that's used in production, you can be sure that we will see that at some point and then we have to deal with it, even though Cloudflare or the other CDNs will hide them for their particular clients.
Fair enough, fair enough.
What's the craziest use case you've seen of Curl so far, where you're like, wait, what?
One of my favorite ones that I actually learned about, I think, was about a year ago when some we shouldn't names but they made their niche printer manufacturer.
They actually used Curl as an internal IPC in their printer.
So they actually did curl communications between the different processes in the printer.
That's an interesting use case.
I thought it was crazy.
Completely non ideal.
But they did it.
It feels a bit like if you got a hammer, everything looks like a nail kind of situation.
Yeah.
And they contacted me because when you use libcurl, you basically set up so that it'll automatically keep x number of connections alive after you've used them.
So that when you want to connect to that hostname again, it'll have the previous one still alive.
So the next request will be much faster.
Right, right.
And they had like, they had that maximum number of connections to, I believe, five or six or whatever, and they had, I think, seven or eight nodes within that printer.
Oh, right.
So basically they communicated with tiny amount too many, so they would close down connections all the time.
So they had to close them and create new ones all the time.
So within that, even within that silly little printer, they have to sort of create and close down connections all the time.
So it's totally ineffective and really slow.
So it was a bad idea to begin with and then use it in a bad way.
That made it even worse.
Any other crazy use case that you can spring to mind of that you've seen?
Actually, not that many, because I think a general trend is that everything is talking network these days.
Right.
So.
And with everything is even talking even more networks tomorrow.
So everything is potentially going to use curl sooner or later because everything is going to be networked in some way or another.
Yeah, that's a fair point.
So therefore there's not a single product that I can't imagine having curl eventually.
Yeah, fair point.
One interesting, I guess spin out of curl.
I guess not spin out is not the right word, is w curl, which came out, I think it was a debian project, right?
Yes, it's the alias actually.
Yeah.
And it's funny because I only use wget for downloading files, but I use curl for everything else and for the simple reason that I never forget if it's capital o or lowercase o for output file.
And that's the sole reason why I use wget.
We get told that quite a lot, quite often.
So, and I think this particular project, that wcurl is just a shell script, right?
It's a shell script wrapping curl.
But Samuel, who created it, he, I think he read a lot of feedback from our recent curl user survey where as usual, a lot of people pointed out exactly that.
Remembering exactly which option you should use to just get that URL down to a file is complicated.
And that's why people in many cases still use wget.
So by providing this wCarl script, the idea being then that, well, this could then be a reason to maybe not install wget if this is the only reason.
Right, right.
And I mean, one of the things, I mean this has come back to like the old school Linux world and I, and the Unix philosophy, I guess with the whole thing you do, one thing you do well and it's, and what I like about Curl is how well it works with the piping.
Right.
So one of the most common use cases I use is piping curl to JQ for instance.
Right, right.
I, yeah.
And I mean that's, it's such a good example of a tool that does one thing, very sharp focus and nothing else.
Right, right.
And I always wanted to be like that.
Right.
A good member in the Unix family, you pipe output from one tool into the other tool.
So of course curl would pipe everything into whatever you wanted to do.
And for me it was always sort of, that's how we define curl.
It shouldn't ever really understand the data.
It just gets the data for you or uploads the data for you, but it doesn't understand the data.
If you want something to understand the data, you pipe that data into that thing that understands it.
Like Eaq for JSON.
Right?
Yeah.
Curl doesn't understand JSON, has no idea that it is JSON.
So you pipe the thing into the thing that knows about JSON and it can show the JSON for you.
Yeah.
And there's no world where there will be a feature creep or Curl would build into, I don't know, say JSON support parsing.
We have a lot of people asking for that, but I can imagine I'm trying to stay up exactly for that reason.
I think there's always the potential for feature creep because there's always this, we're getting bombarded not only with ideas, but patches and things people want to put in there.
Yeah.
We could go in any direction at any point, but I think it's important to remember what Curl is not, or rather where's the boundary, what can we consider and what's definitely not considered?
And I think sort of dealing with the data, I think it's a pretty good and clear line.
Right?
No no.
That's outside of the curl project.
Understanding the data, that's not our business.
So I try to keep that as a sort of, yes, we're on this side of the fence, that's the other side.
Yeah, I completely agree with I think that principle because if you add Jason, we know it.
Somebody's going to ask for xml and before they add that.
Exactly.
And that said, of course it's, sometimes it's hard to stick to that boundary.
And of course in some way, sometimes the boundaries shifted a little bit because everything is always a discussion.
And I also, I'm also sort of always vary that I cannot always just get stuck in a mindset that we set like 26 years ago.
Right.
Because things change.
It's not necessarily that we did the right decisions in the past.
Maybe we should reevaluate and do things a little bit different because maybe things are, but.
So I always want to at least exercise that thought.
But there are some boundaries I think we should really stick to.
And for example, I always try to emphasize that curl is also for just uploads and downloads, transfer oriented.
So if you come to me and say hey, I want to support protocol, blah, that is not about uploads or downloads, then nah, that's not for me.
And I also want the protocols to be oriented around URL's.
So it should be ideally a scheme for that particular protocol for curl to use it.
I guess you have feature creeps a little bit there because if I'm not mistaken there's even support for shfs and similar things that are like.
Yeah, I guess so.
But it's, yeah, there's this, I mean feature creep.
Yes.
So there's a lot of features.
So yeah it's, and it's hard to turn down features because there's always a niche for adding things.
Yeah, yeah, you support everything but you forgot to add this little tiny detail.
Yeah, okay, well support for that and then another tiny detail that we don't support.
We add support for that on and on and on.
Then you end up with 263 command line options.
Yeah, I completely understand that.
All right, let's switch topic a bit to security because if there is something that should be pretty painstakingly obvious right now in particular in light of the XZ vulnerability, is that Libcurl is probably one of the most prime suspects for injection if you want to backdoor any system on the planet.
So let's talk that what are the mitigation strategies in place right now to make sure that doesn't happen for supply chains attack and whatnot?
Yeah, well I think we do just about everything we can do.
So first of all, the Xe attack was awesomely sort of performed by the attackers.
But they also had the most excellent project to attack in several ways.
Right?
Yes.
And one of the ways being that they had huge binary blobs in the test suite that was perfect for smuggle encrypted attacks.
So that's one of the things, right?
We don't have anything that is no uninspectable blobs in the git repository.
And of course, and then we do everything I think that we should.
Everything is tested a lot and we review everything.
Well, I review almost everything.
I can actually confess that I do a lot of merges that only I review.
So I am suddenly sort of the weak link here.
But anyway, so that's what we do.
And then we try to then make sure that we never merge anything that breaks any tests.
And so if the tests are fine and verify everything shouldn't be possible to put any backdoors in.
Well ideally it shouldn't.
And then we do.
Eventually we do, since a short while back we do entirely reproducible torball builds.
So it should be possible for anyone to just verify it and build.
Exactly binary identical release.
Yeah, that's all for a attack vector.
Right.
But let's talk about that, because that's obviously a big push we're seeing with deterministic builds.
So walk me through that.
How was that process?
How do you find that?
Because obviously that's a big daunting task to do for many projects.
Yeah, but it's not that hard for us.
Release for us is just terrible.
So we're only releasing source code at least.
Well it's not entirely true, but we're focusing on those releases now.
At least that's the primary release.
That's terrible.
And when we do a terrible, we actually do the releases in a similar style as XZ because we use water tools.
So we generate a configure and stuff.
And the generated configure is not in git.
So of course in the XZ attack they then injected code between what's in git and what ended up in the turbo.
So that's why the reproducible build is important, because then you can verify that there's nothing ended up in the tarbol that wasn't in git or wasn't generated with a yemenite established tool.
So basically we just then make sure that when we build that turbo, we do that in a reproducible way.
So nowadays I build a tarball with a docker image of a specific hash, basically with the exact tools and make sure that the torbo is built with the right timestamps, blah, blah, so that if you just rerun the same command line that I did, you get an identical copy of the release turbo.
So it's actually pretty straightforward once we got all those tiny details just done correctly.
Okay, that's fair enough.
That's fair enough.
So then the big sort of.
I wouldn't say that it's a big, but the risk is really then that we would just merge someone's back door, you know, someone trying to just deliberately deliver a patch that says, hey, this patch is doing x and it looks good and we merge it, but in reality it actually does y.
And then down the line someone can exploit that and use that backdoor.
But I have never even seen an attempt do that.
Maybe I've just missed it and we have it already.
But I claim that it's really difficult to deliberately plant a backdoor like that does anything substantial.
I think the actual biggest risk for curl, as in many other projects, is just the accidental vulnerability thing that we actually did something stupid a while ago that you can exploit.
And then by exploiting a buffer, some kind of crash, some kind of thing, and that's what you can exploit and do something nasty.
And that kind of security vulnerabilities we of course find and fix every now and then.
I mean that's more in the domain of regular vulnerabilities.
More so than supply chain attacks, I guess.
Yeah.
And looking back at the history of those, you can see that most of those mistakes were mine, so I know they weren't planted.
Yeah, that's fair.
So one thing that, I mean, obviously you all guys are doing really well.
I mean, I looked at the openSSF scorecard for Curl and you have a golden status for openSF scorecard, which is a pretty impressive score actually, because not a whole lot of projects actually make it that far.
Right?
No, right.
And I really made an effort to make sure that we sort of qualify for all those criteria and so.
Yes, but you're not actually, I don't think you ever mention that anywhere else that there is an opensSF scorecard that you actually aced, which I think is a lost marketing opportunity because I think that should be for, that should be the front for Readme of the off the readme file on GitHub, honestly.
Right.
Yeah, I think I have, there's a link somewhere.
Yeah, maybe I should stress that harder.
Yeah, I think.
Yeah, no, that's impressive.
Very few projects actually qualify that gets that far.
It's actually pretty good that the best practice.
So openssF has that best practice site that actually has a lot of questions to fill in.
If you sort of follow a lot of different best practices.
I think it's actually really good.
Sort of just a checklist if you can say yes to all of those, you know, that you're doing things at least not bad.
So that's what I've tried to sort of why I went for the gold too, because I want to make sure that we do everything as good as we possibly can in pretty much every aspect when it comes to running an open source project.
And also I would say it's a critical piece of infrastructure as well that so many people depend on that.
I mean I'm glad you see you guys are focusing on that because that is so important, right?
Yeah.
And in the same style also that sort of.
Yes, as long as no one, you know, it's a success that no one is yelling about it.
Right.
So if as long as no one actually finds their heart bleeding curl, it's good.
Right.
So it's one of those silent successes too.
So everything is good as long as no one finds anything alarming.
So we should just keep it that way.
And of course it's an ongoing effort because it never ends.
Oh yeah.
You're never done with security.
Of course.
Yeah.
Let's talk s bombs because that's something that obviously aligns with openness of core cards like tangentially I guess.
Are there any plans for publishing s bombs for curl?
I think we're just following what everyone else is doing.
And that's also why I talked about our releases are only turbo so we don't actually have any third parties in the tarbella.
We actually don't have any bill of material in that way.
I guess that's because we just ship the code.
So someone else is building stuff.
So we just push the responsibility to the others who are actually building everything and they're responsible.
So basically all the distros that are shipping curl and everything, they build it themselves.
So they get it from us, they build it and they include third party dependencies and everything.
So they are the ones who are actually get to do that.
But I guess you have dependency tree inside of curl.
I would imagine you are depending on some other third party piece of.
Not strictly, we only depend on them when you build curl to use them.
And that's why we don't have the dependency when we ship curl.
Because then you have to build it.
That's fair enough.
But it's useless in isolation, I guess, without building it.
Right, exactly.
Yeah.
But that's also one way that we sort of duck for a lot of issues.
And so we, since we don't ship the binaries, handling the binary thing there is not, so that we avoid a lot of issues and problems by doing it this way.
We do ship a few binaries anyway.
That sort of breaks what I'm saying.
So we do offer binaries for windows, for example.
Right.
So and then in that case we have a pretty, we don't have any really s bombs because pretty much, I think, I don't know really why, but we have a huge detailed log about everything and exactly that we ship and include in that.
So all of that is transparent and available.
I think we're pretty much, we're well covered in that aspect too.
And I think if we ever land on, we should do this to provide s pumps in this format more universally then I'm sure that we will do that as well.
That's fair enough.
That's fair enough.
Let's talk CV's because you posted a blog post last year.
I think it was about the useless or everything that is wrong with CV's, which I think was a kind of entertaining read about a CV filed against Curl, which is kind of I guess a theoretical attack in some sense.
But let's talk a bit about that.
Do you want to recap the article a bit about this CV and your thoughts on that?
Yes, so I actually, so I had this struggle against the CV in many, I've had a lot of weird CV's and security related things over the last few years, but I believe in that case it was a integer overflow in a delay argument to curl.
So if you ask curl to retry on a failed transfer because it has that command line option, basically if the server returns internal error or overloaded or whatever, you can ask her to try again later.
Sure.
And then it'll try 3 seconds later or 12 seconds later.
Then we have another option that says wait this amount of time until you'd retry again.
One of those days we just fixed the little buffer overflow.
So when you set that retry counter to you specify it in seconds and internally we multiply that by 1000 to get the timer into milliseconds and we didn't have any check for that.
So if you set that retry delay to something enormously big, and we will multiply that by 1000.
Sure.
It'll sort of.
Yeah, it will overflow the integer or the long or whatever I don't remember the variable type, but it was a 64 bit on Linux.
So yes, then you get an integral overflow.
So if you then ask curl to retry like once in every 68 million years, and then instead it would retry once every probably 0 second or 60 milliseconds or whatever it turned out it would wrap and become a very small number instead of insanely large.
I consider that just a bug because first of all, no one would ever enter that huge number unless you actually wanted to trigger this.
Okay, and that's fair, someone could do that.
But what's the worst that could happen?
It wouldn't wait 68 million years.
It would actually do the retry immediately after it failed the first one.
And that is exactly how curl behaves.
If you just type multiple URL's after each other on the command line, it would be one, two, three, exactly as it always does.
So yeah, okay, worst case, it doesn't wait forever, it'll just do the next request immediately.
And someone then found out that I used the word integral in a commit message because it was.
And someone then registered.
And CV is just a bug id basically.
So you can just ask for that from mitre or whoever is responsible for that.
And someone then requested a CVE for this bug.
And one day that suddenly just became public because obviously no one told us about it.
So just one day this CVE becomes public.
Integral overflowing curl and rated severity 9.8, I think.
Because that's an integral overflow.
What's the worst that can happen with an integral overflow?
Probably really serious stuff can happen, right?
If it had been in some place that actually matters.
And I of course was just what a CV on.
Because usually we manage all the CV's ourselves, of course, because people report to us, so we know about it, we do the fixes, we talk about it, we announce the details, we write about it, so on.
But in this case, of course it just showed up out of the blue one day.
And then of course it became this huge discussion, is this a security problem?
And I'm a pragmatic guy, I hate this guy.
Sure, it's an integral overflow.
Integral overflows written in c.
They're undefined according to the C standard.
And then the most nihilistic people, they say undefined, then anything can happen.
So yeah, that's true.
By the sort of literal meaning of the words it's undefined, it means that anything can happen.
That's actually true.
So in theory, instead of wrapping everything to 32 milliseconds, instead of 68 million years it could actually do something seriously super crazy and you know, your computer might burn.
But I think that's a really silly argument to have because you can try it through how many compilers you want added and see what do compilers actually do with it.
And well they can wrap that integral in many different ways and some could actually in some ways do something somewhat strange, but in no way did that ever become a security problem.
Anyway, so that was my argument.
So first I wanted to mitre to just reject the CVE because it was stupid.
It wasn't a security problem at all.
And they refuse to do that because yes, it's a security problem, they say, based on what, I don't know, because they won't talk to me.
They just reply and say it's a security problem.
And then of course the score you see when you just google for the CVE is usually you end up on the NVD site, the national vulnerability database, and they set scores on CVE's.
That's just their business.
So the next thing I did was that I contacted NVD and said hey, how on earth did you end up with a 9.8 for this when it's not even a security problem to begin with?
And NVD, they don't remove anything because they say if Mitre added we will host it.
So they won't reject it, they will just.
They can, if I yell at them enough they can reevaluate the score apparently because they did in this case.
So they reevaluated.
So from 9.8 they ended up with 3.3, I think.
Okay, just yelling.
Significant drop.
Exactly.
And by that I figure, okay, sure, 3.3, you can have it like that because 3.3, I don't, I mean, sure.
Then I think for me that sort of just sort of removed the issue from my table really.
So okay, you can have it, 3.3, fine, if you want to.
In my view it's still a bad and invalid CVE, and I call it that.
On the curl webpage they say it's a 3.3.
Sure, fair enough.
All right.
The last thing I want to talk about, which I think you're in a very good position to speak about, is the future funding of open source because that's obviously a lot of the supply chain attacks in open source.
A lot of them stem from the fact that these are projects maintained by one or maybe a handful of people at most, but used everywhere.
Curl is a great example of that where it has its fingers into almost everything that runs Linux or any embedded system more or less now.
The funding is a hard part, right?
You managed to do it by consulting, but it took you 20 some years to get there, right, which is 20 some years more than most projects.
So how do you see the future of open source funding looking like with your experience in this domain?
I don't have any answers.
I don't have any magic bullets there either.
So no, it's a challenge, of course, because I think just the market economy just makes it easy for anyone who wants to just get open source, use it for as long as you want for free.
Never contribute anything back because you don't have to and you can run a successful business without contributing anything back at all.
I think that's the ultimate challenge because you can, companies can get away with it, so they will do that.
And it's really hard to, as a, as producers to suddenly say, hey, wait a minute, you should pay me.
I've given everything away, but now I want you to pay because they don't have to and it's hard for them to do.
Of course, I think in reality and for companies products to be sustainable in the long run, it benefits companies to help the projects that they depend on, right?
So that they can continue doing their products in the future so that all their dependencies also strive to, you know, continue and are secure and good and everything.
So it is good in a lot of companies own view as well.
But I think it's often overlooked by them and by the fact that they can avoid it.
They often do that just out of short sightedness.
And of course then there's the challenge in the other end.
So it's hard for some, in a lot of projects it's hard for people to even, just how do you even start getting money for anything?
If it's a small project, surely it's not a full time thing, but who's going to pay for it?
And how, and how do you go.
From a side project to a full time project?
Right, exactly.
That was my challenge for so long time.
How do you do that, take that step?
It's really difficult.
So yes, there are numerous challenges on how to do that.
And at the same time, of course, if it's just your spare time thing, of course no one can actually require you to have everything top notch all the time.
So it's only natural that you slip and you have bugs or security problems or whatever in your, I mean it's your right to have that.
It's your spare time project you do what you want.
It's not for anyone else to require you to do anything on your spare time.
Do you have any opinions around?
So GitHub sponsors, for instance, they've been trying to gain some momentum.
I mean, we've used that screenly for a while for our open source project.
Some of that works.
We have some.
Has that worked with curl?
Have you seen some success around that?
Usually all the ways when you open up for sponsorship, it's good with sponsors.
So that allows people that it contributes in a easy way when possible.
Usually sponsors never cover enough money to actually drive serious development, though, because usually sponsorships are mostly individuals.
So sponsorships tend up to be in the smaller ranges.
So you need a huge amount of people actually shipping money for that to actually end up employing someone full time in a western world.
So it's really hard.
You need companies involved to actually up the amounts and then you have to do something else than just ordinary sponsorships.
That's at least the usual way things go.
Yeah, because I guess that's the problem, right.
If you are a good enough software engineer that you can write a great piece of software that people actually are using, your going rate on the market to join a company will be six figures in most western countries.
Six figures a lot for an open source project.
So that's basically your alternative option, right?
Yes, exactly.
Yeah.
So it's certainly a challenge.
So I mean, I'm hoping that we have some progress that one of the things that we've at screen that we use open source a lot and one of the things we've been toying with internally is to have kind of a pool internally that we allocate each year to open source projects with a bias towards smaller projects.
Because I think the big projects like, I mean, I guess Curl is a big project by now in relative terms, and Django is and all postgres and all the other big top projects that we use.
But for every one of those, there are plenty of smaller projects that maybe just be one guy.
And my philosophy is the buyer should be on those guys because those are the guys who need the most, right?
Most probably, yes.
Yeah.
So, yeah, exactly.
That's, that's one good way to do it.
And I think also it could possibly also be with angle to, I don't know, over maybe not overlooked projects.
But there's also, I find a, I find it a challenge to be a component in a software tree.
Right.
So I think it's a different case.
At least I sense that when I talk to other open source maintainers who do accept sponsorships.
It's a different case when you, for example, if you do more like more directed end user products, perhaps with more usability, so that you reach more, and then you get sponsorships directly from end users.
But when you're doing a component in someone else's project, then the project using your stuff is the one who's going to get that visibility among users, not you.
So there's a visibility challenge in that case.
And I mean.
Yeah, and I think that has been very evident in the NPM world, right.
Where you have had a lot of supply chains attack with domain squatting and squatting attacks and whatnot.
Right.
And you just.
Because a little component.
Right.
Used by everyone, but no one actually remembers that little component or even knew about it.
Right, exactly.
And that's why I'm very bullish at S bombs, particularly when you're exploding s bombs that will bring light to that world.
Right.
More so, hopefully.
Yeah, it really should.
At least I think that it will.
I mean, I think it'll take time, but I think at least that's sort of a necessary step towards visibility and knowledge and just making everyone aware exactly what the sort of the myriad of tiny projects that we're all building everything on top of.
Yeah, it's like that XKCD cartoon, right, where you have all these projects and they have like the small, like.
Oh, well, I think in that particular example, it's libsl or whatever it is.
Lib opens cell or whatever it is.
That it actually doesn't say the project.
All right, fair enough.
Yeah, 23, 47.
Fair enough.
But it depends.
But it's illustrated as well, right?
It is.
It's starting on the show of giants at the end of the day.
Exactly.
And there are so many of those tiny little pillars, too.
So there's not only one little pillar somewhere.
There are a bunch of those tiny little pillars.
Exactly, exactly.
Well, I unfortunately don't think we're going to solve the problem of open source funding on this call, but this has been a lot of fun for me and I really thankful for the work you're doing on curl.
I've been a fan for a long time, so I was very happy to get you on the call.
So anything you want to shout out before we wrap up the call today?
No, I don't think so.
Join the curl project.
Help us do things.
Amazing.
I believe you.
Good stuff.
Perfect.
Thank you so much, Daniel, and talk to you soon.
Cheers.
Bye.
Found an error or typo? File PR against this file or the transcript.