[00:04]
Vlad A. Ionescu
Thanks for having me.
[00:06]
Viktor Petersson
So, yeah, absolutely.
[00:07]
Viktor Petersson
Very happy to have you on.
[00:09]
Viktor Petersson
We've been trading emails, had a few calls over the last like six months and there's a lot of overlap.
[00:14]
Viktor Petersson
And the big reason why I wanted to have you on the show and do a bit of a kind of a devex SDLC kind of episode today.
[00:23]
Viktor Petersson
And you have good experience in this domain and I kind of wanted to maybe kick it off there with what you're doing today and then we can dive into bigger trends industry.
[00:33]
Viktor Petersson
But you are the founder of a company called Earthly and so maybe we can start off there with a quick intro to Earthly to just kind of level the expectation of the conversations going forward.
[00:44]
Vlad A. Ionescu
Okay, perfect.
[00:45]
Vlad A. Ionescu
Yeah.
[00:46]
Vlad A. Ionescu
So, hi everyone.
[00:47]
Vlad A. Ionescu
I'm.
[00:48]
Vlad A. Ionescu
I'm Vlad.
[00:49]
Vlad A. Ionescu
I'm the CEO of Earthly.
[00:51]
Vlad A. Ionescu
We've been building this company over the last five years or so.
[00:54]
Vlad A. Ionescu
We've been focusing around the area of managing the SDLC ecosystem better.
[00:58]
Vlad A. Ionescu
Our first product that we built is a framework to write better CI cd and that took off as an open source project.
[01:06]
Vlad A. Ionescu
It's now at 11,000 stars on GitHub, is being used by big companies such as Adobe and Roche and a bunch of other big names like that.
[01:17]
Vlad A. Ionescu
And currently we're building our second product which is kind of more aligned with the SDLC and perhaps the supply chain that I know Victor is a big fan of.
[01:30]
Vlad A. Ionescu
Our new product called Lunar is about managing the SDLC at scale in big organizations and managing engineering practices in these organizations.
[01:42]
Viktor Petersson
Good stuff.
[01:42]
Viktor Petersson
There's a lot to unpack there.
[01:44]
Viktor Petersson
So what I find really interesting is kind of like taking a bigger step at this whole wave of companies, I guess that's trying to tackle this CICD pipeline.
[01:54]
Viktor Petersson
I think it's kind of hard.
[01:56]
Viktor Petersson
Can't have a really conversation without mentioning Dagger from solving I guess as well did something similar in this space.
[02:03]
Viktor Petersson
And my impression is my impression, but this is coming from people coming from Fang, largely looking at what does the state of CI look like for non faang companies.
[02:17]
Viktor Petersson
And I believe that's kind of what we started conversation like a few months back.
[02:22]
Viktor Petersson
Right.
[02:22]
Viktor Petersson
Like this is kind of like your ex Googler.
[02:25]
Viktor Petersson
So is like maybe like we start there like how is the CI pipeline different at Fang or Google?
[02:33]
Viktor Petersson
I mean I guess you can really speak about Google versus the rest of us, right?
[02:37]
Viktor Petersson
Like maybe that's a good starting point.
[02:39]
Vlad A. Ionescu
Yeah, exactly.
[02:40]
Vlad A. Ionescu
Well, it's a huge difference.
[02:42]
Vlad A. Ionescu
Right?
[02:42]
Vlad A. Ionescu
So if you might have read articles about Google and Facebook and Twitter or X, I guess these days you might have seen that they have a significant investment internal developer infrastructure.
[02:55]
Vlad A. Ionescu
And I think Google started this trend.
[02:57]
Vlad A. Ionescu
They invented this technology called Basel to.
[03:00]
Vlad A. Ionescu
Basically they started off with a bunch of make files, and those make files were kind of getting unwieldy.
[03:06]
Vlad A. Ionescu
They have this massive mono repo where they shove everything, every single programming language, every single piece of code that they write, and initially it was all managed by a bunch of make files.
[03:16]
Vlad A. Ionescu
Now imagine how quickly that gets out of control, right?
[03:19]
Viktor Petersson
Yeah.
[03:20]
Vlad A. Ionescu
And so, I don't know, maybe 15 years ago or maybe a bit more, they invested in a dedicated internal tool that you might call maybe a CI.
[03:29]
Vlad A. Ionescu
But this was pre Jenkins era perhaps, right.
[03:31]
Vlad A. Ionescu
So like it was very early on.
[03:34]
Vlad A. Ionescu
And essentially it's a build system that understands deeply the dependencies between the different components in the organization.
[03:41]
Vlad A. Ionescu
Right.
[03:42]
Vlad A. Ionescu
And it can do a lot of incremental building, it can do remote building, and it can do heavy caching, which is critical for C code bases that Google was actually heavy on.
[03:53]
Vlad A. Ionescu
And so a lot of that sort of central type of investment resulted in a very sort of standardized type of developer infrastructure.
[04:04]
Vlad A. Ionescu
And that has been actually a massive sort of productivity enhancer for Google.
[04:11]
Vlad A. Ionescu
And if you remember back in the day, Google was the hottest thing to work for.
[04:16]
Vlad A. Ionescu
Part of that was because they had really getting that really, well, like the developer infrastructure, making developers happy, productive and so on.
[04:26]
Vlad A. Ionescu
Now, outside of Google, in parallel, something else was happening.
[04:30]
Vlad A. Ionescu
Containers were coming along.
[04:32]
Vlad A. Ionescu
Google actually internally was kind of, I guess, late or maybe never even took on containers the same way.
[04:38]
Vlad A. Ionescu
But containers outside of FAANG in general created a very different type of shift towards heavy microservice use and freedom for each microservice to be defined completely differently in its own specific way.
[04:54]
Vlad A. Ionescu
Right.
[04:54]
Vlad A. Ionescu
And that meant, you know, there's more decoupling, that means you can have more freedom within each team to innovate better.
[05:02]
Vlad A. Ionescu
But at the same time, that creates massive chaos when it comes to the developer infrastructure.
[05:07]
Vlad A. Ionescu
Right.
[05:08]
Viktor Petersson
It's decoupling, but also like a really complex mesh of interdependencies that are really hard to test in your CI pipeline.
[05:16]
Viktor Petersson
Right?
[05:16]
Vlad A. Ionescu
Exactly.
[05:17]
Vlad A. Ionescu
Yeah.
[05:18]
Vlad A. Ionescu
Testing is just tip of the iceberg, Right.
[05:20]
Vlad A. Ionescu
When you think about, you know, how do you insert now compliance or how do you insert additional verification that maybe individual teams are not thinking about, but at scale.
[05:30]
Vlad A. Ionescu
You're now wondering whether every single team is following the proper engineering standards that your organization wants to abide by.
[05:37]
Vlad A. Ionescu
Right.
[05:37]
Vlad A. Ionescu
Whether that's driven by compliance or, I don't know, maybe protecting your reputation, avoiding incidents in production, preventing security incidents.
[05:46]
Vlad A. Ionescu
And so on all of those things are difficult now to kind of enforce.
[05:51]
Vlad A. Ionescu
It's just a big sea of heterogeneous sort of different things that.
[05:57]
Vlad A. Ionescu
And basically you no longer have a single insertion point.
[06:00]
Vlad A. Ionescu
I think that's the biggest sort of drawback of this sort of movement.
[06:05]
Viktor Petersson
Yeah.
[06:06]
Viktor Petersson
And the docker file is almost like the modern make file.
[06:09]
Viktor Petersson
Right.
[06:09]
Viktor Petersson
It's of the modern CI pipeline and the dev environment.
[06:13]
Viktor Petersson
And yes, one of the biggest pain points that I've seen is GitHub Action has become kind of the norm for CI these days, I would argue.
[06:23]
Viktor Petersson
And the problem is that you can't run this locally, like you can't run GitHub Actions locally, which means that every time you just want to test things, as things are running on CI, you need to push to CI.
[06:38]
Viktor Petersson
And that kind of like, adds a lot of latency when you just want to see if things are working.
[06:43]
Viktor Petersson
So that's the other side of this whole wave of new CI tooling, I would say, like CI agnostic tooling that could also run locally, like maybe speak a bit about how you see that as well.
[06:54]
Vlad A. Ionescu
Yeah, for sure.
[06:56]
Vlad A. Ionescu
The history of CI, I guess, started becoming mainstream with Jenkins.
[07:01]
Vlad A. Ionescu
Jenkins was probably the first popular implementation of such a system.
[07:05]
Vlad A. Ionescu
I guess it was called Hudson back in the day.
[07:07]
Vlad A. Ionescu
But over time we kind of organically grew in this direction, where CI and local dev stack are two completely different things.
[07:16]
Vlad A. Ionescu
Right.
[07:16]
Vlad A. Ionescu
And that hasn't really changed.
[07:18]
Vlad A. Ionescu
And there were some innovations in the field.
[07:20]
Vlad A. Ionescu
Perhaps companies like CircleCI and others have sort of taken Jenkins and made it a SaaS like you no longer have to kind of manage it.
[07:29]
Vlad A. Ionescu
It's also much more isolated.
[07:31]
Vlad A. Ionescu
Every single job starts from scratch with a completely new blank slate and.
[07:35]
Vlad A. Ionescu
And so on.
[07:35]
Vlad A. Ionescu
And these were good innovations, good sort of steps forward, but we never really integrated the magic of containers into this whole experience.
[07:45]
Vlad A. Ionescu
And the beauty of Containers is that it runs consistently no matter where you execute it.
[07:50]
Vlad A. Ionescu
Right.
[07:51]
Vlad A. Ionescu
That's a big thing.
[07:52]
Vlad A. Ionescu
You kind of isolate its dependencies and so on.
[07:55]
Vlad A. Ionescu
For that reason, we haven't really taken the CI experience to put it on the developer's laptop as well, which is part actually of our mission.
[08:04]
Vlad A. Ionescu
With our first product, we allow companies to essentially develop their cicd pipelines on their laptops and be able to test and execute those CICD pipelines on their laptops.
[08:16]
Vlad A. Ionescu
And when they're ready, they can push it up to GitHub and it's pretty much guaranteed that it's going to work, because you've run all of that in containers through the magic of Earthly, we found that to be a good enabler for debugging CICD failures, which are oftentimes very difficult to reproduce.
[08:34]
Vlad A. Ionescu
But also developing new CI CD pipelines much faster.
[08:37]
Vlad A. Ionescu
You no longer have to wait for git commit over and over again.
[08:41]
Vlad A. Ionescu
Do that long cycle like that.
[08:44]
Vlad A. Ionescu
Finally, there's something to be said about cross team collaboration.
[08:49]
Vlad A. Ionescu
We mentioned about microservices where every team has its own dev stack and that means each team has difficulty reproducing the build of the other team.
[08:58]
Vlad A. Ionescu
There is perhaps a readme there.
[08:59]
Vlad A. Ionescu
You have to follow line by line, configure this file and this other file and so on.
[09:04]
Vlad A. Ionescu
Whereas with Earthly, everything is containerized, you just run this one command and everything just works.
[09:10]
Vlad A. Ionescu
Right.
[09:10]
Vlad A. Ionescu
So you never have to kind of wonder how exactly you're going to get this other project set up.
[09:15]
Vlad A. Ionescu
And that has been very powerful for our open source users.
[09:19]
Viktor Petersson
Yeah, And I think, I mean this as CIs evolved, like if you go back into the days of like early day Jenkins, right.
[09:28]
Viktor Petersson
CI was just a bash executor, right.
[09:31]
Viktor Petersson
That was like you just ran a bunch of shell scripts in like a sequence that's no longer true.
[09:37]
Viktor Petersson
Like look at git, look at a modern CI pipeline on GitHub.
[09:40]
Viktor Petersson
Like you have GitHub Actions module.
[09:42]
Viktor Petersson
You do this, you do that, you pull in caching from here.
[09:45]
Viktor Petersson
Like it's so much more complex than way back when you're not dealing with a dsl, essentially it's almost a dsl.
[09:55]
Vlad A. Ionescu
Right, Exactly.
[09:58]
Vlad A. Ionescu
I think in many ways some of the same basic concepts have remained similar.
[10:06]
Vlad A. Ionescu
Yeah.
[10:07]
Vlad A. Ionescu
Now we write in YAML, it's no longer exactly like a bash sort of line by line, execute this and execute that.
[10:14]
Vlad A. Ionescu
And yeah, we have caching, although I would argue that caching is still kind of rudimentary.
[10:20]
Vlad A. Ionescu
And we found that, you know, if you actually invest in understanding the dependencies between projects and you have a system that really can leverage that heavily, you can get to like 10x faster CI builds just by doing that.
[10:33]
Vlad A. Ionescu
Right.
[10:34]
Vlad A. Ionescu
And that's part of what Earthly actually also delivers.
[10:38]
Vlad A. Ionescu
The other thing I've seen in recent CI sort of environments is that there is use of plugins.
[10:44]
Vlad A. Ionescu
Right.
[10:45]
Vlad A. Ionescu
That's kind of natural.
[10:46]
Vlad A. Ionescu
And recently there was this vulnerability in, I was going to say change files.
[10:53]
Vlad A. Ionescu
Right?
[10:53]
Vlad A. Ionescu
Yes.
[10:55]
Vlad A. Ionescu
And before that, some years ago, there was codecov.
[10:57]
Vlad A. Ionescu
Right.
[10:58]
Vlad A. Ionescu
Things that you put in your CI and there's very little governance around.
[11:02]
Vlad A. Ionescu
Like people care a lot about what you put in production, but maybe aren't so careful about what they're putting in their CI, right?
[11:09]
Vlad A. Ionescu
There's no S bomb for CI cd, right?
[11:11]
Vlad A. Ionescu
Or there is, but people don't.
[11:13]
Viktor Petersson
Funny enough, there is a conversation I read this week actually on the system analyst about actually defining this stuff before.
[11:20]
Viktor Petersson
I think they call it a P bomb, like a pipeline bomb essentially.
[11:23]
Viktor Petersson
But you're absolutely correct, right?
[11:25]
Viktor Petersson
In particular in the world we live in with Cursor, that I would imagine Cursor is probably writing a lot of the GitHub action files these days.
[11:33]
Viktor Petersson
And it's just like write me this pipeline and it just pulls in various modules and one of the first thing I always do is like, okay, cool, but do we need this or can we actually do ourselves?
[11:46]
Viktor Petersson
That is the first thing, right?
[11:47]
Viktor Petersson
And also like, and if we do need it, what's the security track record?
[11:52]
Viktor Petersson
Like how many, Even if you use like something very rudimentary, like how many GitHub stars does this have?
[11:57]
Viktor Petersson
Like obviously in the change of change log, perhaps not very good as a proxy, but at least is a proxy rather than a project out there with like five stars on GitHub that we have no audit trailer at all.
[12:11]
Vlad A. Ionescu
Oh yeah, Even the more popular ones, you know, we've seen some really low level stuff that is extremely popular, basically ubiquitous and you know, after many years of exposure, we just figured out, oh, there was a vulnerability here all along, right.
[12:30]
Vlad A. Ionescu
And you don't know what kind of like state funded foreign government has been leveraging that all along or you just don't know.
[12:40]
Vlad A. Ionescu
It's kind of like this big unknown sort of risk you're kind of taking.
[12:44]
Viktor Petersson
Yeah.
[12:44]
Viktor Petersson
And that's it, right?
[12:45]
Viktor Petersson
Like it's.
[12:46]
Viktor Petersson
There are so many levels of abstraction involved, right?
[12:50]
Viktor Petersson
So like if you look at the anatomy of one of these modules, like it's the basic GitHub module, like it's probably, let's say for instance, written in Ruby, also Ruby in npm, like in JavaScript.
[13:02]
Viktor Petersson
Right.
[13:03]
Viktor Petersson
Like will it probably pulling in like 50 or 150 different dependencies which in turn all might be have their own vulnerability track record, right?
[13:12]
Viktor Petersson
So it's such a complex space to do proper security audits on, right.
[13:18]
Viktor Petersson
You kind of need like, not only do you need to pin the mod you're using, I know Open SF got some tools around this for pinning a hash against the version or all that stuff, right.
[13:30]
Viktor Petersson
And that's great, but you also need corresponding like an S BOM essentially against that hash.
[13:36]
Viktor Petersson
So you can all say, okay, cool, this has this.
[13:38]
Viktor Petersson
But also this actually is included in there.
[13:40]
Viktor Petersson
So if there is a CV in there, you can actually find this.
[13:44]
Viktor Petersson
But that's.
[13:45]
Vlad A. Ionescu
Yeah.
[13:45]
Viktor Petersson
Now it becomes such a tremendously complex problem to just run your CI.
[13:51]
Vlad A. Ionescu
Yeah, exactly.
[13:53]
Vlad A. Ionescu
Especially that we want to insert all of the security verification in CI CD when perhaps the goal of each individual application team is to ship features and not to be held back by security scanners and all that.
[14:09]
Vlad A. Ionescu
I think that's where there is a bit of misunderstanding between central teams like platform and security and so on and the individual feature or application teams.
[14:20]
Vlad A. Ionescu
I think in that regard, a lot of times the responsibility of owning the CI CD pipeline is kind of shared.
[14:27]
Vlad A. Ionescu
Right.
[14:27]
Vlad A. Ionescu
So like the platform team or the security team wants to insert a security tool and the application team doesn't even acknowledge the PR whatsoever.
[14:36]
Vlad A. Ionescu
They're just, you know, full steam ahead.
[14:37]
Vlad A. Ionescu
We have to ship this or whatever.
[14:38]
Viktor Petersson
Yeah, right.
[14:39]
Vlad A. Ionescu
And it takes, we've heard from companies.
[14:42]
Vlad A. Ionescu
We were doing this industry research last year.
[14:44]
Vlad A. Ionescu
Sometimes it takes a year for PRs that are submitted by the central teams to get merged into all of the projects.
[14:52]
Vlad A. Ionescu
The long tail is insane.
[14:54]
Vlad A. Ionescu
And all that time you're kind of exposed and don't even know what exactly your posture is all around your organization.
[15:03]
Vlad A. Ionescu
We found that to be a big issue because there's individual tools to extract S BOMs for different types of languages, containers, and you kind of skin the cat in so many ways.
[15:15]
Vlad A. Ionescu
But currently there are no good tools out there to understand whether those tools have actually been implemented everywhere within the organization.
[15:24]
Vlad A. Ionescu
One thing.
[15:25]
Vlad A. Ionescu
Right.
[15:25]
Vlad A. Ionescu
And the other thing is how do we make this shared CI CD pipeline have clear boundaries about who owns what?
[15:33]
Vlad A. Ionescu
You know, perhaps the platform team does need to own some piece of that to be able to answer verification.
[15:38]
Vlad A. Ionescu
Right?
[15:38]
Viktor Petersson
Yeah.
[15:39]
Viktor Petersson
And there's also like not all CI runs are equal either.
[15:43]
Viktor Petersson
Right.
[15:44]
Viktor Petersson
Because if the attack vectors are different, right.
[15:49]
Viktor Petersson
If you have a run that just does something to your checked out code base and doesn't actually write anything back or ship any artifacts or ship any binaries or ship anything.
[16:00]
Viktor Petersson
Like the blast rate is done as pretty slim, right?
[16:02]
Vlad A. Ionescu
Sure.
[16:02]
Viktor Petersson
You could do something nasty.
[16:04]
Viktor Petersson
Maybe you can leak the secrets.
[16:06]
Viktor Petersson
Okay, sure, that's bad, but it's not going to have any significant security issues on you actually what you actually ship into your customers.
[16:16]
Viktor Petersson
Right.
[16:17]
Viktor Petersson
So yeah, that's another element that is so much more complex in this pipeline because you can't like it's hard to I guess programmatically detect that in the pipeline, what it does.
[16:29]
Vlad A. Ionescu
Yeah, exactly.
[16:31]
Vlad A. Ionescu
So you touched on a great point here.
[16:33]
Vlad A. Ionescu
A lot of the companies that need to insert this very strict type of verification, oftentimes invest in central CI CD templates.
[16:41]
Vlad A. Ionescu
So every project would have to import that.
[16:43]
Vlad A. Ionescu
And that means you're going to get all or nothing.
[16:46]
Vlad A. Ionescu
You either import that and you're going to get the whole shebang of security verification.
[16:50]
Vlad A. Ionescu
And it's going to be extremely restrictive.
[16:52]
Vlad A. Ionescu
Even though maybe, like you're saying your application might not be even talking to the Internet, maybe it's not even handling any user data, it might be just an internal thing.
[17:00]
Vlad A. Ionescu
Right.
[17:01]
Vlad A. Ionescu
Maybe not shipping anything at all.
[17:04]
Vlad A. Ionescu
And in that regard, you need a way to create scope for what you care about.
[17:10]
Vlad A. Ionescu
Right?
[17:10]
Vlad A. Ionescu
Have a definition of scope.
[17:12]
Vlad A. Ionescu
What's in scope for this type of compliance, what's in scope for handling pii, what's in scope for things that are exposed to the Internet and so on, and all of these different things you probably want to treat differently because you want to have as much agility as possible in general.
[17:29]
Vlad A. Ionescu
But for the things that are more sensitive, you.
[17:31]
Vlad A. Ionescu
You can afford to kind of insert that scanner, maybe takes 20 extra minutes of your, you know, CI time or whatever.
[17:38]
Vlad A. Ionescu
But you have to kind of treat these differently.
[17:41]
Viktor Petersson
Yeah, yeah.
[17:43]
Viktor Petersson
It's such a complex thing to apply this blueprint across all these things.
[17:48]
Viktor Petersson
Because I guess the benefit with the old style CI where everything was a bash script, was that there was limited attack vectors in that sense.
[17:56]
Viktor Petersson
Right.
[17:57]
Viktor Petersson
Like, there were like fewer things that were easier to audit, I guess, in many ways.
[18:01]
Viktor Petersson
Right.
[18:02]
Viktor Petersson
But now we have this level of abstraction.
[18:05]
Viktor Petersson
You have like the CI modules that you're calling on, you have docker files that you need to be audited.
[18:10]
Viktor Petersson
You have like, there are so many possible attack vectors.
[18:13]
Viktor Petersson
And not only that, like, how do you even audit that?
[18:16]
Viktor Petersson
Right.
[18:16]
Viktor Petersson
Like, what version, like, are you running an UL Docker container with like, I don't know, like an node version that's not even.
[18:22]
Viktor Petersson
That's eul, Right?
[18:23]
Vlad A. Ionescu
Like, right, yes.
[18:25]
Viktor Petersson
It's so complicated.
[18:27]
Viktor Petersson
Right.
[18:27]
Vlad A. Ionescu
So how do you, like, exactly.
[18:29]
Viktor Petersson
How are you approaching this as a problem space?
[18:32]
Vlad A. Ionescu
Yeah.
[18:33]
Vlad A. Ionescu
You know, the way we've been sort of the mantra we've been operating on with this new product has been, you know, there's a lot of monitoring that goes in production.
[18:42]
Vlad A. Ionescu
You know, you have logs, you have metrics, you have instrumentation, you have alerts and firewalls and policies and so on, but there's nothing like that for the sdlc.
[18:53]
Vlad A. Ionescu
The stuff that happens before production, that has been our guiding light towards what we're building here.
[18:59]
Vlad A. Ionescu
We're basically building monitoring for sdlc.
[19:03]
Vlad A. Ionescu
What does that mean you have to understand what exactly is being built and how it is being built throughout your sdlc?
[19:11]
Vlad A. Ionescu
Like I mentioned, there's that high degree of diversity within each project.
[19:15]
Vlad A. Ionescu
You want to be able to insert centrally this verification without having to integrate with individual projects one at a time, where you're depending upon each team to adopt your templates or your additional gates or whatnot.
[19:28]
Vlad A. Ionescu
You need to have independent control of some of these general gates and be able to select.
[19:35]
Vlad A. Ionescu
I want to apply these to the services that are in scope for this compliance versus I want to.
[19:41]
Vlad A. Ionescu
I want to enforce these things in this other way.
[19:45]
Vlad A. Ionescu
For things that maybe have different kinds of requirements from the compliance or security perspective.
[19:51]
Vlad A. Ionescu
Having this ability to insert in a single spot and create a system of record of the posture of each application.
[20:02]
Vlad A. Ionescu
When I mean system of record, I mean things like, is your application properly tested?
[20:07]
Vlad A. Ionescu
Is your application using the right security scanners, Is your application perhaps handling this kind of data or in scope for this other compliance and so on.
[20:19]
Vlad A. Ionescu
All of this posture information can then be used as a way to monitor things.
[20:25]
Vlad A. Ionescu
How is code coverage deviating over time and so on?
[20:28]
Vlad A. Ionescu
Or are you handling your vulnerabilities within the right timeframe?
[20:32]
Vlad A. Ionescu
I believe for many compliances is like 30 days for critical vulnerabilities, for example.
[20:37]
Vlad A. Ionescu
Then you create policies based on these things.
[20:39]
Vlad A. Ionescu
You can say things like, if your code coverage is not 80% or whatever, initially I'm going to monitor it, but over time I might even enforce it and give you feedback in your pr.
[20:52]
Vlad A. Ionescu
As soon as you open a PR and affect your code coverage, I will block your PR and be able to insert myself as a platform team without ever having to edit your CI cd.
[21:05]
Viktor Petersson
Yeah.
[21:06]
Viktor Petersson
So I've heard of people doing similar things with like.
[21:09]
Viktor Petersson
I know there are banks out there who are using like dependency track, for instance, to do like audit on SBOMs.
[21:14]
Viktor Petersson
And then they tie that into their policy agents to say, oh, if you have any cvs, block the deployment.
[21:20]
Viktor Petersson
Right?
[21:21]
Vlad A. Ionescu
Yes, yes.
[21:22]
Viktor Petersson
But that's also a little bit flawed because what if you already have that vulnerability and now you actually, you're improving the posture but you're still blocking the improvement, right?
[21:33]
Viktor Petersson
Yeah, it gets a little bit tricky.
[21:34]
Vlad A. Ionescu
Exactly.
[21:35]
Vlad A. Ionescu
You need what we call gradual enforcement.
[21:37]
Vlad A. Ionescu
And that is like each control you want to enforce has to be treated differently and we have like different levels.
[21:45]
Vlad A. Ionescu
So for example, you could initially just monitor things whether they're, you know, in a good state or not, and perhaps use offline sort of, I would call it an offline sort of campaigns where you kind of advocate for this new practice within the engineering teams.
[21:59]
Vlad A. Ionescu
You know, send them GitHub issues, send them JIRA tickets and whatnot, and get the percentage higher and higher.
[22:06]
Vlad A. Ionescu
And then when it's close to 100% or maybe 80, 90%, you announce that you're going to start blocking PRs.
[22:12]
Vlad A. Ionescu
Right.
[22:13]
Vlad A. Ionescu
And maybe after you start blocking PRs, then the next step is perhaps blocking production.
[22:17]
Vlad A. Ionescu
But sometimes there's, especially in the security space, there's going to be false positives and you want to be careful with blocking things unnecessarily.
[22:26]
Vlad A. Ionescu
So some things will forever stay in this mode where you kind of monitor things, but not necessarily strictly enforce them.
[22:33]
Vlad A. Ionescu
And you kind of couple that with the security team helping out, triage all the false positives and all that.
[22:40]
Vlad A. Ionescu
But we've seen this.
[22:42]
Vlad A. Ionescu
It's hard to have a hard and fast rule for everything.
[22:45]
Vlad A. Ionescu
And you need the flexibility to be able to say, these policies I want to be very strict about because I know they're going to cause incidents, whereas the other policies, I want to be more flexible because they could be false positive and such.
[22:59]
Vlad A. Ionescu
Right.
[23:00]
Viktor Petersson
There's a lot to unpack there.
[23:01]
Viktor Petersson
So the one thing that I'm curious about how you're approaching is the whole, like, problem of building.
[23:07]
Viktor Petersson
Right.
[23:08]
Viktor Petersson
Because, like, how do you monitor a build?
[23:11]
Viktor Petersson
Because you essentially, in order to do it properly, you can need to monitor every single request that your build does outbound.
[23:21]
Viktor Petersson
Right.
[23:21]
Viktor Petersson
You need to have like a firewall, like, oh, you spoke to.
[23:24]
Viktor Petersson
Like, oh, you're building a Debian package, but instead of pulling that, you pull in something completely random where you're like making random requests.
[23:31]
Viktor Petersson
Right?
[23:31]
Viktor Petersson
Yeah.
[23:32]
Viktor Petersson
Is that part of Pipeline as well?
[23:34]
Viktor Petersson
Or like, how granular in your access control do you get there?
[23:38]
Vlad A. Ionescu
Yeah, yeah.
[23:39]
Vlad A. Ionescu
So the way Lunar works is that we have a CICD agent that is one of our insertion points.
[23:44]
Vlad A. Ionescu
The other insertion point is that the code level doesn't depend on the cicd.
[23:48]
Viktor Petersson
Right.
[23:48]
Vlad A. Ionescu
But I think this pertains specifically to cicd.
[23:51]
Vlad A. Ionescu
So in cicd, the agent monitors every single pipeline that runs in your organization.
[23:56]
Vlad A. Ionescu
It runs alongside the CI runner.
[23:59]
Vlad A. Ionescu
If you run GitHub Actions on prem, for example, we run alongside that agent, essentially by understanding everything that runs in every single CI pipeline, we can see the types of processes that you execute.
[24:11]
Vlad A. Ionescu
We allow the user to say things like, every time I see MVN test being executed, I'm going to look for junit XML and put that in my metadata and collect that.
[24:22]
Vlad A. Ionescu
And maybe I have policies around some testing, for example.
[24:24]
Vlad A. Ionescu
Right.
[24:26]
Vlad A. Ionescu
So it's at that level of instrumentation where we look at the syscalls that are being made and are able to be very precise with extracting information from the right places now.
[24:39]
Vlad A. Ionescu
And there's like a collection of plugins that make this easier.
[24:43]
Vlad A. Ionescu
Now when it comes to network traffic, that oftentimes is more difficult.
[24:49]
Vlad A. Ionescu
You can create, you can see what IPs are being accessed, but a lot of times that can be very noisy when you try to create an allow list or a block list based on that.
[25:04]
Vlad A. Ionescu
What you can do instead is perhaps making sure that the different tools that are being used are configured to use the right artifactory or image registry and so on.
[25:13]
Vlad A. Ionescu
That tends to be a bit better from the, you know, the precision of it.
[25:19]
Viktor Petersson
Alternatively, like, you almost have to run everything through like a SOCKS proxy or something like a transparent proxy of sorts.
[25:25]
Viktor Petersson
So you can actually like get what are you actually pulling down.
[25:29]
Vlad A. Ionescu
Exactly.
[25:30]
Vlad A. Ionescu
That tends to be much better.
[25:31]
Vlad A. Ionescu
So like you might, yeah, you might have a kind of like isolated network and then you kind of know that you want to block, I don't know, maven central because you have your own mirror that has been kind of blessed internally.
[25:42]
Viktor Petersson
Right.
[25:43]
Vlad A. Ionescu
And that tends to be like a very effective strategy.
[25:46]
Vlad A. Ionescu
It does.
[25:47]
Vlad A. Ionescu
It kind of slows down teams because if they need like a net new dependency, they have to go through channels to kind of get it approved and everything.
[25:53]
Vlad A. Ionescu
If you're really, you know, big on security for whatever reason, you might be like a highly regulated industry and so on.
[26:02]
Vlad A. Ionescu
That tends to be probably worth it for.
[26:05]
Vlad A. Ionescu
For those kinds of organizations.
[26:06]
Viktor Petersson
Yeah.
[26:07]
Viktor Petersson
I had a Steve Springett on the show quite a few episodes ago talking about these as an attack vector for esp.
[26:13]
Viktor Petersson
Well, he was talking about for esports.
[26:15]
Viktor Petersson
But one of the interesting attack factors there is like, if you're using like private namespaces for private packages and then you have name squashing attacks essentially in the public space.
[26:27]
Viktor Petersson
Right.
[26:27]
Viktor Petersson
So if you know that Salesforce is using this internal package name, but then you go and register that in whatever the artifact, whatever the registry is that they're pulling, like npm, whatever, and then you can infiltrate that and just have a rogue version of the same thing.
[26:45]
Viktor Petersson
So that's an argument for having some kind of proxy that sits in between.
[26:48]
Vlad A. Ionescu
Right, exactly.
[26:50]
Vlad A. Ionescu
That sounds super dangerous.
[26:52]
Vlad A. Ionescu
And I mean, somebody's going to run that thing that NPM install with no authentication by accident, and it's going to pull the wrong thing from the Internet.
[27:00]
Vlad A. Ionescu
Right.
[27:01]
Viktor Petersson
It's just one environmental.
[27:02]
Viktor Petersson
You forgot to define some environment variable and now you're pulling from the Internet rather than from your self defined run.
[27:07]
Vlad A. Ionescu
Right, exactly.
[27:09]
Vlad A. Ionescu
Yeah.
[27:09]
Vlad A. Ionescu
That's very nasty.
[27:10]
Vlad A. Ionescu
That's very nasty.
[27:12]
Vlad A. Ionescu
It's hard to protect against everything, right?
[27:15]
Viktor Petersson
Yeah.
[27:15]
Viktor Petersson
I mean you kind of need to have this like in particular as.
[27:19]
Viktor Petersson
I mean it's annoying because you're adding friction.
[27:22]
Viktor Petersson
Right.
[27:22]
Viktor Petersson
Like one of the big upsides with the container revolution, if you call it may.
[27:27]
Viktor Petersson
Right.
[27:27]
Viktor Petersson
Was that it brought back autonomy to the teams.
[27:30]
Viktor Petersson
They're like, oh, we want to use this, we want to use that.
[27:32]
Viktor Petersson
But now we kind of come in full circle actually.
[27:34]
Viktor Petersson
But actually you probably shouldn't have done that because you actually didn't know the ramifications for doing that.
[27:40]
Viktor Petersson
You just want to solve the problem as quickly as possible.
[27:43]
Vlad A. Ionescu
Exactly, exactly.
[27:44]
Vlad A. Ionescu
Yeah.
[27:45]
Vlad A. Ionescu
There is such cases where oftentimes you can enforce for dependencies to be clearly like explicitly named so that it can only match the internal private one.
[27:57]
Vlad A. Ionescu
Right, yeah.
[27:58]
Vlad A. Ionescu
That it might not be obvious to enforce, but it is one of the things that we focus on as well.
[28:04]
Vlad A. Ionescu
If you have any custom verification, we allow you to write a central, I would call it a global CI.
[28:12]
Vlad A. Ionescu
It's like a definition of a bash set of bash instructions that apply to every single repository and you can use that to collect information about them.
[28:20]
Vlad A. Ionescu
So you might collect information about.
[28:22]
Vlad A. Ionescu
Okay, are you using this dependency by its ambiguous name or by the very specific private name?
[28:31]
Vlad A. Ionescu
And then you can sort of enforce that Through Lunar.
[28:34]
Vlad A. Ionescu
We've seen people come up with extremely custom and very specific things.
[28:42]
Vlad A. Ionescu
They want to add their enforcement based on perhaps past postmortems based on public incidents or based on just their engineering culture.
[28:51]
Vlad A. Ionescu
And I think this kind of flexibility is kind of difficult to define in traditional CICD pipelines.
[28:57]
Vlad A. Ionescu
You kind of need something that sort of runs beside it.
[29:01]
Vlad A. Ionescu
It's dedicated to a different sort of purpose.
[29:04]
Viktor Petersson
Yeah.
[29:04]
Viktor Petersson
And I mean it's going back a little bit to the whole point of like the difference between the fang of the world.
[29:09]
Viktor Petersson
Like they, they have internal tooling that can give you the best of both worlds almost.
[29:16]
Viktor Petersson
Right.
[29:16]
Viktor Petersson
They, they can provide all these guardrails without slowing down development.
[29:21]
Viktor Petersson
But most companies cannot do that.
[29:23]
Viktor Petersson
Right.
[29:23]
Viktor Petersson
That's, that's not realistic for more small to medium teams or even enterprise teams.
[29:28]
Vlad A. Ionescu
Right.
[29:28]
Viktor Petersson
They don't have their tooling for this, Right?
[29:31]
Vlad A. Ionescu
Yes, exactly.
[29:33]
Vlad A. Ionescu
And every company's culture is just very different.
[29:35]
Vlad A. Ionescu
You know, the way they've, the rules of how they deploy to production and you know, the different conventions they use internally.
[29:45]
Vlad A. Ionescu
It's hard to have a one size fits all Especially when, you know, like you have 10 microservices and every now and then one of those 10 microservices is going to be like the weird one.
[29:57]
Vlad A. Ionescu
It's going to have its own sort of deployment way and its own way to, I don't know, extract dependencies or whatnot.
[30:05]
Viktor Petersson
Well, yeah, it's just the pad, like, oh, this is an old application.
[30:08]
Viktor Petersson
We haven't upgraded.
[30:09]
Viktor Petersson
We know it's end of life, but we have no resources to actually update this.
[30:13]
Vlad A. Ionescu
Right, exactly.
[30:15]
Viktor Petersson
Either because it's external vendor.
[30:16]
Viktor Petersson
External vendor, that's sunset it.
[30:19]
Viktor Petersson
Or you just don't have the engineer resource internally to do it.
[30:23]
Vlad A. Ionescu
Yeah, exactly.
[30:24]
Vlad A. Ionescu
And that is sort of manageable when it's like the one odd out, you know, when you have like 10, 20, 30 microservices, but when you start to have like, I don't know, 70, 100, you know, 200 microservices, that's where something being weird is actually the norm.
[30:40]
Vlad A. Ionescu
There's always something weird in your infrastructure and that means you're going to have less ways to enforce proper engineering practices, basically, and security and all that.
[30:52]
Viktor Petersson
Yeah, I mean, I had a Dustin Kirkland from Chain Guard on a few episodes ago and we talked about the whole thing of like rolling versus stable releases, which is kind of a tangential problem space here, which.
[31:05]
Viktor Petersson
And he's a big fan obviously of rolling releases and like you deprecate things as they get deprecated, which is absolutely best engineering practices.
[31:16]
Viktor Petersson
I don't think anybody could make an argument against that.
[31:18]
Viktor Petersson
Well, I'm sure people can, but like, rational is very sensible.
[31:23]
Viktor Petersson
But it does create that problem of like, it's great if you have a good team with good culture, with good velocity, and you can just like, oh, let's fix it up.
[31:32]
Viktor Petersson
But yes, many, if not most engineer teams do not operate like that.
[31:37]
Viktor Petersson
Right?
[31:37]
Vlad A. Ionescu
Yeah, yeah, exactly.
[31:39]
Vlad A. Ionescu
Yeah, that's a great example.
[31:40]
Vlad A. Ionescu
And I think I, I watched that episode and he was saying like, okay, but now you have to upgrade things, but that's fine.
[31:47]
Vlad A. Ionescu
It's like, yeah, I forgot the analogy, but it sounded like, you know, it's either you break your legs and you.
[31:54]
Viktor Petersson
Know, maybe to break your leg or things like that.
[31:56]
Vlad A. Ionescu
Yes, you still break your legs.
[31:58]
Vlad A. Ionescu
Yeah, exactly.
[31:58]
Vlad A. Ionescu
Yeah.
[31:59]
Viktor Petersson
And it's spot on.
[32:00]
Viktor Petersson
Right.
[32:01]
Viktor Petersson
But like, the reality is that a lot of these positions are kind of held on a higher level.
[32:07]
Viktor Petersson
Like in a lot of.
[32:08]
Viktor Petersson
This is a cultural problem more so than a technical problem in many ways.
[32:11]
Viktor Petersson
Right.
[32:11]
Viktor Petersson
It's just like, yes, the budget is to shipping new features not to maintain what we have.
[32:16]
Viktor Petersson
That's how most companies operate.
[32:18]
Vlad A. Ionescu
Exactly, exactly.
[32:20]
Vlad A. Ionescu
Yeah.
[32:21]
Vlad A. Ionescu
Whereas some teams may be more like, you know, like a library team or whatnot and they have to support the use of that library for many different applications in different ways and the decision might be different.
[32:30]
Vlad A. Ionescu
So, so like having this freedom to create your own engineering process is both good, that you know you can use the right tools for your job, the right practices and so on.
[32:43]
Vlad A. Ionescu
It does come with its downsides and you kind of need to have a good plan around, you know, managing those downsides.
[32:49]
Viktor Petersson
Yeah, it's this constant like battle and you kind of need to not stop it.
[32:55]
Viktor Petersson
We at Squeanly, we do what we call system tools week.
[32:59]
Viktor Petersson
So we do two week sprints and then one week system tools, where that week is earmarked, generally speaking, towards like improving our internal tooling.
[33:09]
Viktor Petersson
And you kind of need this time, but essentially generally speaking, because if you just jump from Sprint to Sprint and in small to mid sized team, you're never going to get around to fix these problems, Right?
[33:21]
Vlad A. Ionescu
Yes.
[33:22]
Vlad A. Ionescu
So yeah, yeah.
[33:24]
Viktor Petersson
All right, so let's switch gear a little bit to culture, because I think that's something you spend a lot of time on, engineering, management and culture, which is kind of like.
[33:34]
Viktor Petersson
Well, I guess it's a good segue over to that from that.
[33:38]
Viktor Petersson
And we're both fans of remote teams and building and scaling remote teams.
[33:45]
Viktor Petersson
So maybe let's kick it off there.
[33:47]
Viktor Petersson
Like, what have you learned over the years building and managing engineering teams?
[33:53]
Viktor Petersson
I'm really curious about your input and your thinking around that and what you've learned over the years.
[33:58]
Vlad A. Ionescu
Yeah, And I should preface this by saying that I've managed startup engineering teams, which is perhaps a bit different than FAANG type of teams, right?
[34:07]
Viktor Petersson
Oh, absolutely.
[34:08]
Viktor Petersson
Yeah.
[34:09]
Vlad A. Ionescu
Right.
[34:09]
Vlad A. Ionescu
But I think one of the key decisions as a startup is how do you find talent that is extremely effective and doesn't, you know, doesn't burn your Runway, you know, too fast.
[34:22]
Vlad A. Ionescu
And as a startup founder, you oftentimes, it's not as critical these days, but you oftentimes have to be in the Silicon Valley area to kind of have access to the investors.
[34:32]
Vlad A. Ionescu
For whatever reason, they see that as a benefit, like a dedication, your commitment and whatever.
[34:38]
Viktor Petersson
I would say I used to be so more so 15 years ago or 20 years ago than today, I would argue.
[34:43]
Vlad A. Ionescu
But yeah, 100%.
[34:44]
Vlad A. Ionescu
Ever since COVID I think even that has changed dramatically.
[34:47]
Vlad A. Ionescu
but I happen to be here in the Bay Area and hiring here in the Bay Area is Like, extremely difficult.
[34:55]
Vlad A. Ionescu
You know, like these total compensation packages are.
[34:57]
Vlad A. Ionescu
Are wild.
[34:58]
Vlad A. Ionescu
Especially when they're.
[34:59]
Vlad A. Ionescu
There's, you know, the companies they work for have a certain stock that has been growing for the last few years.
[35:05]
Vlad A. Ionescu
It's not unheard of to have like 500k sort of total compensation.
[35:10]
Viktor Petersson
I saw somebody done a comparison like on, I think, like, I think it was a Level 6 engineer at Facebook.
[35:16]
Viktor Petersson
It's like a total comp is like a million or something like that, north of a million.
[35:19]
Viktor Petersson
And it's just like, yeah, you can't compete with that as a normal company.
[35:23]
Viktor Petersson
That's just exactly.
[35:24]
Vlad A. Ionescu
Yeah.
[35:25]
Vlad A. Ionescu
I mean, imagine, you know, seed stage startup might have raised, I don't know, like, say six, $7 million.
[35:30]
Vlad A. Ionescu
Like, does that mean for two years running away, I'm gonna just afford three engineers, you know, like.
[35:36]
Vlad A. Ionescu
Yeah, that cannot be right.
[35:37]
Viktor Petersson
No.
[35:38]
Viktor Petersson
Right, exactly.
[35:39]
Vlad A. Ionescu
So marrying that sort of requirement of having high talent but doesn't break the bank type of thing.
[35:46]
Vlad A. Ionescu
At my previous company at Shift Left nowadays called Quiet AI, I led the engineering team and with my colleague, the cto, we built a team from scratch.
[35:56]
Vlad A. Ionescu
And we looked internationally, mostly.
[36:00]
Vlad A. Ionescu
Right.
[36:00]
Vlad A. Ionescu
We looked one of the centers that we found our chief scientist and actually the person who ended up creating most of our secret sauce, if you will.
[36:11]
Vlad A. Ionescu
We were able to hire in Germany.
[36:15]
Viktor Petersson
That's an odd choice.
[36:16]
Viktor Petersson
That's not something you normally hear.
[36:18]
Viktor Petersson
Like, usually people stay away from Germany because they have very tight labor laws and it's just very difficult to hire there.
[36:25]
Vlad A. Ionescu
That's true.
[36:26]
Vlad A. Ionescu
That's true.
[36:28]
Vlad A. Ionescu
I think it was not necessarily our top choice because of what you're saying, but also this person was in Germany and actually it worked out really well for us.
[36:38]
Vlad A. Ionescu
And so we built a team around this person in Germany.
[36:43]
Vlad A. Ionescu
And we also hired like about a third of our team was fully remote.
[36:47]
Vlad A. Ionescu
So there were people from like New Zealand or Argentina or Brazil and so on.
[36:53]
Vlad A. Ionescu
And perhaps one of the things I learned early on was that this sort of time zone scattering is extremely difficult to manage.
[37:01]
Vlad A. Ionescu
Even us, the central office here in the Bay Area versus Germany, the time never overlapped.
[37:09]
Vlad A. Ionescu
Somebody had to make a compromise to be able to get on a simple meeting.
[37:12]
Vlad A. Ionescu
They had to stay up later or we had to get up earlier.
[37:19]
Vlad A. Ionescu
That created echo chambers.
[37:20]
Vlad A. Ionescu
We were not aligned from the product standpoint.
[37:23]
Vlad A. Ionescu
In some ways.
[37:25]
Vlad A. Ionescu
You have to find ways to work around that and be able to collaborate better this time around.
[37:31]
Vlad A. Ionescu
As I built the new engineering team here at Earthly, I knew that we had to go remote because of that sort of cost aspect.
[37:38]
Vlad A. Ionescu
But at the same time maintain roughly the same time zones.
[37:42]
Vlad A. Ionescu
Right.
[37:43]
Vlad A. Ionescu
And the prices for engineers are not as good here, sort of in this time zone as they are in Europe.
[37:51]
Vlad A. Ionescu
Basically.
[37:51]
Vlad A. Ionescu
In Europe you get half price, essentially, which is like crazy, right?
[37:55]
Viktor Petersson
Yeah, sounds about right.
[37:56]
Viktor Petersson
Yeah.
[37:57]
Vlad A. Ionescu
Whereas here you still have to pay significant amount of money for the top talent.
[38:03]
Vlad A. Ionescu
But it allowed us to be within the same time zone.
[38:08]
Vlad A. Ionescu
Especially in a startup where the product definition evolves very quickly, you have to be constantly in sync with the engineers.
[38:16]
Vlad A. Ionescu
I think that's key.
[38:18]
Vlad A. Ionescu
I think if you want to manage a remote team, you have to have a very stable and very well defined API between that team and the rest of engineering.
[38:29]
Vlad A. Ionescu
So for example, a great example I've seen at another company was they had the payments team in Turkey, for example.
[38:37]
Vlad A. Ionescu
Right.
[38:37]
Vlad A. Ionescu
Then payments is kind of like, okay, you know, you have to kind of swap for credit card.
[38:42]
Vlad A. Ionescu
You just have to support all these different countries and integrate with Stripe and a bunch of other vendors and so on.
[38:46]
Vlad A. Ionescu
Yeah, but it's kind of like a well known entity.
[38:49]
Vlad A. Ionescu
It doesn't have like dynamic product requirements.
[38:53]
Vlad A. Ionescu
And I think that tends to work really well if you can have that sort of API between teams.
[39:00]
Vlad A. Ionescu
Otherwise I personally prefer to kind of have teams centralized from the time zone perspective.
[39:08]
Vlad A. Ionescu
But also we're embracing the full remote culture.
[39:11]
Vlad A. Ionescu
And when you embrace the full remote culture, you also have to have frequent off sites.
[39:15]
Vlad A. Ionescu
You have to kind of get together frequently and be able to kind of get on the same page.
[39:22]
Vlad A. Ionescu
Sometimes our thinking and ideas deviate over time, even if you have like daily standups or weekly standups or whatnot.
[39:30]
Vlad A. Ionescu
And that kind of helps get everyone on the same page and sort of revitalize the energy of the team and so on.
[39:36]
Viktor Petersson
100%.
[39:37]
Viktor Petersson
I agree with the off sites.
[39:40]
Viktor Petersson
When we started doing that about almost a decade ago, like the culture ship was so phenomenally different.
[39:47]
Viktor Petersson
Like you could just.
[39:48]
Viktor Petersson
It's just so different to have an interact with somebody, even if it's a video call, versus sitting down, having a beer with somebody.
[39:56]
Viktor Petersson
Like you get to know people at a completely different resolution.
[39:59]
Viktor Petersson
Right.
[40:00]
Viktor Petersson
You understand like the nuances and that helps smooth things over a lot more when you go back to a video call once you've had those face to face interactions.
[40:12]
Vlad A. Ionescu
Right, exactly.
[40:14]
Vlad A. Ionescu
And the key thing is building trust.
[40:16]
Vlad A. Ionescu
Right.
[40:16]
Vlad A. Ionescu
It's a bit harder when you're a remote team.
[40:19]
Vlad A. Ionescu
You have to rely heavily on async type of communication, written documents and all that.
[40:24]
Vlad A. Ionescu
And that trust element is sometimes lost in the communication there because you're brief, you Kind of directly say what works, doesn't work or what things to be different and people might perceive that negatively or like adversarial if you will.
[40:41]
Vlad A. Ionescu
Right, yeah.
[40:43]
Vlad A. Ionescu
Whereas in person there's more nuance.
[40:47]
Vlad A. Ionescu
I'm a big fan of team bonding.
[40:50]
Vlad A. Ionescu
So one of the first things we do in off sites is just spend the day hanging out, you know, like.
[40:56]
Vlad A. Ionescu
Yeah, doing non work stuff.
[40:57]
Vlad A. Ionescu
You know, we actually put that as the first day, the fun day.
[41:00]
Viktor Petersson
You know it's funny because a friend of mine reached out because they were doing the first stand up first off site for his new company and he was asking me like what my experience was like and actually very much to your point.
[41:13]
Viktor Petersson
To me the first day is a write off because it's just obviously we travel all that depending on how many times but beyond that it's about getting together and just like even as a manager, like, or leader, see how the dynamics happen organically within a team.
[41:32]
Viktor Petersson
That's super important as well.
[41:34]
Viktor Petersson
And see how other people, how the teams interface with each other as well.
[41:38]
Vlad A. Ionescu
Yes, exactly.
[41:39]
Viktor Petersson
That was like one of my first, like make sure it's at least three days because less than that is just pointless in my opinion.
[41:46]
Viktor Petersson
And first day and last day write offs because like first day of arrival, people tried, people have been up early, like it's not going to be a productive day.
[41:55]
Viktor Petersson
Last day of travel, same thing.
[41:57]
Viktor Petersson
Like people get ready for the airport, people have different flights.
[41:59]
Viktor Petersson
At least three days would be one day of actual productivity.
[42:03]
Vlad A. Ionescu
Exactly, exactly.
[42:04]
Vlad A. Ionescu
And like being locked in a single room for, I don't know, eight hours in that single day is also exhausting.
[42:10]
Vlad A. Ionescu
So you kind of wanted to use that energy with a lot of intentionality, I guess.
[42:16]
Viktor Petersson
Yeah, 100%.
[42:17]
Viktor Petersson
So there's a lot of like, I guess definitions around these, like remote first, remote hybrid, all those stuff.
[42:25]
Viktor Petersson
Right.
[42:26]
Viktor Petersson
And now obviously now we get to see the, the boomer way back to the office and all that stuff.
[42:32]
Viktor Petersson
But that's a whole different conversation.
[42:36]
Viktor Petersson
But I guess you guys are not really remote first because you do have an office as well.
[42:41]
Viktor Petersson
Or, or do you consider yourself entirely remote first?
[42:44]
Vlad A. Ionescu
No, actually at this new company we no longer have any office and we never had an office.
[42:50]
Vlad A. Ionescu
We started at my previous company.
[42:52]
Vlad A. Ionescu
We did, were hybrid.
[42:54]
Viktor Petersson
Right, okay.
[42:55]
Vlad A. Ionescu
And perhaps this other lesson is hybrid is actually much more difficult than fully remote.
[43:00]
Viktor Petersson
Yes.
[43:01]
Vlad A. Ionescu
Because there's going to be like, you know, like people talking in the office about certain things in person, making decisions, moving fast and then the rest of the team is like, okay, but wait, what happened?
[43:12]
Vlad A. Ionescu
We talked about this other thing last week.
[43:14]
Vlad A. Ionescu
You have to now catch me up or whatever.
[43:16]
Vlad A. Ionescu
You know, so much always left behind.
[43:20]
Viktor Petersson
We've coined the phrase.
[43:21]
Viktor Petersson
It's the water cooler effect, right.
[43:23]
Viktor Petersson
Where it's informal communication that happens in an office.
[43:27]
Viktor Petersson
So we try to mimic that in Slack essentially, like have a water cooler.
[43:33]
Viktor Petersson
Like sometimes it works, sometimes it doesn't.
[43:35]
Viktor Petersson
But it's like it's not quite work, but it's stuff that would happen in regular work environment, right?
[43:40]
Vlad A. Ionescu
Yeah, yeah, exactly.
[43:43]
Vlad A. Ionescu
There's a trick for.
[43:44]
Vlad A. Ionescu
We use this.
[43:47]
Vlad A. Ionescu
You have like a little schedule, like a rotation.
[43:50]
Vlad A. Ionescu
Every person asks an interesting question per day in the water cooler channel.
[43:55]
Vlad A. Ionescu
And we found that sparks like interesting conversation that would otherwise not happen because otherwise we're kind of focused on work and like, yeah, you know, nobody wants to talk about, I don't know, cats or whatever.
[44:05]
Vlad A. Ionescu
But if you ask like informal questions that about something fun and every team member comes with something every day, it creates much more sort of engaging conversation.
[44:18]
Vlad A. Ionescu
It's not necessarily about catching up on work or anything.
[44:22]
Vlad A. Ionescu
It doesn't necessarily solve that problem, but it solves a little bit of the trust problem that perhaps is often missing in async communication.
[44:29]
Viktor Petersson
I mean trust is such an important piece here.
[44:33]
Viktor Petersson
I wrote a blog post that made its way to Hacker News a while ago about a decade of remote work.
[44:39]
Viktor Petersson
So I basically captured what I've learned or doing like remotely for a decade.
[44:43]
Viktor Petersson
But one of the biggest things is kind of.
[44:46]
Viktor Petersson
We already touched on the first one being either you go all remote or you all in office.
[44:52]
Viktor Petersson
I think hybrid is kind of like an illusion because.
[44:55]
Viktor Petersson
Yeah, it's just this particular about documentation.
[44:59]
Viktor Petersson
It's a big one.
[45:00]
Viktor Petersson
Like how do you document things?
[45:02]
Viktor Petersson
Like if you're a remote first company.
[45:04]
Viktor Petersson
And obviously there are plenty of good examples, GitLab being one of the good ones about this.
[45:09]
Viktor Petersson
Right?
[45:09]
Viktor Petersson
Then you have a documentation culture is different.
[45:13]
Viktor Petersson
Whereas if you're a kind office based company, there are a lot of like unofficial ways and like wisdom like are spread by asking people walking over to their desks, right?
[45:28]
Vlad A. Ionescu
Yeah, yeah.
[45:29]
Vlad A. Ionescu
It can get a bit political too because like you know this person better than this other person and you're gonna talk with that person about, you know, the requirements or you know, the goals and whatnot much more than this other person.
[45:41]
Vlad A. Ionescu
So there is subtle ways in which you accidentally empower more people differently across the team.
[45:50]
Vlad A. Ionescu
Plus I think there's with this new RTO return to office sort of thing that's Going on right now.
[45:57]
Vlad A. Ionescu
I feel like companies are the norm is now hybrid, which is kind of bad, right?
[46:03]
Vlad A. Ionescu
I mean honestly, in the office and so on is like somewhere.
[46:07]
Viktor Petersson
Honestly, the RTO thing is just a big mass firing in disguise.
[46:13]
Viktor Petersson
That's really what it is, right?
[46:15]
Vlad A. Ionescu
That's, yeah, that's, yes.
[46:17]
Viktor Petersson
It's not like, yeah, I, I don't believe in it.
[46:21]
Viktor Petersson
I think it's just people want an easy way to weed out the people who are not as dedicated and that is NAR policy is like, okay, cool, 20 plus people don't want to come back.
[46:31]
Viktor Petersson
Cool fire dude.
[46:32]
Viktor Petersson
Done.
[46:33]
Vlad A. Ionescu
Yeah, yeah.
[46:34]
Vlad A. Ionescu
The sad thing about it is like a lot of times you're going to weed out the people that perhaps have kids or something.
[46:42]
Vlad A. Ionescu
Like it's a bit of discrimination in disguise.
[46:45]
Vlad A. Ionescu
I, it's, you can't really put your finger on it, but it is going to sort of affect different people differently because of that.
[46:52]
Viktor Petersson
Oh man, I, I, there's no way I'd have a desk job where I had to go and commute in every day.
[46:56]
Viktor Petersson
Like no way.
[46:56]
Viktor Petersson
I would never do it.
[46:57]
Viktor Petersson
I've done remote now for like what, 15 plus years.
[47:00]
Viktor Petersson
Like there's no way I'm going to allocate an hour plus a day commuting.
[47:04]
Viktor Petersson
Like there's no way, like it's just a way better way to spend that time.
[47:07]
Viktor Petersson
So.
[47:08]
Vlad A. Ionescu
Absolutely.
[47:11]
Viktor Petersson
The other thing you mentioned that I have the same kind of observation as you have is trust when it comes to remote.
[47:16]
Viktor Petersson
Right.
[47:17]
Viktor Petersson
The trust is so difficult, but it's kind of like the whole agree up remote.
[47:22]
Viktor Petersson
Because if there's no trust like things, there are a lot of cultural implications of that.
[47:31]
Viktor Petersson
Right?
[47:31]
Viktor Petersson
There are a lot of things.
[47:32]
Viktor Petersson
But speak to me about how you view this and how you've kind of building accountability and like as part of this.
[47:44]
Viktor Petersson
Because remote is difficult, right?
[47:45]
Viktor Petersson
Because yeah.
[47:47]
Viktor Petersson
The one thing you could say about an office is like, oh yeah, you showed up cool.
[47:50]
Viktor Petersson
But you can't quite say that about somebody working remotely and Right.
[47:56]
Vlad A. Ionescu
Oh yeah.
[47:57]
Vlad A. Ionescu
Like how do you measure people and make sure that they're actually at work?
[48:01]
Vlad A. Ionescu
Right.
[48:01]
Vlad A. Ionescu
And yes, and there have been so many bad takes on this, you know, like spying on employees, computers, making sure they're like moving their mouse or whatever.
[48:11]
Viktor Petersson
Yeah, yeah.
[48:12]
Vlad A. Ionescu
I think that's bad.
[48:13]
Vlad A. Ionescu
Right.
[48:15]
Vlad A. Ionescu
If you get to that point where you have no idea if your employees are productive, you're not measuring your employees properly.
[48:21]
Vlad A. Ionescu
Properly.
[48:21]
Vlad A. Ionescu
Right.
[48:22]
Vlad A. Ionescu
You should be measuring them on output, not on activity.
[48:26]
Viktor Petersson
Yeah, that's so much so true.
[48:28]
Viktor Petersson
Right.
[48:29]
Viktor Petersson
Because like one of the things that I, I'm fully on board with is like if my people are working remotely, I'm okay with them going and doing things.
[48:38]
Viktor Petersson
Make sense.
[48:38]
Viktor Petersson
If they want to go and get their haircut at like 2pm on a Wednesday and they have no other meetings, like, go ahead, like, why not?
[48:45]
Viktor Petersson
That's fine.
[48:46]
Viktor Petersson
I don't care.
[48:46]
Viktor Petersson
Like just as long as you get your stuff done.
[48:48]
Viktor Petersson
Right.
[48:49]
Viktor Petersson
Whatever.
[48:50]
Vlad A. Ionescu
Yeah.
[48:50]
Vlad A. Ionescu
And the way I've also talked to remote teams about this is that, hey, you can take your kid to daycare, do your haircut, go shopping, whatever.
[49:00]
Vlad A. Ionescu
You need to be flexible.
[49:02]
Vlad A. Ionescu
And occasionally we will ask for you to be flexible for us.
[49:06]
Vlad A. Ionescu
An incident on the weekend and whatnot.
[49:08]
Vlad A. Ionescu
Two way streets, so.
[49:09]
Vlad A. Ionescu
Exactly.
[49:10]
Vlad A. Ionescu
We, we don't typically ask that many times, you know, but I think everyone would prefer if that were the case more frequently than not.
[49:19]
Vlad A. Ionescu
Right.
[49:23]
Vlad A. Ionescu
I started my career actually back in Romania where post communist country, perhaps the culture was much more based on measuring activity as opposed to output.
[49:35]
Vlad A. Ionescu
Kind of like you were doing communism regimes.
[49:38]
Vlad A. Ionescu
Exactly.
[49:38]
Vlad A. Ionescu
Low trust and all that.
[49:40]
Vlad A. Ionescu
And so I just took it for granted initially.
[49:43]
Vlad A. Ionescu
Oh, it's nine to five, I really have to be, if I'm late by five minutes, I have to kind of put that in the system and all that.
[49:51]
Vlad A. Ionescu
I now found it extremely ridiculous that many companies have just continued that way.
[49:58]
Vlad A. Ionescu
Not just in Romania, but in the western world a lot as well.
[50:03]
Vlad A. Ionescu
Right.
[50:03]
Viktor Petersson
But I think that's a boomer mindset.
[50:05]
Viktor Petersson
Right.
[50:06]
Viktor Petersson
It's like, oh, but in the desk.
[50:08]
Viktor Petersson
But like in the chair at the desk, like that's more important than what you get as output, right?
[50:14]
Vlad A. Ionescu
Yeah, yeah, perhaps.
[50:16]
Vlad A. Ionescu
Exactly.
[50:17]
Vlad A. Ionescu
And I think there is something to be said about hiring the right middle management and lower management teams.
[50:26]
Vlad A. Ionescu
Right.
[50:26]
Vlad A. Ionescu
If, if those teams are unable to properly measure the output of their employees, that's a much bigger problem than your ICs not working or your not being able to trust them.
[50:37]
Vlad A. Ionescu
Right.
[50:38]
Vlad A. Ionescu
So I'm a fan of hiring people that are self sufficient, that they can be the kind of like the CEO of their own jobs.
[50:47]
Vlad A. Ionescu
Right.
[50:48]
Vlad A. Ionescu
And trust them completely on what they execute on and work with them sort of on results and outcome and such as opposed to, yeah, you know, the nuts and bolts of like, were you, where was your butt in the seat at this time and whatnot.
[51:05]
Viktor Petersson
Yeah.
[51:05]
Viktor Petersson
This is interesting.
[51:06]
Viktor Petersson
Right?
[51:06]
Viktor Petersson
Like, and have you, is there any framework that you adhere to in terms of quantifying these things that you found working better than others?
[51:18]
Vlad A. Ionescu
We don't have like a very formal framework, and I think that would be kind of natural for a small company like ours.
[51:26]
Vlad A. Ionescu
I think at scale you probably need some sort of framework, but it's probably similar to career ladders and such.
[51:33]
Vlad A. Ionescu
You know, you typically have performance reviews and you measure people based on what they were able to achieve and such.
[51:41]
Vlad A. Ionescu
I think those are great on their own for the measurement of, you know, was this employee present at work?
[51:50]
Vlad A. Ionescu
Sort of from the execution standpoint and less so, you know, other sort of information.
[51:58]
Vlad A. Ionescu
I've seen big companies, though, insert things like, you know, badging data and computer activity data in their performance review.
[52:08]
Vlad A. Ionescu
I think that's kind of backwards.
[52:10]
Viktor Petersson
Yeah, yeah, I'd agree with that.
[52:13]
Vlad A. Ionescu
There's.
[52:14]
Viktor Petersson
There's a framework that I kind of like, which is entrepreneurial operating system.
[52:17]
Viktor Petersson
They have like.
[52:18]
Viktor Petersson
They have like three very simple questions for like, performance reviews.
[52:21]
Viktor Petersson
It's funny because I was working, literally working on this today, so it's fresh in mind.
[52:24]
Viktor Petersson
But they have like, can you do it?
[52:27]
Viktor Petersson
Do you have the capacity to do it?
[52:28]
Viktor Petersson
And do you want to do it?
[52:29]
Viktor Petersson
Those are like the three main questions, like, if you can, like, those are good determinations of, like, is this a good fit for the role?
[52:36]
Viktor Petersson
And if you're delivering it.
[52:39]
Viktor Petersson
Yeah, it's very distilled down.
[52:40]
Viktor Petersson
The simple.
[52:41]
Vlad A. Ionescu
Yeah, that makes a lot of sense.
[52:43]
Vlad A. Ionescu
I think those three questions are probably the.
[52:46]
Vlad A. Ionescu
The best balance.
[52:47]
Vlad A. Ionescu
You don't want to create like, you know, a review system that takes everybody offline for a whole week or something.
[52:55]
Viktor Petersson
Yeah.
[52:55]
Vlad A. Ionescu
And you still want the data from it.
[52:57]
Vlad A. Ionescu
You kind of have to balance the sort of precision and number of questions and all that.
[53:02]
Viktor Petersson
Absolutely.
[53:03]
Viktor Petersson
Is there one lesson that you would say for somebody who's new to managing a remote engineering team that they need to get, like, what's the one advice you would give them?
[53:17]
Vlad A. Ionescu
Great question.
[53:18]
Vlad A. Ionescu
I would say I can think of two or three here.
[53:23]
Vlad A. Ionescu
So one I kind of already mentioned, which is think about time zones and how your teams are going to collaborate.
[53:29]
Vlad A. Ionescu
Right.
[53:30]
Vlad A. Ionescu
I think that's critical.
[53:33]
Vlad A. Ionescu
The other part of it is perhaps how do you work together and managing the async sort of communication style and being fully bought it into that style.
[53:48]
Vlad A. Ionescu
Like, default to documenting decisions, default to reviewing design documents, default to relying on GitHub comments and Slack messages as opposed to word of mouth and such.
[54:05]
Vlad A. Ionescu
This is kind of like a muscle.
[54:07]
Vlad A. Ionescu
You have to.
[54:08]
Vlad A. Ionescu
Maybe it's a bit unintuitive for someone who's new to this, but if you trust these, a few official tools, I Think that's also important.
[54:18]
Vlad A. Ionescu
Don't use both notion and drive and so on.
[54:20]
Vlad A. Ionescu
You have to kind of standardize on a limited set of tools and then any communication that goes through that tool is just as official as any other channel.
[54:32]
Vlad A. Ionescu
Right.
[54:33]
Vlad A. Ionescu
And investing deeply in that is really important.
[54:37]
Vlad A. Ionescu
So I think that's perhaps my top two sort of takes on remote work.
[54:44]
Viktor Petersson
Fair enough, Fair enough.
[54:46]
Viktor Petersson
All right, let's switch a bit more gear to something that is kind of more back to your story before, which is AI in the pipeline, I guess, and more specifically in the sdlc really.
[55:00]
Viktor Petersson
And I, I'm curious about your take on that.
[55:03]
Viktor Petersson
Like I, I foresee a world where AI will write code and AI will review code and then we have agents talking to agents and then there's probably some human loop somewhere to review and make sure that nothing crazy is going on.
[55:17]
Viktor Petersson
But talk to him more about how you see that shifting given that you are deep inside the STLC world.
[55:25]
Viktor Petersson
Really.
[55:26]
Vlad A. Ionescu
Yeah.
[55:27]
Vlad A. Ionescu
There's a few ways this could go towards.
[55:29]
Vlad A. Ionescu
Right.
[55:29]
Vlad A. Ionescu
And I think it's hard to really predict the future, but I can think of the far future type of system is where they're no longer software engineers, they're more like PM's writing prompts.
[55:44]
Vlad A. Ionescu
Right.
[55:44]
Vlad A. Ionescu
And iterating with a system.
[55:46]
Vlad A. Ionescu
And it's sort of all taken care of behind the scenes.
[55:48]
Vlad A. Ionescu
Perhaps in such an environment there's more reason to standardize the tech stack.
[55:53]
Vlad A. Ionescu
Like if the AI is going to write everything, why not write it in a single programming language?
[55:57]
Vlad A. Ionescu
And maybe it's like extremely geared towards AI type of workloads.
[56:03]
Vlad A. Ionescu
So maybe it's still transparent so you can see and audit the logic behind the scenes and so on.
[56:09]
Vlad A. Ionescu
So it's still human readable, but it's not as necessarily as diverse.
[56:14]
Vlad A. Ionescu
So that's one possible option.
[56:15]
Vlad A. Ionescu
I think in the short and medium term there's going to be a lot of hybrid work between human and machine, if you will.
[56:23]
Vlad A. Ionescu
And in this hybrid work, kind of like with self driving cars, the problem would be so much easier if there were no humans on the street.
[56:30]
Vlad A. Ionescu
Right.
[56:30]
Vlad A. Ionescu
You couldn't have beacons and we have no car accidents.
[56:35]
Vlad A. Ionescu
Exactly, yeah.
[56:37]
Vlad A. Ionescu
And the uncertainty of it all just goes away tremendously.
[56:42]
Vlad A. Ionescu
Right.
[56:43]
Vlad A. Ionescu
And that is kind of similar with AI.
[56:45]
Vlad A. Ionescu
Right.
[56:46]
Vlad A. Ionescu
You still have to rely a lot on human engineers and this might be the norm for a very long time for all we know.
[56:54]
Vlad A. Ionescu
We have no idea how fast it's to evolve.
[56:56]
Vlad A. Ionescu
Right.
[56:56]
Vlad A. Ionescu
And in this short and medium sort of time span we probably need to treat computers with the same care as sort of an engineer to some extent.
[57:07]
Vlad A. Ionescu
So either that way you say like, you know, an AI agent is an engineer and you apply the same principles.
[57:14]
Vlad A. Ionescu
Somebody has to review their code, you have to scan their for their stuff for vulnerabilities, you have to enforce engineering standards and so on.
[57:22]
Vlad A. Ionescu
And so that's one way to see it.
[57:24]
Vlad A. Ionescu
Or perhaps the AI is like an extension of the human and everything the human authors is with the help of AI.
[57:31]
Vlad A. Ionescu
I think that is where we are right now.
[57:33]
Vlad A. Ionescu
Right.
[57:34]
Vlad A. Ionescu
And the human might be a little bit sometimes careless about what the AI introduces.
[57:41]
Vlad A. Ionescu
And for that reason you need even more verification on top to make sure that no strange things have been introduced, no vulnerabilities have been sort of added to the code base.
[57:55]
Vlad A. Ionescu
Maybe whatever training the AI had was a bit out of date and using old dependencies or whatnot.
[58:03]
Vlad A. Ionescu
Exactly.
[58:04]
Vlad A. Ionescu
Right.
[58:04]
Vlad A. Ionescu
You want to make sure you're not sort of behind the curve of what security today is in terms of dependencies and all that, or engineering practices that are specific to your company.
[58:17]
Vlad A. Ionescu
So I think in the short term there's much bigger need for this kind of guardrails and verification and making sure everything is abiding according to engineering standards and security standards that you want to enforce in your organization.
[58:36]
Vlad A. Ionescu
So yeah, I think over time we're kind of learning as an industry how to manage all this.
[58:44]
Vlad A. Ionescu
But definitely it's a huge shift that is perhaps changing our mindsets the way we think about SDLC and all that.
[58:53]
Viktor Petersson
Yeah, I mean, I'm a big subscriber of your first vision.
[58:56]
Viktor Petersson
Right.
[58:56]
Viktor Petersson
I think engineers will essentially become managers of AI agents.
[59:02]
Viktor Petersson
Right, right.
[59:03]
Viktor Petersson
And, but this is the important part because I'm sure you saw this broke this week as were recording on Twitter, this guy who wrote some Microsoft, wrote some Sauce product and that is like, oh, holy shit.
[59:16]
Viktor Petersson
People are injecting stuff into my database, they're bypassing payments.
[59:19]
Viktor Petersson
I have no idea what this, my code base does.
[59:21]
Viktor Petersson
And it's like he was a non engineer who written some piece of software using Cursor, essentially, or the likes of Cursor.
[59:28]
Viktor Petersson
Right.
[59:28]
Viktor Petersson
And it's just like there is a scenario where we will have these AIs who can write good enough code autonomously that a non engineer can manage.
[59:42]
Viktor Petersson
But I think that's pretty far out, right?
[59:45]
Vlad A. Ionescu
It is, it is.
[59:46]
Vlad A. Ionescu
And going at it with a naive mindset like I imagine a non technical CEO is going to assume.
[59:54]
Vlad A. Ionescu
Okay, we have to kind of heavily rely on AI now to be efficient and they're going to just create a lot of chaos, you know, because like that AI is going to write stuff that is pretty much a black box for everyone else.
[01:00:07]
Vlad A. Ionescu
And it's going to be riddled with vulnerabilities and like duplicated code, no logic complete.
[01:00:14]
Viktor Petersson
That's the funny thing.
[01:00:15]
Viktor Petersson
Like if you think about logically, it's probably cheaper for an LLM to write new code than to maintain existing code.
[01:00:23]
Vlad A. Ionescu
Oh yeah, 100%, yes.
[01:00:25]
Viktor Petersson
So it will.
[01:00:27]
Viktor Petersson
It's just got to duplicate everything like a third time, right?
[01:00:30]
Vlad A. Ionescu
Yes, yes, exactly.
[01:00:32]
Vlad A. Ionescu
Yeah.
[01:00:33]
Viktor Petersson
So I had this conversation with some friends a few weeks ago and I'm basically like, my big bet is that in the next 12 months we're going to have a lot of these companies that managed to build a product, manage to capture revenue, and then they realized they have no idea how the price written, what it does behind the scenes.
[01:00:53]
Viktor Petersson
It's completely a black box to your point.
[01:00:56]
Viktor Petersson
And then they're gonna, well, they're gonna be in for a world of pain and there are probably gonna be a lot of people who gotta make a lot of money trying to savage the disaster that is their code base.
[01:01:10]
Vlad A. Ionescu
Oh yeah, oh yeah, exactly.
[01:01:12]
Vlad A. Ionescu
And catching up from that, you're just kind of digging a hole for yourself.
[01:01:16]
Vlad A. Ionescu
And getting out of that hole is going to be very expensive at the end of the day.
[01:01:19]
Vlad A. Ionescu
Like the original goal might have been to save money, but you know what, it's not actually going to work out that way in the long term.
[01:01:27]
Viktor Petersson
No, I mean there's an argument that is that, well, these product would have not existed otherwise, so therefore it's a net benefit.
[01:01:33]
Viktor Petersson
But you can still, like there is probably like a happy medium.
[01:01:37]
Viktor Petersson
Like you still need like AI is not going to replace engineers anytime soon.
[01:01:42]
Viktor Petersson
It's going to 10x good engineers, but it's probably going to hurt shitty engineers.
[01:01:48]
Vlad A. Ionescu
Oh yeah, exactly.
[01:01:50]
Vlad A. Ionescu
And a lot of companies that are not engineering culture first, you're not necessarily seen as tech companies.
[01:01:59]
Vlad A. Ionescu
They happen to depend on tech.
[01:02:01]
Vlad A. Ionescu
Just because just about every company on earth nowadays depends on tech.
[01:02:04]
Vlad A. Ionescu
And having an engineering team in house, I feel like those are the companies that are gonna suffer more from this, you know.
[01:02:11]
Vlad A. Ionescu
Yeah, it's really hard.
[01:02:13]
Vlad A. Ionescu
Even in today's market where, you know, people are laying off engineers like crazy.
[01:02:19]
Vlad A. Ionescu
Even today it's still hard to get top engineering talent to kind of drive engineering culture properly in all these organizations.
[01:02:27]
Vlad A. Ionescu
And yeah, they're gonna be in trouble.
[01:02:31]
Vlad A. Ionescu
Hopefully, you know, it's.
[01:02:33]
Vlad A. Ionescu
We will manage as an industry.
[01:02:34]
Vlad A. Ionescu
But the writing is on the wall.
[01:02:38]
Viktor Petersson
In some way, 100%.
[01:02:40]
Viktor Petersson
And if you.
[01:02:40]
Viktor Petersson
Actually, for me, I think if you follow the money, you get very close to the truth here.
[01:02:46]
Viktor Petersson
Because the people that are most bullish and most vocal about this, like, oh, AI.
[01:02:50]
Viktor Petersson
All code written by AI in the next five years.
[01:02:54]
Viktor Petersson
Almost everybody who says makes those bold claims are either writing tools that are writing or in the AI ecosystem and directly benefit from it.
[01:03:04]
Viktor Petersson
Or there are people who want to basically slash your entire engineering budget and not hire engineers.
[01:03:11]
Vlad A. Ionescu
Right?
[01:03:11]
Vlad A. Ionescu
Yes, yeah, exactly.
[01:03:14]
Vlad A. Ionescu
One of these two visions is kind of crazy in the short term.
[01:03:18]
Viktor Petersson
Yeah, yeah, exactly.
[01:03:20]
Viktor Petersson
No, it's, it's going to be an interesting ride.
[01:03:22]
Viktor Petersson
I mean there is a scenario where the LMS and the tools will catch up and by the time this disaster of a tornado will hit the tools, we're good enough to fix themselves.
[01:03:36]
Viktor Petersson
And that's a possible scenario.
[01:03:38]
Viktor Petersson
I guess we can't rule that out.
[01:03:40]
Vlad A. Ionescu
Yes.
[01:03:41]
Viktor Petersson
And in case.
[01:03:42]
Viktor Petersson
Fantastic.
[01:03:42]
Viktor Petersson
And maybe that was the right bet then.
[01:03:45]
Vlad A. Ionescu
Yeah, exactly.
[01:03:47]
Vlad A. Ionescu
It's like that, yeah.
[01:03:49]
Vlad A. Ionescu
The curve of like, you know, intelligent people making one decision and like.
[01:03:54]
Viktor Petersson
Yeah, yeah.
[01:03:55]
Vlad A. Ionescu
The average.
[01:03:56]
Vlad A. Ionescu
It's like there's going to be a decision that makes sense for the most intelligent and the least intelligent ones and if that comes to become the truth, it's going to be interesting.
[01:04:08]
Vlad A. Ionescu
But yeah, for now I'd say from my perspective, I would be more careful than.
[01:04:16]
Vlad A. Ionescu
Yeah.
[01:04:18]
Vlad A. Ionescu
The immediate current state of affairs with AI feels like not ready yet.
[01:04:24]
Vlad A. Ionescu
You know, you can sort of create demos where you build from scratch with AI and sort of have mini games and such and they're maybe bug free mostly and they're safe, that's fine.
[01:04:37]
Vlad A. Ionescu
But managing something bigger than that, it's just not real yet.
[01:04:43]
Viktor Petersson
I would challenge a little bit because I think Cursor and the likes of are fantastic at doing busy work and just like refactoring busy work.
[01:04:53]
Viktor Petersson
Right.
[01:04:54]
Viktor Petersson
Stuff that would take you forever to do.
[01:04:55]
Viktor Petersson
But it's not super complicated.
[01:04:57]
Viktor Petersson
You can describe it easily.
[01:04:59]
Viktor Petersson
Where they struggle is when things gets too complicated.
[01:05:02]
Viktor Petersson
Right.
[01:05:03]
Viktor Petersson
Because the context window is just not big enough to.
[01:05:06]
Viktor Petersson
Yes, it's not there.
[01:05:08]
Viktor Petersson
Right.
[01:05:08]
Viktor Petersson
Which is why they just rather write new code than actually refactor the existing code.
[01:05:13]
Viktor Petersson
Because it's too expensive to read in all the context.
[01:05:16]
Vlad A. Ionescu
Yes, exactly.
[01:05:19]
Vlad A. Ionescu
Yeah.
[01:05:20]
Vlad A. Ionescu
They're just not going to follow all the requirements and maybe that will be surprising.
[01:05:24]
Vlad A. Ionescu
You have to have strong tests for all the requirements and all that still takes a lot of work.
[01:05:30]
Viktor Petersson
Yeah.
[01:05:32]
Viktor Petersson
And not the security is crazy.
[01:05:34]
Viktor Petersson
The amount of times I've had cursory.
[01:05:36]
Viktor Petersson
It's injecting and editing a lock file and managing dependencies.
[01:05:39]
Viktor Petersson
Like, whoa, hold on a second.
[01:05:41]
Viktor Petersson
Don't just change things.
[01:05:45]
Viktor Petersson
But yeah, like it's, it's a crazy world.
[01:05:48]
Viktor Petersson
But I mean, I'm still bullish on the fact that this is gonna be a net benefit and there's no, I mean, I'm 100 sure there's not benefit.
[01:05:55]
Viktor Petersson
Right.
[01:05:55]
Viktor Petersson
It's just, I think it's too naive and there are too many people that are too naive about what things can do today.
[01:06:02]
Viktor Petersson
It's just like, yeah, they will be able to do at some point, but we're not there yet.
[01:06:06]
Viktor Petersson
Today.
[01:06:06]
Viktor Petersson
It's a tool that allows good engineers to better but not non engineers to become engineers.
[01:06:15]
Vlad A. Ionescu
That's the other thing I'm wondering as well.
[01:06:18]
Vlad A. Ionescu
What does the next generation of engineers, what does that look like?
[01:06:22]
Vlad A. Ionescu
Right, because there's less need for the junior engineers.
[01:06:26]
Vlad A. Ionescu
Right?
[01:06:26]
Vlad A. Ionescu
The AIs can do a lot of that.
[01:06:29]
Vlad A. Ionescu
But now there's going to be maybe like a skills gap because those junior engineers will not become senior engineers.
[01:06:36]
Vlad A. Ionescu
Or maybe they're fine on the trajectory that maybe AI will get better faster than they would anyway.
[01:06:41]
Vlad A. Ionescu
So maybe there's less need for senior engineers.
[01:06:45]
Viktor Petersson
Well, I mean, I would argue most likely there's less in need for junior engineers.
[01:06:49]
Vlad A. Ionescu
Right?
[01:06:50]
Viktor Petersson
And I mean this is a fantastic point, right?
[01:06:52]
Viktor Petersson
And I've done a lot of hiring over the years and one of the things I have been surprised by, I mean, I'm getting old, but I'm hiring fresh grads these days.
[01:07:01]
Viktor Petersson
Like, I'm surprised by how many of them do not understand fundamentals anymore.
[01:07:07]
Vlad A. Ionescu
Oh yeah, yes, right, exactly.
[01:07:09]
Viktor Petersson
Like, because the CS education, okay, there are the good schools, like there's the Stanford of the world and all those, of course they have like world class engineering schools.
[01:07:20]
Viktor Petersson
But for the bulk volume of engineering schools, like I've had like people coming out of CS program, like you have to teach them how to use like git.
[01:07:29]
Viktor Petersson
And it's like, what the hell have you been doing for the last four years?
[01:07:34]
Vlad A. Ionescu
Exactly.
[01:07:36]
Vlad A. Ionescu
There's so many abstraction levels that make things so much easier for us, the engineers that we kind of forgot where we came from.
[01:07:44]
Vlad A. Ionescu
And now that's a knowledge gap that is gonna affect people, I think.
[01:07:49]
Viktor Petersson
Yeah, because I would imagine we're like roughly the same age and we grew up in, grew up using like Linux early on.
[01:07:58]
Viktor Petersson
Like you grew up learning a lot of those things by doing and Like I have no CS degree but I've schooled CS people a lot because just I've been exposed to it for so long.
[01:08:09]
Viktor Petersson
Right, you just learned lower level stuff by virtue of playing around with it.
[01:08:14]
Viktor Petersson
Right.
[01:08:15]
Viktor Petersson
And that's.
[01:08:15]
Viktor Petersson
Yeah, that knowledge gap is very real today and I think it's going to be even.
[01:08:21]
Viktor Petersson
It's going to be 100 start in like 5, 10 years.
[01:08:25]
Viktor Petersson
Right, with these level abstractions.
[01:08:27]
Vlad A. Ionescu
Yeah, yeah, exactly.
[01:08:29]
Vlad A. Ionescu
You're going to be building with REACT or like AI agents and so on and you have no idea how HTML works or HTTP.
[01:08:37]
Vlad A. Ionescu
So like yeah, today, yeah.
[01:08:42]
Viktor Petersson
Whilst today we're like annoyed that new hire newer graduates do not understand like the Linux kernel or whatever.
[01:08:52]
Viktor Petersson
Like that's going to be like move like five level abstraction further and they're like it's going to be so high level that the Linux kernel is like Cobalt to us essentially.
[01:09:04]
Vlad A. Ionescu
Right, yeah, exactly.
[01:09:05]
Vlad A. Ionescu
Yes, yes.
[01:09:07]
Vlad A. Ionescu
Because yeah, you could argue the previous generation are complaining that we don't know how punch cards work or whatever.
[01:09:13]
Viktor Petersson
Right, exactly.
[01:09:14]
Viktor Petersson
Yeah, yeah, it's an interest analogy.
[01:09:17]
Viktor Petersson
but this has been super interesting.
[01:09:21]
Viktor Petersson
I think we covered a lot of good ground here and I'm very happy about what we got covered.
[01:09:25]
Viktor Petersson
So thank you so much for coming on the show.
[01:09:28]
Viktor Petersson
Before we wrap, are there any closing notes, any shout outs you want to do people want to learn more about Earthly or so on?
[01:09:36]
Vlad A. Ionescu
Yeah, well, thank you for having me.
[01:09:39]
Vlad A. Ionescu
And, and yeah, if you're interested in improving your engineering excellence, your engineering standards or managing SDLC at scale in your organization, check out earthly.dev dev and yeah, hit me up.
[01:09:52]
Vlad A. Ionescu
I love to talk to you.
[01:09:54]
Vlad A. Ionescu
So many thanks Victor.
[01:09:56]
Viktor Petersson
Amazing.
[01:09:57]
Viktor Petersson
Perfect.
[01:09:57]
Viktor Petersson
Thanks for watching bud.
[01:09:58]
Viktor Petersson
Have a good one.
[01:09:59]
Viktor Petersson
Talk soon.
[01:10:00]
Vlad A. Ionescu
Cheers.
[01:10:03]
Viktor Petersson
All right.
[01:10:03]
Viktor Petersson
I think we covered a lot of ground though.
[01:10:04]
Viktor Petersson
That was, that was actually really good.
[01:10:05]
Viktor Petersson
I think it was a lot of good stuff there.
[01:10:09]
Viktor Petersson
This episode will go live not the next Friday, but the Friday after I think because I have another one that I need to pipeline before that is very timely but sounds good.