Podcast
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Links
Follow Me
Follow Me
Join Viktor, a proud nerd and seasoned entrepreneur, whose academic journey at Santa Clara University in Silicon Valley sparked a career marked by innovation and foresight. From his college days, Viktor embarked on an entrepreneurial path, beginning with YippieMove, a groundbreaking email migration service, and continuing with a series of bootstrapped ventures.
Nerding out about Prometheus and Observability with Julius Volz
Play On
15 JAN • 2024
1 hour 5 mins
Share:
In this episode, I’m joined by Julius Volz, co-founder of Prometheus and founder of PromLabs, to explore the fascinating world of systems monitoring and observability. Julius’s journey from working on Borgmon at Google to co-creating Prometheus offers unique insights into how modern monitoring systems evolved.
We start with the technical foundations of Prometheus. What particularly caught my attention was Julius’s explanation of their dimensional data model and how it revolutionized metrics-based monitoring. His breakdown of common pitfalls, especially around metric design and “cardinality bombs,” provides invaluable guidance for anyone implementing Prometheus.
The conversation gets especially interesting when we dive into long-term data storage challenges. Julius shares practical insights about solutions like Cortex and Thanos, demonstrating how to handle large datasets effectively. His live demonstration of PromQL, showing functions like rate, irate, and increase, reveals the powerful querying capabilities that make Prometheus stand out.
I was particularly intrigued by our discussion of future trends in observability. Julius’s thoughts on eBPF integration, OpenTelemetry, and the OpenMetrics project show how the monitoring landscape continues to evolve. We also explore the simplicity of writing Prometheus exporters, highlighting how accessible the technology can be even for those with minimal coding experience.
If you’re interested in systems monitoring, observability, or infrastructure management, you’ll find plenty of practical insights here. Julius brings both deep technical knowledge and hands-on experience to the discussion, making complex monitoring concepts accessible while maintaining their technical depth.
Transcript
Show/Hide Transcript
Hello and welcome to this episode of Nerding out with Victor.
Today.
I got a very special guest with me today, Julius from Prometheus.
Maybe we should start with doing a quick intro to yourself, Julius, for people who are not familiar with who you are and about what Prometheus is in big picture.
Yeah.
So I'm Julius, I live in Berlin.
I am the co founder of the open source Prometheus monitoring system, but then also the founder and the sole person behind the company Promlabs.
So Prometheus is the open source, open governance monitoring system that is being developed and used by many people and there's many companies around it and Promlabs is just myself and it's just one of those companies.
Got it.
Perfect.
And I guess for those not familiar Prometheus.
Do you want to give kind of a sense of how widely used Prometheus is today and how it's being used in general?
Yeah, I mean Prometheus has pretty much become the de facto standard in metrics based systems monitoring in the open source world.
At least.
There's some closed and hosted competitors of course, but at least for the metrics based monitoring, it is pretty much the standard.
And yeah, I mean so you can find it really everywhere from small startups to really big banks, corporations, enterprises, even people run it at home to monitor their homes.
And yeah, you can use it in the classic data center use case to monitor your IT in a data center, but people also monitor hardware with it like sensors and chips and wind parks and all that kind of stuff.
It's super interesting.
Let's take a stroll down memory lane because I'm really curious about the early days of Prometheus and really dive into that and if I'm not mistaken started around 2012 or so back at SoundCloud.
Back in those days.
Do you kind of speak a bit more about what happened and what led to the invention of Prometheus and what kind of pain point you can solve with for that?
Yeah, exactly.
So this was 11 years ago, 2012.
So by now all of this is not as exciting maybe anymore, but back then it was.
So my job previous to SoundCloud was at Google as a site reliability engineer in one of the services.
And all the site reliability engineers at Google used a tool called Borgman to monitor their production services at Google, whether this was Google Search or Gmail or the service I was on, which was an internal backup service.
And this was a tool that was very similar to what Prometheus is now.
So the idea was to collect time series, store Them with a dimensional data model, so a metric name and then a set of labels attached to them so you can see in detail where something happened.
Which is great especially for dynamic cloud based systems which Google already had back then.
Now the thing is, after Google I went to SoundCloud and Matt Proud, another ex Googler, also went to SoundCloud at roughly the same time.
And were basically hired to try and make SoundCloud more stable and reliable and faster as platform or system engineers.
And we really found that the monitoring system world, especially in the open source world outside of Google, was severely lacking at that time.
Either the systems could only do alerting, but didn't really have any idea of history like time series, right?
Nagios didn't really have much of a data model to speak of or the alerting conditions you could create in there were very simplistic things and check scripts and so on.
And then you had systems like Graphite or opentsdb which either didn't have a dimensional data model and or didn't have a proper query language to do math between whole sets of numbers that are correlated in some ways and so on.
And then also efficiency of storage, UIs, all those things were kind of really lacking.
And so eventually we got to the point where we said, well, let's at least try in our free time, like on the weekends and after work at first to build something that is more similar to what were used to at Google.
And this became Prometheus.
We threw it up on GitHub under an Apache license from day zero, but we hadn't really told many people about it yet.
Then from there on, maybe two months in, we had the roughest of prototypes that could collect some data, store it and show it.
But then the first prototype is very easy and then comes the 99% of hard work.
So after that it was years of a convincing people that this makes sense to build our own monitoring system, then to actually build it until it was stable and scalable enough and wouldn't just oom all the time and all those things.
And then also to have at SoundCloud internally enough of a killer use case for it, which came and hadn't given that context which made Prometheus so important at SoundCloud specifically is that SoundCloud had very early on built a cluster scheduler like Kubernetes, much simpler of course, but before Docker, even before anyone else in the world was using containers, basically not everyone, but they built this using go version 0.9 something and like raw LXC containers and these hundreds of microservices running on that cluster would just be rescheduled on different hosts and different ports every time a new revision was rolled out.
And so that's what made it extra hard to monitor it.
With existing monitoring systems like Graphite for time series or Nagios for alerting when there was a latency spike, it was almost impossible to find out was it the entire service that is just getting slower or was there one specific instance contributing to this because we didn't have enough scalability to track per instance process level stats and very short lived time series and all that.
So yeah, the killer use case really started coming or I guess the first one of those was instrumenting this internal cluster scheduler system.
So every developer immediately now could see without doing anything, instance level stats of memory usage, CPU usage and so on.
Like all these per process, per container stats that are completely commonplace these days.
And they could key it by process, by revision by different labels that they cared about to track whether something was related to a revision or a specific host, et cetera.
Right, that was one thing that really helped it internally.
And then we needed a dashboard builder.
So I had built Prom Dash back then before Grafana was out.
So Grafana didn't really exist yet.
Prom Dash doesn't exist anymore.
So by now we're also all just using Grafana.
There's some new competitors on the horizon now.
And yeah, I mean at some point we reached an internal threshold of where people started, you know, adopting this more and more, finding it so useful that eventually there was an edict saying no new service without Prometheus Metrics.
And over the years we, yeah, we already kind of like told some other ex Googlers hey, we are writing a system similar to Borgmon.
Maybe you want to use it at your next company as well.
So we attracted like a handful of external users already.
But we only really fully published this in beginning of 2015 with a blog post from SoundCloud and another early user company.
And that's when things really exploded on Hacker News and everywhere else.
And you know, shortly later we joined the cncf and now the project is completely neutrally hosted in a foundation that belongs to the cncf.
So that's for people who don't know, that is the Cloud Native Computing foundation, which in turn belongs to the Linux Foundation.
And they were just setting that up originally just kind of Google with the lf.
They set it up to house Kubernetes neutrally.
And were the second project joining that and.
Yeah, so I guess that's the beginnings of it all.
And since then everything has grown a lot and evolved a lot.
Yeah, that's.
I mean, I think there are.
The interesting thing here is because it was such a important or the timing really played in the favor because I think it was just around that time where the whole ephemeral workload really started taking off and it sounded like obviously what you guys were doing predated kubernetes, which is kind of the default runtime these days.
But back then, traditional monitoring tools, they were really not built for ephemeral workloads.
Right.
I think that's.
That was.
Yeah, it sounds like that was the big killer feature that really gained adoption.
Right.
Yeah, especially maybe not ephemeral in the sense of seconds because Prometheus is not the greatest at that.
Because at least you need some thing that can then keep the state of whatever just ran but great for short lived time series or ephemeral workloads in the sense of things that maybe just run a day and then they get rescheduled with a new revision or on a new, you know, process or something.
So not just tracking one fixed set of identified time series over time, but really, you know, having a lot of churn in the identities of your time series over the day as developers roll out new revisions and scale down or up apps and so on.
Yeah.
So more ephemeral worker pools rather than ephemeral in the sense of like serverless, I guess.
Yeah, yeah.
You can also make that work, but that you need some extra bits for that.
Yeah, yeah.
So one thing I'm really curious about, obviously you have building and work with Prometheus now.
Well, since day zero by definition.
But what I'm really curious about is what is a common mistake you see in metrics?
I'm sure you've done you.
I've.
I think I've said some blog posts you've been through about this and what?
There's.
There is some Prometheus consultancy out there that have written quite a few blog posts.
But I'm curious about your vantage point of that.
Like what really are the most common mistakes people do when they're implementing Prometheus?
I guess in general.
Yeah.
And one of the companies writing blog posts about exactly this is myself.
So that is Promlabs.
Also if you go to promlabs.com and you go to the blog section, there's an article called Avoid these six mistakes when getting started with Prometheus and the very clickbaity title leads you to blog post mentioned.
I read the post some time ago.
Yeah, some of the common things that stump newbies and these are very common.
I think the really the most common one is cardinality bombs.
This is just when you discover this concept of dimensionality and labels for the first time, especially back then when it was new, people first were a little bit skeptical, what are these labels?
But then they said, oh, this is really useful.
Let's put everything into a label.
Like let's also split up all our time series by the exact user ID or an email or something of which there are an unbounded huge number of possible values.
And the main thing in Prometheus is that every set of labels generates one time series automatically.
The unique label set identifies a time series together with a metric name.
And so if you have one label in there that has a million different values, not only do you generate a million time series, but a million times whatever the other possible number of values are on the other labels on that same time series.
So it all multiplies together, the cardinality.
And your a big Prometheus server will maybe comfortably handle 10 or so million time series.
It really depends on how large you, how many resources you give it.
So several million time series are kind of normal for a big Prometheus server, maybe even more.
But that is your total budget.
And you want to kind of design your metrics in such a way that you stay under that total budget.
And how you exactly do that is kind of up to you.
Whether you have fewer processes to monitor because the process is also a label dimension, or whether you in the process split up a metric by fewer labels.
Or maybe there's ways to summarize the potential different values within one label into groups so you don't have to.
Instead of tracking a million different status codes, maybe you can group them into error success or something else.
Or something like this.
Right?
A very common thing where this happens is also when people want to track HTTP request statistics and they put an entire path in there, like hey, slash posts, user ID, post ID or whatever.
And these IDs, right?
They can get very high cardinality.
And there you could also say, hey, before I track this, I will actually replace these high cardinality bits in my path with something that is just a placeholder.
So now I'm just generating one label value for that particular pattern of labels.
So that's one.
I'm just looking here for others.
The others are even a little bit more technical, I guess, that are in my blog post.
So I Wouldn't go into them too deeply, but I think this whole metrics design is one of the biggest ones that everyone gets hit with initially.
Yeah, I can definitely relate to that.
When we first started adopting Prometheus, we tagged and over tagged everything and it became, like you correctly pointed out, kind of unusable after a while.
So less is more, I guess in some ways and start expanding from there.
Speaking of metrics in general, so those are some common mistakes with Prometheus.
If you're putting your sre hat back on for a moment, what do you consider kind of the blueprint or the best practice for server monitors in general?
If you're looking away from the application stack but more metric that you care about when you do observability and how you do go about to do that and set up those dashboards.
So how do you think about that?
If you think from that angle, you.
Mean like monitoring servers.
So assuming we are.
Yes.
Like the health of the server in general.
Right.
And so what are the metrics that you care about in terms of monitoring a server in general really?
If you're thinking from like best practices for somebody who hasn't had proper monitoring in place, perhaps, and setting that up from day zero, presume that would involve node exporter.
Perhaps.
But I'm just curious, what do you think about that in general, how you see that Monitoring in general.
Right.
So yeah, when it comes to like monitoring hosts.
Right, that's what you mean.
Yes.
There's often not that much you actually want to alert on.
So it depends what your environment is, of course, and what your philosophy is.
But often for hosts at least there's.
The node exporters is one of those agents that the Prometheus project officially offers that exposes all kinds of metrics about the host it is running on to the Prometheus monitoring system.
So you can collect CPU memory usage and network usage and everything you would expect these days if you're running it on Linux, it gets it from the PROC file system, from the sys file system, like all these different kernel interfaces and then just bridges it over to Prometheus metrics.
And so the first step is just run the node exporter really easy.
It's easy to deploy on every host you have and then just monitor it so you have the metrics.
You can configure it in different ways to for example, exclude some pseudo file systems or Docker container volumes and stuff like that.
A lot of those exclusion flags are set pretty reasonably by default, so often you can just start it without Any flags and it'll work fine.
And yeah, then there's just like a few things you might want to alert on.
Like, you know, back then it was pretty common to alert on stuff like load average being high or certain resource usage being high.
But the problem is that sometimes that doesn't really point to a real problem and might actually generate too noisy alerts that don't immediately affect the service in a bad way.
And so alerting I would usually more approach from the kind of the SLA perspective of what does the user of whatever service you offer expect or what kind of agreement do you have with them or what service level objective SLO did you make for yourself?
At least this is the latency we want to have like 90th percentile latency should always below 100 milliseconds.
We should have a 99.9% uptime over given months and so on.
And then have alerts more based on these user visible metrics rather than on all possible underlying causal metrics.
You still want to collect all those underlying causal host metrics to be able to figure out when you do get alerted what might be the reason, what might be the underlying cause, are there any weird things in your graphs and so on.
But host wise there's not a lot of things you wouldn't alert directly on.
But of course there's some stuff like disk usage at least if it's completely full or you can predict that it is already very, it's maybe at 80% full and you're filling up quite fast and you can use a linear prediction in the Prometheus query language promql that it will be full in one day.
If it develops further like this, then it's still a good idea to alert on these imminent dangers to your actual service.
But other than that I would be very conservative with actual host level alerts.
And again it depends a lot on your environment of course, because also some environments will be completely happy if one host completely dies, whereas others will just, you know, completely have an issue if you don't have that redundancy.
Right.
Yeah, I mean I like the vantage point for SLO or SLA is of what is customer impacting really.
I think that's a good, I think that's a good view of how what you should alert.
I think that's a good guiding principle.
You kind of alluded to alerting, but do you want to speak a bit about Alert Manager as well?
Because that's obviously an important corner piece in Prometheus at large.
Yeah, so the idea how Prometheus alerting works in general is that first Prometheus collects all these dimensional metrics, and then anything useful you want to do with the collected data, you probably will use the Prometheus query language, PromQL.
So this is like a dimensional, very useful functional query language for processing your time series data and generating some answers based on the selected data, whether you want to aggregate or select or do mathematical correlations, and so on.
And you can use this query language both for dashboarding, ad hoc debugging, and automation, but also for alerting.
So alerting uses the same PromQL query language.
So this is not the alert manager yet that still happens in Prometheus.
The alerting rules are actually configured as part of the Prometheus server, and the Prometheus server on a regular basis runs the contained PromQL expression in an alerting rule.
And if there is any output, any time series being returned from this PromQL expression in the rule, those outputs will become alerts, subject to some other thresholds that are configured in the rule, and so on.
And then once they do become alerts, they get sent to this separate server component that is called the Alert Manager.
And so the alert manager just receives fully baked, hey, here's a problem style alerts from various Prometheus servers in your infrastructure, and then it routes them based on their labels to some.
So basically the alert manager conflict is like a big routing tree, and you end up in some node of that routing tree based on what labels your alert had.
And you might root on stuff like the team or the service or the severity level of the alert, right?
And then in that routing node that you actually reach, you can configure all kinds of things like do I want to send this to slack or to PagerDuty, for example, to which teams Slack, and so on.
You can say, hey, throttle this in a certain way over time, group this by certain labels.
So, for example, hey, don't send me one alert for every host that is down, but send me one alert that contains all the hosts that are down.
So you can kind of bake many notifications into one with more detail so you don't get flooded.
And yeah, there's a bunch more settings that are possible in the alert manager, but basically this the component where all your Prometheus servers send their raw alerts to, and then it routes and throttles and aggregates and finally dispatches your final alert notifications to a human.
Okay, that's perfect.
One thing that I think people that start using Prometheus eventually run into is long term storage because obviously Prometheus is not designed for long term storage, it's designed for short term storage where I think Its default is 14 days I believe it is, if not mistaken and then you can set it to higher.
But I think it would probably peak out about a month or so on most hosts before it becomes useless.
Kind of what I mean, what's your view on that?
There have been quite a lot of development with Cortex is one of them and quite a few others.
How do you see that evolving and what are you thinking around that?
How do you recommend people looking?
Good point.
So Prometheus comes with its own built in TSTB time series database storing files on local disk.
And I would have to say for many use cases that are not huge, this actually works pretty well even as a long term storage.
So there are people like me who have years of data in a single Prometheus database.
But of course if you do collect a lot of data about millions of different time series every couple of seconds, eventually you will hit the scalability bottleneck of one disk and this local TSDB implementation.
And you may also hit depending on your requirements, robustness or reliability issues, when that one disk dies or you lose the data or something, there's ways to back it up so you can still make it work.
So first of all I wanted to say actually you can use the normal Prometheus TSDB quite well for certain long term storage scenarios, but eventually of course you will outgr what a single node can do.
And there's different ways for going around that.
So one is, hey, let's configure the Prometheus server to send all the data or subset off that it collects to some remote storage system.
So it doesn't only store it locally, but forwards it to some remote endpoint.
You can even now if you want to, you could even turn off the local storage completely so it only forwards.
And then that remote endpoint needs to implement a protocol that we call remote write.
This is a protocol that the Prometheus monitoring system standardized.
And by now there are so many different integrations for that protocol.
Like all the major cloud providers accepted.
If you go to like, you know, Amazon, Google and all these, you can usually send Prometheus remote right there or also Chronosphere, Grafana cloud, all these players and they have their own proprietary databases for how they can then scale and store that data in different ways and give you extra features on top and give you a global view over all the data from Your different Prometheus servers.
So this is one option and there are also open source alternatives to this one is Cortex.
And yeah, this also came from a long overlap with Prometheus maintainers.
I was one of the original Cortex creators as well, along with Tom Wilkie.
And this is more of a clustered system that you can run locally, but it is quite complex.
So it's like mainly for people who want to offer Prometheus data storage as a service in their big organization, as a central point where all the different teams can send their Prometheus data and then it will offer you a PromQL compatible API that you can use with Grafana and other tools, even Alert Manager and so on.
So that is really a horizontally scalable multi tenant tool.
It is a bit more complex to deploy.
So most people who do want to have some kind of long term storage on premise tend to use a thing that works a little bit differently called Thanos.
And Thanos works in this way, at least the way it was originally envisioned.
Now it supports different modes as well.
Is you run still all your different Prometheus servers in your infrastructure, monitoring different services and regions and data centers and so on.
But now you want to add both a global view over them and long term storage and then also some ha deuplication and so on.
This is what Thanos can do.
So you still have your Prometheus servers, but now you run little sidecars next to each of them and they all integrate together with the Thanos Querier into this kind of mesh where you can in a federated way query over all of them at once, give them a single PromQL query and the query knows which Prometheus servers it actually needs to hit based on some labels in your query.
So it doesn't need to always go to all of them.
And these sidecars running next to each Prometheus server can also ship all the finished TSDB blocks.
So everything that is a bit older than like 2 or 3 hours to an object storage like either S3 or minio or GCS and others.
And then the Querier, the Thanos Querier can integrate that long term older data as well.
And so there it's really safely backed up right in any of those object storages, you can easily replicate it and all that.
So yeah, you get like pretty cheap very long term storage, including some down sampling features and so on.
So most people who want to scale Prometheus internally, they just use Thanos.
And then others who really want to invest in a dedicated team running such a cluster for other teams would use Cortex.
There's also Mimir, which is a Grafana fork of Cortex now.
And then others again, they would choose some cloud services to do that.
Yeah, no, we've been using tunnels at screen enough for a while and we use it both for metrics on our clusters infrastructure, but also on actual devices.
So pushing metrics from actual devices.
But one of the things I find great and this kind of beautiful simplicity of Prometheus is really the output.
Because if you look at how a slash metrics feed looks like, it is so simple and kind of that's what's beautiful about it in many ways.
Because writing a Prometheus exporter is very trivial.
I mean, you can literally write one in Bash versus many other monitoring frameworks that I worked with in the past.
And I guess that combined with the modularity of the remote write, which is kind of recursive almost because you use the same way to push to it as you can do to push further out in the federation.
I find that as a beautiful implementation.
So so far, what is the most interesting use case you've seen of Prometheus or out of the box use case of Prometheus?
You spoke a bit about monitoring your home, but I'm curious, like what's the most crazy use case you've seen so far?
Yeah, the ones that I always find the coolest or most interesting is when I hear someone talk somewhere or tell me about physical use cases like we had.
So maybe I'll just mention like three or four different examples.
One was a prom con.
So we also have a Prometheus conference, a promcon.
Talk about someone monitoring a big wind power park with Prometheus.
So he could even have the rotational angle of all the different, what do you call them, windmills or whatever they are.
Yeah, as a metric.
So like the angle as a metric, the wattage as a metric and everything.
So that was cool.
I heard from a big container shipping company who shall remain unnamed, that they have Prometheus running on all of their vessels and reports home telemetry about the ship, about where it is and what it's doing.
That I found really cool.
One guy from Accenture once gave a meetup talk about the train system in Germany, the Deutsche Bahn, where you have all these displays, signage systems telling you when the next train is coming and which one it is.
There were almost 100,000 or so of those in Germany in the different train platforms.
And they had basically installed the node exporter or something akin to it in each of these.
And we're monitoring it remotely using Prometheus.
So that's kind of cool because then you're on a train platform somewhere and you're like, oh cool.
That's like where basically my software is running in and you can show that.
So I really like those because, you know, all the data center use cases, they're cool.
Of course that's what Prometheus was made for.
But they're kind of the usual thing.
Maybe one tiny use case that someone also reported is they had mold in their bathroom at home and now basically they wanted to monitor the conditions to not get mold again.
So they just installed Prometheus on like a little Raspberry PI or something at home and monitor temperature, humidity and air pressure, I think.
And then you can like calculate in promql whether the dew point is reached and then send an alert in case that happens.
And then you would open the windows or something.
Right.
And so it really scales down to those really tiny use cases as well.
That's really cool.
So we covered quite a lot about PromQL, but maybe it warrants a bit of a deeper intro to what PromQL is as it's such an important building block in Prometheus.
And I'm not sure you want to do some speaking about the background of it and like the history of it.
My understanding is it really derived from kind of reverse engineer something that Google had in sense.
But maybe you want to speak a bit about like the design implementation and perhaps do a quick demo of how it works if that's.
Yeah, could do that.
So prompt.
Yeah, it's the one unifying query language in the Prometheus ecosystem that you know, allows you to do dashboarding, alerting, debugging, other use cases as well.
Automation for example.
And it is a non SQL style query language where you really just write a functional expression that some function that might take other parameters that are in turn.
Again an expression that could take other parameters.
So it could form like an arbitrarily deep tree.
But eventually if you evaluate that tree, you'll get some kind of result so you can select some data.
Then you might.
If it's counters, then you could take the rate of increase around it and then maybe you would want to sum up all the individual rates but keep certain dimensions in the sum.
So we still group by the path but get rid of all the other.
The instance level detail.
For example, you might have a whole set of error rates divided by a whole other set of error rates by a whole set of total rates correlated on identical label Sets so you can get a whole list of error rate ratios out and then alert on that.
You can filter.
You have a bunch of different functions basically to compute different things.
Very geared around systems monitoring.
So not like deep machine learning statistics or so, but really quick functions that behave pretty simply to give you right now answers about your infrastructure and whether you should alert someone, for example.
And yeah, I mean, if you like, I could try sharing my screen and just showing you what the language looks like.
Let's do it.
Let's try that.
Okay, so I have this.
Let's see.
I have.
Should I share?
Yeah, I'll share the entire screen here.
Of course it only shows me.
Okay, I will do the entire screen because otherwise it does not show me the windows that I have on a different virtual desktop.
So let me know once you can see stuff.
I can see it.
I'll just maybe zoom in a little bit more even here.
So this is Promlens is one of the possible interfaces for dealing with PromQL.
Basically what you have here is you have an expression input box and you can, you know, first of all you can select some data, of course.
Now, okay, let me reload that.
So for example, some HTTP level counter metrics counting for three different demo service instances, which process they come from, which group of processes.
This is the job.
The method for which an HTTP request was handled, the path on which it happened, the status code that was the result of the request.
So more for the response status code, I guess.
And these are raw counter values.
So if you just graph them like this, they don't look too useful because these only go up basically starting from whenever this demo service was started.
So they start at zero and then they just go up.
So what you want to know is you want to see like how fast do they go up.
And I could add a rate function around this here.
Not going too much in detail now about how this works in detail.
But then you will see at every point in the graph averaged over a 5 minutes window.
What's the actual per second request rate for the different dimensional combinations Here you can see if I hover over this.
This is method get path API bar status 200 and you get that for the different other ones as well.
If we wanted to now we could sum this and only keep the path dimension.
For example now we would have it only for every path instead of all the full details.
This is the summed up rates, aggregated rates.
I could still break out the method if I wanted to.
And since I'm using PROM lens, it's showing me my textual query as a kind of laid out tree here as well, where I can see how many results I have in each of the sub nodes of my query.
Right?
This is like a full, could be a first class expression by itself.
This demo API request duration, seconds count, metric name and then this one, the rate call also produces 27 results.
But those 27 results get shrunken down by the aggregation to only 5 results because we only have 5 path and method combinations that we end up with here, down here.
Now one further thing we could do is we could say, hey, let's only select the ones that ended with a 500 so errors, right?
This would be like error rates by path and method combination.
And now we could correlate those error rates for each path and method, which we can also see here as a table as current values.
We could correlate those to the total rates we get for the same dimensional combinations of method and path.
So now I could say, hey, divide this entire thing by basically the same thing, except here I'm, you know, in below the division, I'm not going to select only the arrows, I'm going to select all the requests to arrive at a total sum.
And I can indent this a little bit better here.
And yeah, now what happens is that this binary operator automatically does a join on identical label sets between the first and second operand.
We can also show that here how the matching is happening.
We see that sum will not actually find label combinations that have 500 errors.
So those will not produce an output.
And yeah, by default that's what happens.
You can customize these binary operations to match on a subset of labels to allow many to one or one to many.
Matching won't do that now.
So what we're getting now is basically ratios of how many errors there are.
I could multiply those by 100 to get percentages.
So we have 0.3% of errors in this first combination, for example, 0.7 in the second.
And now I could extend this to some kind of alerting rule where I say, hey, give me all the ones that are larger than 0.5%.
And so I just filtered this entire list of output time series down to the ones that have a sample value of larger than 0.5%.
And now I would get exactly the problematic label combination.
This is just one example of what you can do with PromQL.
There's histogram stuff in there's other stuff, but it's really just working with this label based data model and yeah, if you want to learn more about PromQL and all the different constructs you can write, I also created a cheat [email protected] promql cheat sheet.
You can find it also here under the resources.
And here I try to just list all the major patterns that people usually Write when doing PromQL.
So this won't be everything, but you can open each of these patterns in PROM Lens by just clicking on a button here and then you have some demo data to actually work with and play with expression and change it and see what it does.
You can also let the different nodes get explained to you here what they actually do.
You get the explanation down here.
So yeah, that's just like a super duper brief intro to what PromQL looks like.
And yeah, importantly and I'll stop sharing here, it's only used for reading data.
So if you want to write data into Prometheus or delete some data that happens over totally different paths.
Like Prometheus writing into the TSDB after his it has collected some data or if data has become too old it will become deleted depending on your retention policies.
But PromQL itself only always gives you some results.
It's a read only language and really flexible at giving you precise answers and conditions.
So one thing, I mean just very simple and question I guess around promql I guess maybe you can just explain it quickly like when you look at demo examples of Prometheus particular in the Stack Overflow forums and so on.
The most common functions is rate and irate.
Do you want to just help explain the difference between the two just quickly for.
Yeah, maybe people like myself, maybe I.
Should also if you go to my YouTube channel, I think it's so PromLabs is the channel name on YouTube.
I have a whole video about the three types of different rates and also the increase function that explains the exact differences.
Basically there are three different functions for telling how fast the counter is going up and those are rate, irate and increase.
The rate function gives you.
Maybe I should share my screen again because it's so graphically demonstratable.
That totally makes sense.
So I'll just do that again.
So let me know again once you see stuff these other ones here.
So now I'm just in a vanilla Prometheus server expression entry interface.
So let's say I do the rate over these request rate metrics that I saw that I used earlier.
What we're doing now is at every resolution step in the graph we are basically selecting five minutes of past data.
So if we're producing like this output point in the graph, we're looking five minutes backwards.
We're taking all the raw data, the only increasing counters, and we are kind of calculating an average of how fast per second is the counter or each of these counters increasing under that window.
And that per second increase value becomes the output point for this step in the graph.
And it can make these resolution steps a little bit more visible here.
If I set the resolution to five minutes, then we get a lot of explicitly visible resolution steps along the graph.
So you can really imagine that every one of these steps we look back this amount of time that I specify the expression.
And yeah, in this case this would be an entire step because this is five minutes and this is five minutes.
And it just summarizes that entire steps samples into an average per second value.
Now what happens if I make this window a little bit smaller, but also I will go back to the original resolution and start with five minutes.
So five minutes, I'm smoothing over five minutes so it will be relatively smooth.
And if I make this averaging window smaller, like one minute, I get more spiky rates because now I'm averaging over fewer samples.
Same If I do 30 seconds, then it gets like really spiky.
And I only scrape data every 15 seconds in this Prometheus server.
So eventually I even have to be careful that I don't make the window too small, because otherwise I will not always at every point be lucky enough to actually span two samples with my averaging window.
Right.
If they're 15 seconds apart, it could be that there are not two of them under a 20 seconds window.
This gets exacerbated if I do a 16 seconds window.
Then only sometimes are they actually going to be two samples under the rate window to be able to even compute a rate.
And the graph will mostly be cappy.
So you do have to choose your window large enough to at least robustly select two samples.
But what if you wanted to say something like, hey, I want my graph to behave as spiky as possible.
Like react as much as possible to the most two recent data points while still not having to care too much about how small?
Exactly.
I can choose this rate window.
So that's the irate function, or also called instant rate function.
It also gives you a per second rate of increase.
But under this provided window here, it will only ever choose the latest two samples and see how much per second does the counter increase between them.
So it doesn't matter anymore how large or small you make this here you will always get the same result providing I anchor it to a constant time here.
So one hour versus one minute it will look the same as long as you make this window large enough to always cover two data points.
Again, if I make it too small, you will run into issues again.
But that's basically what the irate function is about.
Normally I would not recommend using it unless you have a specific reason because it will skip over most of the data in your range interval here you might have five minutes worth of data in there and but now you're only looking at the latest two data points in there to give you some answer and they might not really be representative.
And so usually people use rate.
It's good to average or smooth over some amount of time.
Not too long maybe.
But if you are in a super zoomed in graph like this and you want to see really the latest developments, then maybe irete can make sense at times, right?
But yeah, use it sparingly.
And then a last function called increase is basically identical to rate except that it does not convert the output unit to per second.
So the shape of the graph, if I go back to the one hour graph, let's say, or let's go to two hours, the shape will look identical with rate and increase here.
The only difference is the Y axis.
So here we get a per second Y axis and if I go to increase with the same window now we get increases not per second but per one minute because I have a one minute window here.
Or if I do 15m, then we would have increases per 15 minutes.
And so you see again, but this kind of intermingles two aspects.
We intermingle the smoothing window size so how smooth the graph will look and the output unit.
So if I change this window, I will both change the smoothing and what unit I actually output, which at times might be actually what you want, right?
What's the total increase over one day or so?
Then you don't have to multiply it in the end with the number of seconds in a day, but at other times that's not actually what you want.
So most of the time by default the rate function is actually great.
It gives you a predictable per second output unit and that's great if you then want to combine it and divide it and so on with other constructs in your query.
If you keep everything at these base units, everything will stay more predictable and yeah, so that's why most of the time you will see just the rate function.
Got it.
Okay, that's really cool.
Any other promql questions?
Or answer the screen.
Yeah, I think rate, I would presume is your starting point for a lot of metrics where when you're starting out, I guess that is the function that you start using in most common.
Most common, yeah.
Rate and then summing of rates I would say is one of the most common things you will see.
Yeah, cool.
No, I think that's a good.
That's a good crash course and quick introduction.
Nice.
Cool.
Let's turn our eyes to the future a little bit and I'm curious about a few things to pick your brains on.
The first one is EBPF is all the rave these days in observability.
Is there any play for Prometheus to tap into EBPF for, I guess collecting metrics with less resource overhead?
Or how do you see that change in kind of observability in Prometheus?
Yeah, so myself, I'm not a big EBPF expert, of course it's awesome kernel level interface to allow you to tap into pretty much any place in the kernel and extract information or do stuff, whatever you want in there.
There are EPP exporters for Prometheus, so if you just Google EBPF exporter, you will find one by Cloudflare for example.
And yeah, you can configure it in different ways to give you all kinds of host metrics.
And I would actually also have to read the detailed documentation about what exactly it can do.
But yeah, basically it will enable you to collect metrics about all kinds of things you can, you know, instrument with ebpf.
You don't anticipate a world where an EBPF based node exporter would replace the regular GO based node exporter in the future?
It's a good question.
I mean, at the moment the question is what would that give us if, you know, if we could get at stats that are not exposed using the current interfaces which are fine, like the PROC file system sys file system and certain syscalls.
It could be that there are certain stats in the Linux kernel that we would also want to expose that would be possible in that way.
Let me just see if anyone has already added anything EBPF wise to the node exporter.
You never know because there's so many different Prometheus components these days.
Right?
Ebp, I don't find anything EBPF at least module wise in the node exporter yet.
But of course it makes sense because it's also node level metrics that you might at some point want to put something like that directly into the Node exporter and until that time run the separate EBPF exporter.
But yeah, I could guess what are the upsides either being able to get at metrics that you don't otherwise have, being able to get at them without writing as much custom parsing code because it's a bit annoying to parse those proc file system files sometimes and also maybe to get something more efficiently.
But so far it has not happened in the node exporter, but totally for it, if it makes sense.
I'm just thinking like you could increase the scraping period, for instance, and to get metrics every second instead of every 15 seconds, perhaps without taking the.
I mean usually that's like most of the time the scraping frequency is not really bottlenecked on the producer of the metrics and more by the Prometheus server being able to ingest a lot of data.
It's fairly good at that, but that's still usually the bottleneck.
But yeah, there's some modules in the node exporter that take a little bit more time to produce data, which are some of them I think also switched off by default.
So it could be that there's.
I'm not sure if there's something in there that could be made like way more efficient by ebpf because already the native kernel interface that exposes it, I hope most of those will already be relatively efficient.
Fair enough.
Open telemetry seems to be kind of taken big, well taken.
Observability kind of at a pretty high pace right now.
It's gaining a lot of momentum.
Obviously Prometheus is kind of a building block in that.
How do you see open telemetry kind of as the observability stack of the future for cloud, native workloads or how do you envision that?
Yeah, I mean, so open telemetry came from a little bit of a different corner originally.
Very tracing focused with open tracing and open senses and then merging into open telemetry.
So that was their.
The prime kind of initial use case that was most stable.
And then there were logs and metrics, with metrics now also being more stable and people adopting it.
And they focus a lot on the instrumentation API aspect, like standardizing the instrumentation API.
What should it look like to instrument your application and what metadata should there be in certain interfaces for metrics and so on?
And in a lot of cases those are similar to what we do in Prometheus, but in some other cases they're a bit different.
They have slightly different metric types, they have slightly different allowed character sets, especially in label Names, they can be arbitrary UTF 8 characters if I'm not mistaken.
Or at least we want to allow that now in Prometheus as a result of, you know, wanting more open telemetry compatibility.
And they also have different standardizations of what labels will you want to always collect about where a given metric came from, the resource label.
And in Prometheus we have something similar called target labels that Prometheus attaches to wherever it pulls metrics from.
But they're slightly different and they're more of an admin configurable stuff and less standardized.
And so now there's efforts in the Prometheus team wanting to of course still be relevant in this world.
And people want to use Prometheus as an otL metric store.
Prometheus will still only do the metrics part and not logging or tracing, but at least for the metrics part.
If you're using oatmeal telemetry to instrument your apps, then we want to be able to store OTAL metrics in a more native and better way.
You can already send in an experimental way OTAL metrics to Prometheus.
There's an experimental OTAL receiver in there, or there's different ways of making it work.
You can either send the OTLP stuff directly to Prometheus using the new experimental interface, or you could bridge it earlier on into a Prometheus native remote write format and send that to Prometheus.
But we want to in Prometheus extend the data model somehow to allow more characters so we are able to store more of the data model that OTAL has.
For metrics, we're looking at ways of supporting what they call, what's it called, a delta, not delta cardinality.
When you count like either when you send absolute counters that only increase over all of time, or you only send deltas from one push to the next.
And in a push based monitoring system that's easier because your process, it counts a lot of things from one push to the next and then it's, it resets the counter to zero and so it will only like, at every push it will send the delta that has happened between two pushes.
Right?
And the problem with a pull based monitoring system like Prometheus is that pools are supposed to be idempotent and any service process that is being monitored by Prometheus could be monitored by either 0, Promethei or like by many Prometheus servers at the same time.
And it shouldn't then any scrape from a Prometheus server shouldn't Just reset whatever the current count is.
So we have to add an additional little translation layer in there to potentially transform these delta temporality, I think counters into absolute counters again or support them natively or something like this.
So this is still being hashed out and there's a couple of other things around mapping these resource and target labels.
Like do we always take this huge amount of hotel labels and just put it into Prometheus metric?
That would get super messy.
Or do we allow users to map like configure which ones to map over?
And how do we always just put it into a separate metric that you then have to join in?
That's also annoying.
So there's definitely some little incompatibilities that will make, you know, I would still recommend people like use native Prometheus instrumentation.
It's going to be way less code and it's going to work faster and more simple.
But if you do want to use otel, like if you just are interested in the metrics use case and you still want to use otel, then we are still going to work on making that better.
Okay, makes sense.
Yeah, I guess that's fundamentally a different thing between push and pull.
I mean that was at least in our use case of screenly, that was a bit of an inconvenient thing.
When you work with edge devices, for instance, you can't scrape them because obviously they're not in your data center.
So the push ball there makes a lot more sense for our use case.
But I know Prometheus heavily favors pull versus push.
Yep.
For different reasons.
But yeah, absolutely, absolutely.
But I'm saying that might push that push use case a bit further, which we would welcome.
Makes sense for anyone with like super highly segmented networks or you know, end devices at customers and so on, where, you know, polling is really great if you can reach everything easily in your own data center and so on.
But yeah, it gets harder as you break things up.
Absolutely, absolutely.
There's another project that I know you've been a little bit touching upon as well.
It's OpenMetrics.
Do you want to speak a bit about that and how that kind of.
There's a lot of open here, but.
How that relates to the whole openmetrics was an idea of actually taking what we had as Prometheus metrics text transfer protocol and just standardizing it more with.
With some changes requested by other parties and stakeholders.
So this was an initiative started by Richard Hartman of the Prometheus project and he got others together from other companies and said, hey, let's do an RFC Internet standard.
And it made it pretty far and it's also supported by Prometheus, but by now basically it was decided by the Prometheus team and the Open Metrics team.
Like hey, I think we still want to do some things differently and let's just actually merge Open Metrics back into Prometheus again and not have it as a separate project anymore.
So it's just going to be archived in the cncf and it's still, I think explored a lot of the needs of, you know, if we want to standardize a metrics transfer protocol or evolve Prometheuses further, what are needs that other parties actually want in such a thing.
And so it did help, but it will no longer exist to my knowledge, like as a separate project.
Okay, but it sounds like you made headways in terms of that you can now push Prometheus style metrics to most cloud vendors, which I guess it sounds like that kind of paved the way to that.
Yeah, that's a different thing because like the Open Metrics system is so you have the Prometheus server and you have your monitor target.
That's what happens between those two.
And then the protocol that we speak to the outside world, that's the remote write protocol.
So that's a totally different one that is independent of the scrape format.
Right.
It has slightly different needs as well because you will want to be able to send multiple samples from multiple scrapes for many more different time series over this remote write protocol.
Whereas the pull based open Metrics or the previous text based format on which it is based, they're really for going to a single monitored process or target and pulling its current metric state.
Just the one number for each metric at this point in time.
Okay, that's perfect.
I think we're up on time.
But it's been very informational and helpful for me and I think that's really good.
And do you want to do a quick shout up if somebody needs consulting service around Prometheus where they can get in touch with you and how they can.
Yeah, how they can.
Oh yeah.
Take their Prometheus game to the next level?
I'd love to.
So I mean these days I mostly do training around Prometheus so no longer just, you know, per hour consulting.
I have two main things I do besides other partnerships and still open source development.
So those are live trainings around the PromQL query language where I really explain over a multi hour session kind of with a setup like this here right now and screen sharing and material and everything, how PromQL works, how you can use it, all the different pitfalls with exercises and so on.
So I do those, you know, quite often with teams in companies.
And the other thing is, so you can find [email protected] and you go to, I think at the very top there's a live trainings link and I also have self paced courses, another link on the side and that leads you to training.promlabs.com and those are training modules.
So courses that you can either buy as an individual or you can buy for your entire engineering team as a company to learn all the basics of Prometheus, like from what is Prometheus to all the advanced PromQL stuff and alerting and dashboarding and node exporter metrics and remote storage and all that.
So it's textual and images and screenshots and quizzes and interactive stuff where you set up all these components and string them together and build setups.
So it's not video content.
But that makes it actually possible for me to keep up to date all the time, which would be really hard with videos.
So yeah, individuals buy them and companies do as well.
And of course I can recommend them.
Perfect.
Thank you so much for your time today, Julius.
Much pleasure having you in the show and I talk to you soon.
Alrighty.
Thank you so much.
See you.
Thank you.
Bye.
Found an error or typo? File PR against this file or the transcript.