Having rich information about how an application is performing is highly valuable, and helps draw a better picture of the application performance, usage, and resource utilization. In many cases, we do not have enough data and might end up patching the application with incremental updates, adding more metrics each time.
Fortunately, there are a variety of free, open-source tools available that provide time-series databases and allow us to accrue and store all sorts of metrics, on our application, which very little effort.
While individual metrics at an instant in time may not mean much on their own, having access to metrics over an extended period of time allows for in-depth trend analysis.
In this post, we’ll review one way of gathering metrics from an Asp.Net Core application using Prometheus, and visualizing them with Grafana.
Let’s quickly review some basic definitions before we dive in.
What are Application Metrics?
Metrics are numeric measurements; The metrics are usually collected over time, as time-series, which means they change over time. The metrics differ from one use case to another, for example in a web application, they can measure request-response time, for a compute server they can measure CPU and memory utilization.
What is Prometheus?
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud.
https://prometheus.io/docs/introduction/overview/
Prometheus collects or scrapes metrics data from an application, via pull model, and stores them as time series data.
What is Grafana?
Grafana is open source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. In plain English, it provides you with tools to turn your time-series database (TSDB) data into beautiful graphs and visualizations
Getting started | Grafana Labs
Setup and implementation
Step 1: Docker preparations
We will be using Prometheus and Grafana as docker containers, to scrap and view the metrics data from the .NetCore application.
Let’s begin with creating a simple docker-compose file:
version: '3.7'
networks:
onprem: {}
services:
prometheus:
image: prom/prometheus:latest
ports:
- 9090:9090
networks:
- monitoringnet
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
volumes:
- ../volumns/prometheus.yml:/etc/prometheus/prometheus.yml:ro
grafana:
image: grafana/grafana
networks:
- monitoringnet
ports:
- 3000:3000
networks:
monitoringnet:
{}
We will update the default Prometheus.yml file with a new scraping job targeting our Asp.Net Core application:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["host.docker.internal:9090"]
- job_name: "netcoreapp"
static_configs:
- targets: ["host.docker.internal:5000"]
Note: I’ve mounted a volume to the Prometheus docker image so I can easily change the Prometheus.yml file contents.
We can now start the Docker containers and set up and data source in Grafana to be Prometheus.
docker compose -f netcore-metrics-promethues-net.yml upNow, we need to add Prometheus as a new data source. To do so, we need to navigate to localhost:3000, perform login to Grafana (default credentials should be “admin” for username and password). Then from the left navigation menu, select settings then data sources, and add Prometheus as a new data source:
This step is done, we can move on to the coding part.
Implementing metrics with Prometheus-net library
First, we need to create a new web API project:
dotnet new webapiThis created a new .Net Core WebApi project.
The next step is to add Prometheus-net library, which can be used as a NetCore middleware with a standalone kestrel metrics server:
dotnet add package prometheus-net.AspNetCore
dotnet restoreAfter we’re done creating the new project and adding the Prometheus-net package, we need to update the startup.cs file to enable the metrics.
To activate the core middleware exporter, we need to add the following line:
- After app.UseRouting() add app.UseHttpMetrics().
To activate HTTP request metrics, we need to add the following:
- Add endpoints.MapMetrics() after endpoint.MapControllers();
The code should be like this now:
app.UseRouting();
// Netcore monitoring addition
app.UseHttpMetrics();
app.UseAuthorization();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
endpoints.MapMetrics();
});
For the very basic setup, this should be enough. What did we get?
The application metrics are exposed via localhost:5000/metrics.
Adding custom metrics
The prometheus-net package comes with a set of predefined metrics, however, in most cases we will need metrics to measure application-specific usage and scenarios. First, let’s review what types of metrics does Prometheus support:
- Counter – A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.
- Gauge – It is a metric that represents a single numerical value that can arbitrarily go up and down.
- Histogram – A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
- Summary – Similar to a histogram, a summary samples observations (usually things like request durations and response sizes).
For more information on metric types, please refer to Prometheus’s official documentation.
Creating a Counter metric
In order to create a custom counter metric, we need to create a static member on the relevant class, and increment its value every time the action we want to measire is invoked:
private static readonly Counter LogicInvocationCounter = Metrics
.CreateCounter("logic_invocation", "Number of invoked jobs.");
Next, we increment the value of the counter whenever the desired logic is invoked:
[HttpGet]
public IEnumerable < WeatherForecast > Get() {
var rng = new Random();
LogicInvocationCounter.Inc();
return Enumerable.Range(1, 5).Select(index => new WeatherForecast {
Date = DateTime.Now.AddDays(index),
TemperatureC = rng.Next(-20, 55),
Summary = Summaries[rng.Next(Summaries.Length)]
})
.ToArray();
}
Custom metric with middleware integration
We can also create a custom metric reporter and connect it as middleware to counter method invocation. To do so, first we need to create a metric reporter class:
public class CounterMetricReport {
private readonly Counter _requestCounter;
public CounterMetricReporter() {
_requestCounter = Metrics.CreateCounter("total_requests", "The total number of incoming requests");
}
public void RegisterInvocation() {
_requestCounter.Inc();
}
}
Next, we need to create the middleware class:
public class MetricMiddleware {
private readonly RequestDelegate _request;
public MetricMiddleware(RequestDelegate request) {
_request = request ??
throw new ArgumentNullException(nameof(request));
}
public async Task Invoke(HttpContext httpContext, CounterMetricReporter reporter) {
try {
await _request.Invoke(httpContext);
} finally {
reporter.RegisterInvocation();
}
}
}
Now, let’s register the CounterMetricReporter as singleton service, then connect the MetricMiddleware:
public void ConfigureServices(IServiceCollection services) {
services.AddControllers();
services.AddSwaggerGen(c => {
c.SwaggerDoc("v1", new OpenApiInfo {
Title = "app1", Version = "v1"
});
});
services.AddSingleton<CounterMetricReporter>();
}
And we’re done. Now everytime a request reaches the server, the middleware will be triggered and the counter will increase.