Create a custom django prometheus metric

Share this content
  •  
  •  
  • 39
  •  
  •  
  •  
  •  
  •  
    39
    Shares

Today I’ll show you how to create your custom prometheus metric for your django application. Prometheus is an awesome tool to monitor our stack, specially with its integration with grafana and its cool dashboards.

The example metric is for a recent use case. I wanted to measure my website visitors favourite pages, and where they came from.

I wanted to achieve this without tracking, cookies, paid services or having to use Google Analytics. Since I was already using a Prometheus/ Grafana suite to have visibility of my platform health, I decided to use it for this goal as well.

This was the end result, on grafana:

A custom django prometheus metric on grafana

First step: having django-prometheus up and running!

Just follow the instructions on the project readme, they should be just:

pip install django-prometheus

Then, on settings.py

INSTALLED_APPS = [
   ...
   'django_prometheus',
   ...
]

MIDDLEWARE = [
    'django_prometheus.middleware.PrometheusBeforeMiddleware',
    # All your other middlewares go here, including the default
    # middlewares like SessionMiddleware, CommonMiddleware,
    # CsrfViewmiddleware, SecurityMiddleware, etc.
    'django_prometheus.middleware.PrometheusAfterMiddleware',
]

And lastly, on your urls.py:

urlpatterns = [
    ...
    url('', include('django_prometheus.urls')),
]

Also, if you are deploying your code using gunicorn or similar(and you should if you are deploying to a production environment!), pay attention. You need to declare multiple ports since the workers could block each other while trying to service the Prometheus scrape request. This is also very well explained in the django-Prometheus documentation.

PROMETHEUS_METRICS_EXPORT_PORT_RANGE = range(8001, 8050)

Second step: adding our custom django Prometheus metric

This is the actual first step in creating our custom prometheus metric on our django app. register the new metric.

Now it might be a good time to refer to the official Prometheus documentation about metrics. After a good read, we can register it:

Our custom metric will count the number of requests by view, path and referer.

The general idea is to first create a new Metrics class, that inherits from django_prometheus.Metrics. On that we will register own actual metric:

  1. we want it to be a Counter, because it will be increasing over time.
  2. we will unoriginally name it django_http_requests_total_by_view_path_referer,
  3. and add a description that will be useful for documentation purposes.
  4. lastly, our metric will have three labels: view, path and referer so that we can group and filter by them.

Last but not least, do not forget to call the parent method so that we register the remaining metrics.

from django_prometheus.conf import NAMESPACE
from django_prometheus.middleware import Metrics
from prometheus_client import Counter

class CustomMetrics(Metrics):
    def register(self):
        self.requests_total_by_view_path_referer = self.register_metric(
            Counter,
            "django_http_requests_total_by_view_path_referer",
            "Count of requests by view, path, referer.",
            ["view", "path", "referer"],
            namespace=NAMESPACE,
        )
        return super().register()

Third step: creating our custom PrometheusMiddleware

Now that we have our shiny metric, we need to measure it. To do so, we will create two classes that inherit from PrometheusBeforeMiddleware and PrometheusBeforeMiddleware – they will contain our actual metric collection logic.

PrometheusAfterMiddleWare has a bunch of methods where we can add our logic. These methods are called on different moments of the request/response lifecycle, and so receive different parameters:

  • For example, process_exception() receives the request and exception objects,
  • process_response receives() request and response object,
  • the process_view method() receives the request and view related objects.

Take a look at the django-prometheus existing code to figure out the best place to put your metric, putting it closer to similar metrics. For this case process_response() seemed like a good candidate because the view name has been resolved by then.

from django_prometheus.middleware import Metrics, PrometheusBeforeMiddleware, PrometheusAfterMiddleware

...

class AppMetricsBeforeMiddleware(PrometheusBeforeMiddleware):
    metrics_cls = CustomMetrics


class AppMetricsAfterMiddleware(PrometheusAfterMiddleware):
    metrics_cls = CustomMetrics

    def _get_referer_name(self, request):
        referer_name = "<unnamed referer>"
        if hasattr(request, "META"):
            if request.META is not None:
                if request.META.get("HTTP_REFERER") is not None:
                    referer_name = request.META.get("HTTP_REFERER")
        return referer_name

    def process_response(self, request, response):

      self.label_metric(self.metrics.requests_total_by_view_path_referer,
                        request,
                        view=self._get_view_name(request),
                        path=request.path,
                        referer=self._get_referer_name(request)).inc()

        return super().process_response(request, response)

The actual implementation is very simple:

  1. Label the metric with the view name, the path, and referer values:
    1. To get the view we use the django-prometheus built-in _get_view_name function,
    2. To get the referer we adapt the aforementioned function’s logic to get the referer field from the HTTP request header dict, and lastly,
    3. We get the path from the django request object.
  2. We finally increment the resulting metric
  3. Call the parent method to get all the other metrics.

Last step: plugging everything together

This is the easiest step! We just need to replace django-prometheus middleware with our custom middleware in django settings!

Assuming your custom Middleware is in DjangoSite/MyApp/custom_metrics.py:

MIDDLEWARE = [
    'MyApp.custom_metrics.AppMetricsBeforeMiddleware',
    # .. all other middleware
    'MyApp.custom_metrics.AppMetricsAfterMiddleware'
]

If you go to your /metrics endpoint you should see it:

Custom prometheus metric exported from django

Then is can be exported to prometheus so we can do all the cool things with grafana:

A custom django prometheus metric on grafana

Wrapping up

Although it has some tricks, creating your custom Prometheus metric for your Django application is not very difficult.

For my web analytics use-case, it would be trivial to get the user-agent field to know how my visitors are reading my site, if they are registered users or not, etc.

You might like