Monitoring your django site: how to and first steps

  1. How to architect a django website for the real world?
  2. Monitoring your django site: how to and first steps
  3. How to add privacy friendly analytics to your Django website

This is the second post in my Promozilla series. Today I will discuss how I added monitoring to Promozilla, a Nintendo Switch promotion tracking website built with django. In the first post, I described the general architecture.

Why?

Monitoring our django site solves a very simple question. If we want our django website to be used by the world, it is nice for it to be running in the first place.

Monitoring enables us to do that, and preferably in a proactive way. Of course we can ssh in every day and read the logs to check for any issues, but that’s far from perfect.

First, we are not alerted if a problem arises, which means that the site can be down for hours or days before we realise it. Lastly, that’s not really clarifying since the logs may contain too much noise or missing some information.

So there has to be a better way! And there is!

It is fundamental to know the status of our applications. Today I’ll discuss about the purple section of the diagram: monitoring.

How?

Django’s ecosystem makes it very easy to monitor our website. With some dependencies installed and a little bit of configuration, we can be monitoring our django website in no time.

I designed promozilla monitoring to solve both of those issues. Bugsnag alerts me via email if any issue arises and grafana/prometheus allows me to have a global picture of the different components and how they are evolving over time (any degradation while I was away?).

Lastly, take into consideration that if you host your monitoring stack in the same place you host the rest of the infrastructure, if both go down you are out of luck.

The components

Error monitoring: Bugsnag for django

Bugsnag in its simplest core is a dashboard for exceptions. One problem in my previous projects was that the only way for me to know if the sites were up was to either a) visit them, b) have someone complain, c) read the logs. But this is no way to sleep nicely at night. Fortunately, bugsnag is good peace of mind creator.

It provides a very simple django middleware that can be integrated with your django project. Every time an unhandled exception occurs, you receive an email with its stack trace and some very nice details. This can be tuned, of course, but the value it brings out of the box with its free plan is tremendous. One strong point, since it is hosted outside your server: if it is is unreachable you’ll still be able to somehow see how it went down.

It has some pretty good documentation on how to start using bugsnag with django. I seriously recommend bugsnag.

Dashboards: Grafana

Grafana is a powerful monitoring tool. It contains tracing, logging and dashboarding functionalities. To keep things simple, I decided to just use grafana’s dashboard with django.

The dashboards grafana provides are ideal to understand not only business metrics (most popular pages on the site/ who is referring us), but also application metrics (number of errors, database connections, average response time, etc).

Below I’m showing the dashboard I built for promozilla. The first screenshot contains business metrics: visitors over time, number of new accounts and referrers. The second screenshot has service-quality metrics: response time, error rates and requests served.

Grafana dashboard with django site popularity metrics: page views, referrers and new accounts
The dashboard can contain details about our django website popularity: popular pages, page views and referrers
Grafana dashboard with django site popularity metrics:  latenct, requests and errors
Grafana dashboards are also very useful for stability and service monitoring of our django site: latency, requests and errors.

Here I am also showing metrics retrieved from traefik, my reverse proxy, which has Prometheus support out of the box.

If you want to have some inspiration, Grafana has a dashboard and widget showcase page and of course, documentation.

The important thing is: Grafana is not a data storage solution. We need to have a service responsible for just querying and storing our system metrics. For that, I added Prometheus into the mix.

Prometheus with django out of the box

Prometheus is the perfect storage solution to integrate with grafana. At it’s core it is a time-series database designed for metrics storing and querying.

The way it works is tremendously simple: services like django expose an endpoint that Prometheus periodically visits to collect the data. This is called a pull model, where it visits the application to collect the metrics, as opposed to a push model, where the applications send the metrics to the database. This makes everything simpler (e.g. prometheus can be offline and the applications are not affected).

Fortunately for us, django has an extension that exposes a boatload of metrics – but if you are curious, I created a tutorial on how to Create a custom django prometheus metric.

Also, the reverse proxy that I used, traefik, has Prometheus support out of the box, which enables some of the plots you see in the previous screenshots. This is the great thing about prometheus. Because it is so ubiquitous, you do not need to reinvent the wheel to add monitoring to your applications.

Lastly, Prometheus has a particular data model and query syntax when compared to more traditional query languages like SQL. It has some particular concepts like metrics (counter, gauge, histogram and summary) and dimensions that are worth getting familiar with before delving right into it.

Caution: in my experience, Prometheus is quite heavy on the RAM side – be careful if you are running your website in the same machine.

  1. How to architect a django website for the real world?
  2. Monitoring your django site: how to and first steps
  3. How to add privacy friendly analytics to your Django website

Django monitoring: conclusion

I think my django monitoring approach is very solid because it has alerting for when something goes wrong, with bugsnag, but also provides with a birds eye view of recent events and application changes with dashboards provided by grafana and prometheus. I hope this was useful and I’m welcome to any feedback!

You might like

Create a custom django prometheus metric

Today I’ll show you how to create your custom prometheus metric for your django application. Prometheus is an awesome tool to monitor our stack, specially with its integration with grafana and its cool dashboards.

The example metric is for a recent use case. I wanted to measure my website visitors favourite pages, and where they came from.

I wanted to achieve this without tracking, cookies, paid services or having to use Google Analytics. Since I was already using a Prometheus/ Grafana suite to have visibility of my platform health, I decided to use it for this goal as well.

This was the end result, on grafana:

A custom django prometheus metric on grafana

First step: having django-prometheus up and running!

Just follow the instructions on the project readme, they should be just:

pip install django-prometheus

Then, on settings.py

INSTALLED_APPS = [
   ...
   'django_prometheus',
   ...
]

MIDDLEWARE = [
    'django_prometheus.middleware.PrometheusBeforeMiddleware',
    # All your other middlewares go here, including the default
    # middlewares like SessionMiddleware, CommonMiddleware,
    # CsrfViewmiddleware, SecurityMiddleware, etc.
    'django_prometheus.middleware.PrometheusAfterMiddleware',
]

And lastly, on your urls.py:

urlpatterns = [
    ...
    url('', include('django_prometheus.urls')),
]

Also, if you are deploying your code using gunicorn or similar(and you should if you are deploying to a production environment!), pay attention. You need to declare multiple ports since the workers could block each other while trying to service the Prometheus scrape request. This is also very well explained in the django-Prometheus documentation.

PROMETHEUS_METRICS_EXPORT_PORT_RANGE = range(8001, 8050)

Second step: adding our custom django Prometheus metric

This is the actual first step in creating our custom prometheus metric on our django app. register the new metric.

Now it might be a good time to refer to the official Prometheus documentation about metrics. After a good read, we can register it:

Our custom metric will count the number of requests by view, path and referer.

The general idea is to first create a new Metrics class, that inherits from django_prometheus.Metrics. On that we will register own actual metric:

  1. we want it to be a Counter, because it will be increasing over time.
  2. we will unoriginally name it django_http_requests_total_by_view_path_referer,
  3. and add a description that will be useful for documentation purposes.
  4. lastly, our metric will have three labels: view, path and referer so that we can group and filter by them.

Last but not least, do not forget to call the parent method so that we register the remaining metrics.

from django_prometheus.conf import NAMESPACE
from django_prometheus.middleware import Metrics
from prometheus_client import Counter

class CustomMetrics(Metrics):
    def register(self):
        self.requests_total_by_view_path_referer = self.register_metric(
            Counter,
            "django_http_requests_total_by_view_path_referer",
            "Count of requests by view, path, referer.",
            ["view", "path", "referer"],
            namespace=NAMESPACE,
        )
        return super().register()

Third step: creating our custom PrometheusMiddleware

Now that we have our shiny metric, we need to measure it. To do so, we will create two classes that inherit from PrometheusBeforeMiddleware and PrometheusBeforeMiddleware – they will contain our actual metric collection logic.

PrometheusAfterMiddleWare has a bunch of methods where we can add our logic. These methods are called on different moments of the request/response lifecycle, and so receive different parameters:

  • For example, process_exception() receives the request and exception objects,
  • process_response receives() request and response object,
  • the process_view method() receives the request and view related objects.

Take a look at the django-prometheus existing code to figure out the best place to put your metric, putting it closer to similar metrics. For this case process_response() seemed like a good candidate because the view name has been resolved by then.

from django_prometheus.middleware import Metrics, PrometheusBeforeMiddleware, PrometheusAfterMiddleware

...

class AppMetricsBeforeMiddleware(PrometheusBeforeMiddleware):
    metrics_cls = CustomMetrics


class AppMetricsAfterMiddleware(PrometheusAfterMiddleware):
    metrics_cls = CustomMetrics

    def _get_referer_name(self, request):
        referer_name = "<unnamed referer>"
        if hasattr(request, "META"):
            if request.META is not None:
                if request.META.get("HTTP_REFERER") is not None:
                    referer_name = request.META.get("HTTP_REFERER")
        return referer_name

    def process_response(self, request, response):

      self.label_metric(self.metrics.requests_total_by_view_path_referer,
                        request,
                        view=self._get_view_name(request),
                        path=request.path,
                        referer=self._get_referer_name(request)).inc()

        return super().process_response(request, response)

The actual implementation is very simple:

  1. Label the metric with the view name, the path, and referer values:
    1. To get the view we use the django-prometheus built-in _get_view_name function,
    2. To get the referer we adapt the aforementioned function’s logic to get the referer field from the HTTP request header dict, and lastly,
    3. We get the path from the django request object.
  2. We finally increment the resulting metric
  3. Call the parent method to get all the other metrics.

Last step: plugging everything together

This is the easiest step! We just need to replace django-prometheus middleware with our custom middleware in django settings!

Assuming your custom Middleware is in DjangoSite/MyApp/custom_metrics.py:

MIDDLEWARE = [
    'MyApp.custom_metrics.AppMetricsBeforeMiddleware',
    # .. all other middleware
    'MyApp.custom_metrics.AppMetricsAfterMiddleware'
]

If you go to your /metrics endpoint you should see it:

Custom prometheus metric exported from django

Then is can be exported to prometheus so we can do all the cool things with grafana:

A custom django prometheus metric on grafana

Wrapping up

Although it has some tricks, creating your custom Prometheus metric for your Django application is not very difficult.

For my web analytics use-case, it would be trivial to get the user-agent field to know how my visitors are reading my site, if they are registered users or not, etc.

You might like