How to add privacy friendly analytics to your Django website

  1. How to architect a django website for the real world?
  2. Monitoring your django site: how to and first steps
  3. How to add privacy friendly analytics to your Django website

What are web analytics?

Why do we need cookieless analytics in a django site? We now have a website running, and we want to measure its success. Today I’ll show you how I added web analytics to promozilla, my django side project, while respecting my users privacy.

Web analytics provide a way to understand how our website is being used, and who are the users. It collects metrics like referrers (where the users clicked to visit the site), search engine keywords, landing pages, countries of origin, preferred languages or device information like operating system, type of device or screen size.

The good thing is that it provides analytics on top of the metrics, so we can understand the general trends and our users preferences to make decisions. For example, if the bounce rate is much higher in mobile rather than desktop, maybe our website is not mobile friendly. Or maybe if our second biggest country is France, localizing to french might be a good idea.

The biggest contender in this space is without doubt Google Analytics, but it come with an hefty downside: our privacy. Since it stores cookies in our end-user’s computers, it is able to track them across sessions, but it also stores information that I think is not really needed for our case.

Promozilla

This is the third post in a series about my side project, promozilla. Promozilla is a Nintendo Switch promotion tracker built using Django. In the previous post, I showed how to add monitoring to our django website. Today I will focus on the bottom left corner of its architecture: privacy-friendly cookieless analytics.

Promozilla is a django site with cookieless analytics

Cookieless analytics

Come cookieless analytics. Cookieless analytics are perfect for a django website, because they provide analytics to track the site popularity without infringing on your users privacy and are easy to set up. Why? It only takes a single line of code and you do not even have to add those cookie consent banners!

Demo of data centurion analytics features
Source: Data Centurion

Data centurion?

There are many solutions in this space, like Matomo or Fanthom Analytics but I decided to settle for Data Centurion because it provides a nice free plan for our side project.The free plan comes with unlimited websites but only 1000 page views per month. The paid plans start at 2.99 euros per month, well under Matomo’s 29€ and Fanthom Analytics 14€ plans. This allows your site to grow without having to pay a big subscription upfront. Plus it’s a new contender in this space, and who doesn’t like a underdog?

How to add cookieless analytics to our django website?

1)Create account

You can create a free account in Data Centurion’s web page: https://datacenturion.io/register

2) Add website

Click on: https://datacenturion.io/websites/new and fill in the required information. I prefer to set the ignored ip addresses later. Don’t forget to click on the “Notifications” checkmark if you want to receive to tasty statistics. Lastly, hang on to the script at the end of the page- it will be useful later.

Data centurion new website page

3) Adding cookieless analytics to our django website – base template

We need to connect our website to our Data centurion account and we do that by putting that last snippet in a django html template. The script has some javascript that runs every time a page is loaded, it then sends the anonymized data back to the server. It is important to put this script in some place that is loaded in every page, to ensure our django website analytics are calculated everywhere. For example, promozilla has a base template every page inherits to have access to the static files and general page structure, like navbar, footer and content. I placed the snippet in the <header> element.

Django html template with data centurion tracking snippet

4) Ignore Ip addresses

One last thing: we do not want to skew our analytics and artificially inflate our stats – unless your self-esteem requires that. If you are not like that, you can simply add your ip address to a blacklist and so your activity will not count towards your site usage. Super easy:

  1. Just google “what is my ip address” and copy it
  2. Go to your website settings page in Data centurion
  3. Copy you address to the “blocked ips” text field and you are good to go!

Event tracking

Data centurion also has event tracking as feature, but unfortunately it is well hidden – there is no documentation about it. You can track specific actions of your site, like new accounts, sales, etc, and then you can do analytics on top of it. You can see more information about it in this medium post.

Conclusion

Web analytics are important tools to track our django website usage and measure its success. Cookieless analytics provide a way to achieve that without infringing our users privacy. Lastly, they are very easy to add to our site and some providers, like Data centurion, provide free plans, so there’s really no excuse to not add them.

  1. How to architect a django website for the real world?
  2. Monitoring your django site: how to and first steps
  3. How to add privacy friendly analytics to your Django website

You might like

Create a custom django prometheus metric

Today I’ll show you how to create your custom prometheus metric for your django application. Prometheus is an awesome tool to monitor our stack, specially with its integration with grafana and its cool dashboards.

The example metric is for a recent use case. I wanted to measure my website visitors favourite pages, and where they came from.

I wanted to achieve this without tracking, cookies, paid services or having to use Google Analytics. Since I was already using a Prometheus/ Grafana suite to have visibility of my platform health, I decided to use it for this goal as well.

This was the end result, on grafana:

A custom django prometheus metric on grafana

First step: having django-prometheus up and running!

Just follow the instructions on the project readme, they should be just:

pip install django-prometheus

Then, on settings.py

INSTALLED_APPS = [
   ...
   'django_prometheus',
   ...
]

MIDDLEWARE = [
    'django_prometheus.middleware.PrometheusBeforeMiddleware',
    # All your other middlewares go here, including the default
    # middlewares like SessionMiddleware, CommonMiddleware,
    # CsrfViewmiddleware, SecurityMiddleware, etc.
    'django_prometheus.middleware.PrometheusAfterMiddleware',
]

And lastly, on your urls.py:

urlpatterns = [
    ...
    url('', include('django_prometheus.urls')),
]

Also, if you are deploying your code using gunicorn or similar(and you should if you are deploying to a production environment!), pay attention. You need to declare multiple ports since the workers could block each other while trying to service the Prometheus scrape request. This is also very well explained in the django-Prometheus documentation.

PROMETHEUS_METRICS_EXPORT_PORT_RANGE = range(8001, 8050)

Second step: adding our custom django Prometheus metric

This is the actual first step in creating our custom prometheus metric on our django app. register the new metric.

Now it might be a good time to refer to the official Prometheus documentation about metrics. After a good read, we can register it:

Our custom metric will count the number of requests by view, path and referer.

The general idea is to first create a new Metrics class, that inherits from django_prometheus.Metrics. On that we will register own actual metric:

  1. we want it to be a Counter, because it will be increasing over time.
  2. we will unoriginally name it django_http_requests_total_by_view_path_referer,
  3. and add a description that will be useful for documentation purposes.
  4. lastly, our metric will have three labels: view, path and referer so that we can group and filter by them.

Last but not least, do not forget to call the parent method so that we register the remaining metrics.

from django_prometheus.conf import NAMESPACE
from django_prometheus.middleware import Metrics
from prometheus_client import Counter

class CustomMetrics(Metrics):
    def register(self):
        self.requests_total_by_view_path_referer = self.register_metric(
            Counter,
            "django_http_requests_total_by_view_path_referer",
            "Count of requests by view, path, referer.",
            ["view", "path", "referer"],
            namespace=NAMESPACE,
        )
        return super().register()

Third step: creating our custom PrometheusMiddleware

Now that we have our shiny metric, we need to measure it. To do so, we will create two classes that inherit from PrometheusBeforeMiddleware and PrometheusBeforeMiddleware – they will contain our actual metric collection logic.

PrometheusAfterMiddleWare has a bunch of methods where we can add our logic. These methods are called on different moments of the request/response lifecycle, and so receive different parameters:

  • For example, process_exception() receives the request and exception objects,
  • process_response receives() request and response object,
  • the process_view method() receives the request and view related objects.

Take a look at the django-prometheus existing code to figure out the best place to put your metric, putting it closer to similar metrics. For this case process_response() seemed like a good candidate because the view name has been resolved by then.

from django_prometheus.middleware import Metrics, PrometheusBeforeMiddleware, PrometheusAfterMiddleware

...

class AppMetricsBeforeMiddleware(PrometheusBeforeMiddleware):
    metrics_cls = CustomMetrics


class AppMetricsAfterMiddleware(PrometheusAfterMiddleware):
    metrics_cls = CustomMetrics

    def _get_referer_name(self, request):
        referer_name = "<unnamed referer>"
        if hasattr(request, "META"):
            if request.META is not None:
                if request.META.get("HTTP_REFERER") is not None:
                    referer_name = request.META.get("HTTP_REFERER")
        return referer_name

    def process_response(self, request, response):

      self.label_metric(self.metrics.requests_total_by_view_path_referer,
                        request,
                        view=self._get_view_name(request),
                        path=request.path,
                        referer=self._get_referer_name(request)).inc()

        return super().process_response(request, response)

The actual implementation is very simple:

  1. Label the metric with the view name, the path, and referer values:
    1. To get the view we use the django-prometheus built-in _get_view_name function,
    2. To get the referer we adapt the aforementioned function’s logic to get the referer field from the HTTP request header dict, and lastly,
    3. We get the path from the django request object.
  2. We finally increment the resulting metric
  3. Call the parent method to get all the other metrics.

Last step: plugging everything together

This is the easiest step! We just need to replace django-prometheus middleware with our custom middleware in django settings!

Assuming your custom Middleware is in DjangoSite/MyApp/custom_metrics.py:

MIDDLEWARE = [
    'MyApp.custom_metrics.AppMetricsBeforeMiddleware',
    # .. all other middleware
    'MyApp.custom_metrics.AppMetricsAfterMiddleware'
]

If you go to your /metrics endpoint you should see it:

Custom prometheus metric exported from django

Then is can be exported to prometheus so we can do all the cool things with grafana:

A custom django prometheus metric on grafana

Wrapping up

Although it has some tricks, creating your custom Prometheus metric for your Django application is not very difficult.

For my web analytics use-case, it would be trivial to get the user-agent field to know how my visitors are reading my site, if they are registered users or not, etc.

You might like