17.

2: Illustrating Design Concepts With a Simple API

TIP: Classy Django REST Framework Is a Useful Reference

For working with the Django Rest Framework, we’ve found that http://cdrf.co
is a great cheat sheet. It is patterned after the famous ccbv.co.uk reference site,
but tailored for Django REST Framework.

Now we’ll wire this into our flavors/urls.py module:

Example 17.5: Wiring in API Views

# flavors/urls.py
from django.urls import path

from flavors.api import views

urlpatterns = [
# /flavors/api/
path(
route='api/',
view=views.FlavorListCreateAPIView.as_view(),
name='flavor_rest_api'
),
# /flavors/api/:uuid/
path(
route='api/<uuid:uuid>/',
view=views.FlavorRetrieveUpdateDestroyAPIView.as_view(),
name='flavor_rest_api'
)
]

What we are doing is reusing the URLConf name, making it easier to manage when you
have a need for a JavaScript-heavy front-end. All you need to do is access the Flavor resource
via the {% url %} template tag.

In case it’s not clear exactly what our URLConf is doing, let’s review it with a table:

Url View Url Name (same)

/flavors/api/ FlavorListCreateAPIView flavor_rest_api
/flavors/api/:uuid/ FlavorRetrieveUpdateDestroyAPIView
flavor_rest_api

Table 17.3: URLConf for the Flavor REST APIs

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 213

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 17: Building REST APIs With Django REST Framework

WARNING: Our Simple API Does Not Use Permissions

We overrode the default IsAdmin permission with IsAuthenticated. If you im-
plement an API using our example, don’t forget to assign user permissions appro-
priately!
ä django-rest-framework.org/api-guide/authentication
ä django-rest-framework.org/api-guide/permissions

The end result is the traditional REST-style API definition:

Example 17.6: Traditional REST-style API definition

flavors/api/
flavors/api/:uuid/

TIP: Common Syntax for Describing REST APIs

It’s not uncommon to see syntax like what is described in the Wiring in API Views
code example. In this particular case, /flavors/api/:uuid/ includes a :uuid
value. This represents a variable, but in a manner suited for documentation across
frameworks and languages, and you’ll see it used in many third-party REST API
descriptions.

We’ve shown you (if you didn’t know already) how it’s very easy to build REST APIs in
Django. Now let’s go over some advice on maintaining and extending them.

17.3 REST API Architecture

Building simple, quick APIs is easy with Django REST Framework, but extending and
maintaining it to match your project’s needs takes a bit more thought. This is usually where
people get hung up on API design. Here are some tips for improving your design:

17.3.1 Use Consistent API Module Naming

Just like anything else, how things are named needs to be consistent across a project. Our
preferences for naming module related to API design is as follows:

Example 17.7: Our Preferences for Naming API-Related Modules

flavors/
├── api/

214 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 17.3: REST API Architecture

│   ├── __init__.py

│   ├── authentication.py
│   ├── parsers.py
│   ├── permissions.py
│   ├── renderers.py
│   ├── serializers.py
│   ├── validators.py
│   ├── views.py
│   ├── viewsets.py

Please observe the following:

ä We like to place all our API components into a package within an app called api/ .
That allows us to isolate our API components in a consistent location. If we were to
put it in the root of our app, then we would end up with a huge list of API-specific
modules in the general area of the app.
ä Viewsets belong in their own module.
ä We always place routers in urls.py. Either at the app or project level, routers belong
in urls.py.

17.3.2 Code for a Project Should Be Neatly Organized

For projects with a lot of small, interconnecting apps, it can be hard to hunt down where
a particular API view lives. In contrast to placing all API code within each relevant app,
sometimes it makes more sense to build an app specifically for the API. This is where all
the serializers, renderers, and views are placed.

Therefore, the name of the app should reflect its API version (see Section 17.3.7: Version
Your API).

For example, we might place all our views, serializers, and other API components in an app
titled apiv4.

The downside is the possibility for the API app to become too large and disconnected from
the apps that power it. Hence we consider an alternative in the next subsection.

17.3.3 Code for an App Should Remain in the App

When it comes down to it, REST APIs are just views. For simpler, smaller projects, REST
API views should go into views.py or viewsets.py modules and follow the same guidelines we

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 215

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 17: Building REST APIs With Django REST Framework

endorse when it comes to any other view. The same goes for app- or model-specific serializers
and renderers. If we do have app-specific serializers or renderers, the same applies.

For apps with so many REST API view classes that it makes it hard to navigate a single
api/views.py or api/viewsets.py module, we can break them up. Specifically, we move our
view (or viewset) classes into a api/views/ (or api/viewsets/ ) package containing Python
modules typically named after our models. So you might see:

flavors/
├── api/
│   ├── __init__.py
│   ├── ... other modules here
│   ├── views
│   │   ├── __init__.py
│   │   ├── flavor.py
│   │   ├── ingredient.py

The downside with this approach is that if there are too many small, interconnecting apps,
it can be hard to keep track of the myriad of places API components are placed. Hence we
considered another approach in the previous subsection.

17.3.4 Try to Keep Business Logic Out of API Views

Regardless of which architectural approach you take, it’s a good idea to try to keep as much
logic as possible out of API views. If this sounds familiar, it should. We covered this in
Section 8.5: Try to Keep Business Logic Out of Views. Remember, at the end of the day,
API views are just another type of view.

216 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 17.3: REST API Architecture

Figure 17.1: An Ice Cream as a Service API.

17.3.5 Grouping API URLs

If you have REST API views in multiple Django apps, how do you build a project-wide
API that looks like this?

Example 17.9: Project-Wide API Design

api/flavors/ # GET, POST

api/flavors/:uuid/ # GET, PUT, DELETE
api/users/ # GET, POST
api/users/:uuid/ # GET, PUT, DELETE

In the past, we placed all API view code into a dedicated Django app called api or apiv1,
with custom logic in some of the REST views, serializers, and more. In theory it’s a pretty
good approach, but in practice it means we have logic for a particular app in more than just
one location.

Our current approach is to lean on URL configuration. When building a project-wide API
we write the REST views in the api/views.py or api/viewsets.py modules, wire them into

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 217

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 17: Building REST APIs With Django REST Framework

a URLConf called something like core/api_urls.py or core/apiv1_urls.py, and include that

from the project root’s urls.py module. This means that we might have something like the
following code:

Example 17.10: Combining Multiple App API Views Into One

# core/api_urls.py
"""Called from the project root's urls.py URLConf thus:
path('api/', include('core.api_urls', namespace='api')),
"""
from django.urls import path

from flavors.api import views as flavor_views

from users.api import views as user_views

urlpatterns = [
# {% url 'api:flavors' %}
path(
route='flavors/',
view=flavor_views.FlavorCreateReadView.as_view(),
name='flavors'
),
# {% url 'api:flavors' flavor.uuid %}
path(
route='flavors/<uuid:uuid>/',
view=flavor_views.FlavorReadUpdateDeleteView.as_view(),
name='flavors'
),
# {% url 'api:users' %}
path(
route='users/',
view=user_views.UserCreateReadView.as_view(),
name='users'
),
# {% url 'api:users' user.uuid %}
path(
route='users/<uuid:uuid>/',
view=user_views.UserReadUpdateDeleteView.as_view(),
name='users'
),
]

218 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 17.3: REST API Architecture

17.3.6 Test Your API

We find that Django’s test suite makes it really easy to test API implementations. It’s cer-
tainly much easier than staring at curl results! Testing is covered at length in Chapter 24:
Testing Stinks and Is a Waste of Money!. We even include in that chapter the tests we wrote
for our simple JSON API (see Section 24.3.1: Each Test Method Tests One Thing).

17.3.7 Version Your API

It’s a good practice to abbreviate the urls of your API with the version number e.g.
/api/v1/flavors or /api/v1/users and then as the API changes, /api/v2/flavors
or /api/v2/users. We prefer this method or the host name style of versioning
v1.icecreamlandia.com/api/users. When the version number changes, existing cus-
tomers can continue to use the previous version without unknowingly breaking their calls
to the API.

Also, in order to avoid angering API consumers, it’s critical to maintain both the existing
API and the predecessor API during and after upgrades. It’s not uncommon for the depre-
cated API to remain in use for several months.

When we implement a new version of the API, we provide customers/users with a dep-
recation warning well in advance of breaking API changes so they can perform necessary
upgrades and not break their own applications. From personal experience, the ability to send
a deprecation warning to end users is an excellent reason to request email addresses from
users of even free and open source API services.

Django REST Framework has built-in mechanisms for supporting the preferred
scheme we list above as well as other approaches. Documentation for this exists at
django-rest-framework.org/api-guide/versioning/

17.3.8 Be Careful With Customized Authentication Schemes

If you’re building an API and need a customized authentication scheme, be extra careful.
Security is hard, and there are always unpredictable edge cases, which is how people pen-
etrate sites. We’ve only had to implement customized authentication schemes a few times,
but we kept the following in mind:

ä If we’re creating a new authentication scheme, we keep it simple and well tested.
ä Outside of the code, we document why existing standard authentication schemes are
insufficient. See the tipbox below.
ä Also outside of the code, we document in depth how our authentication scheme is
designed to work. See the tipbox below.

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 219

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 17: Building REST APIs With Django REST Framework

ä Unless we are writing a non-cookie based scheme, we don’t disable CSRF.

TIP: Documentation Is Critical for Customized Authentication

Writing out the why and how of a customized authentication scheme is a critical
part of the process. Don’t skip it! Here’s why:
ä Helps us validate our reasoning for coming up with something new. If we
can’t describe the problem in writing, then we don’t fully understand it.
ä Documentation forces us to architect the solution before we code it.
ä After the system is in place, later the documentation allows us (or others) to
understand why we made particular design decisions.

17.4 When DRF Gets in the Way

Django Rest Framework is a powerful tool that comes with a lot of abstractions. Trying to
work through these abstractions can prove to be extremely frustrating. Let’s take a look on
overcoming them.

17.4.1 Remote Procedure Calls vs REST APIs

The resource model used by REST frameworks to expose data is very powerful, but it doesn’t
cover every case. Specifically, resources don’t always match the reality of application design.
For example, it is easy to represent syrup and a sundae as two resources, but what about the
action of pouring syrup? Using this analogy, we change the state of the sundae and decrease
the syrup inventory by one. While we could have the API user change things individually,
that can generate issues with database integrity. Therefore in some cases it can be a good idea to
present a method like sundae.pour_syrup(syrup) to the client as part of the RESTful
API.

In computer science terms, sundae.pour_syrup(syrup) could be classified as a Remote

Procedure Call or RPC.

References:

ä en.wikipedia.org/wiki/Remote_Procedure_Call
ä en.wikipedia.org/wiki/Resource-oriented_architecture

Fortunately, RPC calls are easy to implement with Django Rest Framework. All we have to
do is ignore the abstraction tools of DRF and rely instead on its base APIView:

220 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 17.4: When DRF Gets in the Way

Example 17.11: Implementing pour_syrup() RPC with DRF

# sundaes/api/views.py
from django.shortcuts import get_object_or_404

from rest_framework.response import Response

from rest_framework.views import APIView

from ..models import Sundae, Syrup

from .serializers import SundaeSerializer, SyrupSerializer

class PourSyrupOnSundaeView(APIView):
"""View dedicated to adding syrup to sundaes"""

def post(self, request, *args, **kwargs):

# Process pouring of syrup here,
# Limit each type of syrup to just one pour
# Max pours is 3 per sundae
sundae = get_object_or_404(Sundae,
,→ uuid=request.data['uuid'])
try:
sundae.add_syrup(request.data['syrup'])
except Sundae.TooManySyrups:
msg = "Sundae already maxed out for syrups"
return Response({'message': msg}, status_code=400)
except Syrup.DoesNotExist
msg = "{} does not
,→ exist".format(request.data['syrup'])
return Response({'message': msg}, status_code=404)
return Response(SundaeSerializer(sundae).data)

def get(self, request, *args, **kwargs)

# Get list of syrups already poured onto the sundae
sundae = get_object_or_404(Sundae,
,→ uuid=request.data['uuid'])
syrups = [SyrupSerializer(x).data for x in
,→ sundae.syrup_set.all()]
return Response(syrups)

And our API design would look like this now:

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 221

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 17: Building REST APIs With Django REST Framework

Example 17.12: Sundae and Syrup API Design

/sundae/ # GET, POST

/sundae/:uuid/ # PUT, DELETE
/sundae/:uuid/syrup/ # GET, POST
/syrup/ # GET, POST
/syrup/:uuid/ # PUT, DELETE

17.4.2 Problems With Complex Data

Okay, we’ll admit it, we make this mistake with DRF about once a month. Let’s sum up
what happens in very simple terms with the following API design:

Example 17.13: A Cone and Scoop API

/api/cones/ # GET, POST

/api/cones/:uuid/ # PUT, DELETE
/api/scoops/ # GET, POST
/api/scoops/:uuid/ # PUT, DELETE

1 We have a model (Scoop) that we want represented within another (Cone)

2 We can easily write a GET of the Cone that includes a list of its Scoops
3 On the other hand, writing a POST or PUT of Cones that also adds or updates
its Scoops at the same time can be challenging, especially if it requires any kind of
validation or post processing
4 Frustration sets in and we leave to get some real-world ice cream

While there are nicely complex solutions for nested data, we’ve found a better solution. And
that is to simplify things just a bit. Example:

ä Keep the GET representation of the Cone that includes its Scoops
ä Remove any capability of the POST or PUT for the Cone model to modify Scoops
for that cone.
ä Create GET/POST/PUT API views for Scoops that belong to a Cone.

Our end API will now look like this:

222 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 17.4: When DRF Gets in the Way

Example 17.14: A Cone and Scoop API

/api/cones/ # GET, POST

/api/cones/:uuid/ # PUT, DELETE
/api/cones/:uuid/scoops/ # GET, POST
/api/cones/:uuid/scoops/:uuid/ # PUT, DELETE
/api/scoops/ # GET, POST
/api/scoops/:uuid/ # PUT, DELETE

Yes, this approach does add extra views and additional API calls. On the other hand, this
kind of data modeling can result in simplification of your API. That simplification will result
in easier testing, hence a more robust API.

For what it’s worth, if you take a close look at the Stripe API reference (stripe.com/
docs/api) you’ll see they follow our pattern. You can view complex data, but you have to
create it bit-by-bit.

17.4.3 Simplify! Go Atomic!

In the previous two subsections (RPC Calls and Problems With Complex Data), we’ve
established a pattern of simplification. In essence, when we run into problems with DRF
we ask these questions:

ä Can we simplify our views? Does switching to APIView resolve the problem?
ä Can we simplify our REST data model as described by views? Does adding more
views (of a straightforward nature) resolve the problem?
ä If a serializer is troublesome and is outrageously complex, why not break it up into
two different serializers for the same model?

As you can see, to overcome problems with DRF, we break our API down into smaller,
more atomic components. We’ve found that it’s better to have more views designed to be
as atomic as possible than a few views with many options. As any experienced programmer
knows, more options means more edge cases.

Atomic-style components help in these regards:

ä Documentation is easier/faster because each component does less

ä Easier testing since there are less code branches
ä Bottlenecks are easier to resolve because chokepoints are more isolated
ä Security is better since we can easily change access per view rather than within the
code of a view

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 223

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 17: Building REST APIs With Django REST Framework

17.5 Shutting Down an External API

When it’s time to shut down an older version of an external API in favor of a new one, here
are useful steps to follow:

17.5.1 Step #1: Notify Users of Pending Shut Down

Provide as much advance notice as possible. Preferably six months, but as short as one month.
Inform API users via email, blogs, and social media. We like to report the shutdown noti-
fication to the point that we worry people are getting tired of the message.

17.5.2 Step #2: Replace API With 410 Error View

When the API is finally shut down, we provide a simple 410 Error View. We include a very
simple message that includes the following information:

ä A link to the new API’s endpoint.

ä A link to the new API’s documentation.
ä A link to an article describing the details of the shut down.

Below is a sample shutdown view that works against any HTTP method:

Example 17.15: Code for a Shutdown

# core/apiv1_shutdown.py
from django.http import HttpResponseGone

apiv1_gone_msg = """APIv1 was removed on April 2, 2017. Please

,→ switch to APIv2:
<ul>
<li>
<a href="https://www.example.com/api/v3/">APIv3
,→ Endpoint</a>
</li>
<li>
<a href="https://example.com/apiv3_docs/">APIv3
,→ Documentation</a>
</li>
<li>
<a href="http://example.com/apiv1_shutdown/">APIv1 shut
,→ down notice</a>
</li>
</ul>

224 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 17.6: Rate-Limiting Your API

"""

def apiv1_gone(request):
return HttpResponseGone(apiv1_gone_msg)

17.6 Rate-Limiting Your API

Rate limiting is when an API restricts how many requests can be made by a user of the API
within a period of time. This is done for a number of reasons, which we’ll explain below.

17.6.1 Unfettered API Access Is Dangerous

In the ancient times (2010) we launched the djangopackages.org website. The project,
started during the Django Dash contest, was an instant hit for the Django community. We
sprinted on it constantly, and its feature set grew rapidly. Unfortunately, we quickly hit the
rate limit of GitHub’s first API. This meant that after a certain amount of API requests per
hour, we weren’t allowed to make any more until a new hour passed.

Fortunately at DjangoCon 2010 we had the opportunity to ask one of the founders of
GitHub if we could have unlimited access to their API. He graciously said ‘yes’ and within
a day we could get data from GitHub as much as we wanted.

We were delighted. Our users were delighted. Usage of the site increased, people were thirsty
for data as to what were the most active projects. So desirous of data were we that every hour
we requested the latest data from GitHub. And that caused a problem for GitHub.

You see, this was 2010 and GitHub was not the giant, powerful company it is today. At 17
minutes past each hour, Django Packages would send thousands of requests to the GitHub
API in a very short period. With unfettered access we were causing them problems.

Eventually, GitHub contacted us and requested that we scale back how much we were using
their API. We would still have unlimited access, just needed to give them breathing room.
We complied, checking data once per day instead of by the hour, and at a more reasonable
rate. We continue to do so to this day.

While modern GitHub can certainly handle much, much larger volumes of API access then
it could in late 2010, we like to think we learned a shared lesson about unfettered access to
an API: Grant such access cautiously.

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 225

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 17: Building REST APIs With Django REST Framework

17.6.2 REST Frameworks Must Come With Rate Limiting

Controlling the volume of REST API access can mean the difference between joyful tri-
umph or utter disaster in a project.

TIP: HTTP Server Rate Limiting

It’s possible to use nginx or apache httpd for rate limiting. The upside is faster per-
formance. The downside is that it removes this functionality from the Python code.
For a python-based system, DRF provides some basic tools that can be built upon.
ä nginx.com/blog/rate-limiting-nginx/
ä httpd.apache.org/docs/2.4/mod/mod_ratelimit.html
ä django-rest-framework.org/api-guide/throttling/
#setting-the-throttling-policy

17.6.3 Rate Limiting Can Be a Business Plan

Imagine we launch an API-based startup that lets users add images of toppings to images
of ice cream. We know that everyone will want to use this API, and come up with several
tiers of access that we tie into pricing:

Developer tier is free, but only allows 10 API requests per hour.
One Scoop is $24/month, allows 25 requests per minute.
Two Scoops is $79/month, allows 50 requests per minute.
Corporate is $5000/month, allows for 200 requests per minute.

This method is called Bucket-Based Pricing. Now all we have to do is get people to use
our API.

TIP: Consider Usage-Based Pricing Instead of Rate Limit-Based

Pricing
Randall Degges makes a strong argument that the business plan of bucket based
pricing we suggest in this section is not a good business plan. He advocates unlimited
usage from the beginning, where API users are charged in volume, with the price
per unit going down as usage increases. His version of our chart might then be:
ä First 1000 requests made per month are free
ä 1001 to 100,000 requests made per month cost 0.4 cents each
ä 100,001 to 200,000 requests made per month cost 0.3 cents each
ä 200,001+ requests made per month cost 0.2 cents each
The advantage of Usage-Based Pricing is that it incentives volume over trying to

226 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 17.7: Advertising Your REST API

fit into buckets or tiers. The downside is it will require more work to set up this
billing plan in your project. Third-party services like Stripe and Paddle can add fees
or additional amounts to subscriptions, so it’s possible to do in common payment
systems.
Reference: rdegges.com/2020/the-only-type-of-api-services-ill-use

17.7 Advertising Your REST API

Let’s assume we’ve built our REST API and want outside coders and companies to use it.
How do we go about doing that?

17.7.1 Documentation
The most important thing to do is to provide comprehensive documentation. The eas-
ier to read and understand the better. Providing easy-to-use code examples is a must.
You can write it from scratch or use auto-documentation tools provided by django-
rest-framework itself and various third-party packages (django-rest-framework.org/
topics/documenting-your-api/#third-party-packages. You can even embrace
commercial documentation generation services like readthedocs.com and swagger.io.

Some of the material in Chapter 25: Documentation: Be Obsessed might prove useful for
forward-facing REST API documentation.

17.7.2 Provide Client SDKs

Something that may help spread use of your API is to provide a software development
kits (SDK) for various programming languages. The more programming languages covered
the better. For us, we’ve found the must-have languages include Python, JavaScript, Ruby,
PHP, Go, and Java.

In our experience, it’s a good idea to write at least one of these libraries ourselves and create
a demo project. The reason is that it not only advertises our API, it forces us to experience
our API from the same vantage point as our consumers.

Fortunately for us, thanks to the underlying OpenAPI document object model of
(openapis.org/), DRF provides JSON Hyperschema-compatible functionality. When
this format is followed for both the client- and server-side, OpenAPI allows for writing
dynamically driven client libraries that can interact with any API that exposes a supported
schema or hypermedia format.

These tools should work immediately with most DRF-powered APIs:

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 227

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 17: Building REST APIs With Django REST Framework

ä Command line client

https://www.django-rest-framework.org/api-guide/schemas/
#generating-an-openapi-schema
ä Multi-language Client SDK generator
github.com/OpenAPITools/openapi-generator
ä swagger-cli for use with JavaScript
npmjs.com/package/swagger-cli

For building Python-powered client SDKs, reading Section 23.9: Releasing Your Own
Django Packages might prove useful.

17.8 Additional Reading

We highly recommend reading the following:

ä en.wikipedia.org/wiki/REST
ä en.wikipedia.org/wiki/List_of_HTTP_status_codes
ä github.com/OAI/OpenAPI-Specification
ä jacobian.org/writing/rest-worst-practices/

17.9 Other Approaches for Crafting APIs

For the reasons explained at the beginning of this chapter, we recommend Django Rest Frame-
work. However, should you choose not to use DRF, consider the following approaches:

17.9.1 CBV Approach: JsonResponse with View

It is possible to use Django’s built-in django.http.JsonResponse class, which is a sub-
class of django.http.HttpResponse , on django.views.generic.View. This sup-
ports the full range of HTTP methods but has no support for OpenAPI. . This is proven
to work with async views.

Example 17.16: Simple JsonResponse View

class FlavorApiView(LoginRequiredMixin,View):
def post(self, request, *args, **kwargs):
# logic goes here
return JsonResponse({})

def get(self, request, *args, **kwargs):

# logic goes here
return JsonResponse({})

def put(self, request, *args, **kwargs):

228 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 17.10: Summary

# logic goes here

return JsonResponse({})

def delete(self, request, *args, **kwargs):

# logic goes here
return JsonResponse({})

17.9.2 FBV approach: django-jsonview

While you can use DRF with function-based views, that approach doesn’t have all the fea-
tures of CBV-based DRF. A simpler approach is to use django-jsonview. The downside is
that when you get into the full range of HTTP methods and more complex API designs,
we’ve found in Django FBVs become a hindrance due to their lack of focus on building
APIs or support for OpenAPI.

17.9.3 django-tastypie
django-tastypie is a mature API framework that implements its own class-based view sys-
tem. Predating Django REST Framework by 3 years, it’s a feature-rich, mature, powerful,
stable tool for creating APIs from Django models.

django-tastypie.readthedocs.io/

Figure 17.2: A tasty pie is one filled with ice cream.

17.10 Summary
In this chapter we covered:

ä Why you should use Django Rest Framework

ä Basic REST API concepts and how they relate to Django Rest Framework
ä Security considerations
ä Grouping strategies
ä Simplification strategies

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 229

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 17: Building REST APIs With Django REST Framework

ä Fundamentals of basic REST API design

ä Alternatives to Django REST Framework

Coming up next, we’ll go over the other side of REST APIs in Chapter 19: JavaScript and
Django.

230 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

18 | Building GraphQL APIs With
Django

WARNING: This Chapter is in Progress

With the Async Views still not in core, the landscape for Django and GraphQL
remains incomplete. We also did not have enough time to finish consolidating our
thoughts.
Therefore, in the days to come we will expand on the material here. We are open
to suggestions on topics and items to cover, please submit them to github.com/
feldroy/two-scoops-of-django-3.x/issues

Started in 2012, GraphQL is a newcomer to the world of API formats. Unlike REST,
GraphQL comes with schemas, types, and one of our favorite features is a a built-in method
to handle real-time update (Subscriptions). Clients of an API specify the data they want,
and can easily figure out queries because the schemas and types makes it very easy to build
instrospection tools. GraphQL responses are easily serialized to JSON or YAML. For these
reasons and more, GraphQL has seen rapid growth in the past few years.

If you want to learn GraphQL, we recommend the resources at graphql.org: graphql.org/

learn

Because GraphQL comes with a query language and types named in clear English terms
we’ve found it easier to teach beginners how to use GraphQL than REST.

18.1 Dispelling the Performance Myth

A lot of people, upon first learning about how GraphQL clients define the data they want
to receive, raise concerns about performance. The specific issue raised is how can one cache
the data raised by a query if each and every access request could be different?

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 231

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 18: Building GraphQL APIs With Django

GraphQL Method REST Method Action

query GetRecords() GET /records Read-only Query
subscription GetRecords() no equivalent Open websocket and up-
date on any changes
query GetRecord(id: X) GET /record/:id Read-only Query
subscription GetRecord(id: GET /record/:id Open websocket and up-
X) date on any changes
mutation CreateRecord() POST /records/ Create record
mutation UpdateRe- PUT /records/:id Update record
cord(id: X)
mutation DeleteRecord(id: DELETE /records/:id Delete record
X)

There are several things that destroy this myth:

1 By specifying only the data they need rather than accepting large REST-style re-
sponses, clients consume significantly less data. This can have a dramatic impact on
both server and database overhead.
2 Much like with a REST API, common acces patterns can still be identified and han-
dled better through caching, indexing, or code optimization.

Do keep in mind we’re not saying that GraphQL APIs are immune to bottlenecks, we’re
just saying that they are at least as manageable as REST APIs, if not more so due to the
lower bandwidth.

Our preferred library for building GraphQL APIs with Django is Ariadne. Here’s why:

ä Fully asynchronous when used with ASGI and Channels, allowed real-time updates
from Django projects via GraphQL Subscriptions.
ä If you aren’t ready for async yet, you can still run under WSGI.
ä Schemas comes first, meaning you are forced to spec out your design before starting to
code. Since GraphQL schemas carry so much weight, this is a very powerful feature.
ä Creating queries and mutations requires following a simple and lightweight API.
ä Supports Apollo Federation, a specification for composing multiple GraphQL ser-
vices into one GraphQL service.

PACKAGE TIP: What About Graphene?

Graphene was the first mainstram Python library to support the GraphQL spec-
ification, and it even comes with a Django integration. The ability to use Django
forms or serializers in nodes is a definate bonus. However, while we respect the ef-
fort put into it, we have found its shortcomings insurmountable for our own projects.

232 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 18.2: GraphQL API Architecture

Specifically:
ä Graphene-Django doesn’t support GraphQL Subscriptions, meaning to get
real-time updates one is forced to either poll the API from the client or use a
separate websocket process. Either is an unnacceptable burden and one that
we hope Django’s new Async Views will empower the library maintainers to
correct.
ä On other platforms, such as Apollo server, GraphQL APIs require little code
to implement. In comparison, to us Graphene feels heavyweight.
Reference: docs.graphene-python.org/projects/django/

18.2 GraphQL API Architecture

Building simple, quick GraphQL APIs is easy with Ariadne, but extending and maintaining
it to match your project’s needs takes a bit more thought. This is usually where people get
hung up on API design. This section illustrates some things to keep in mind.

18.2.1 Don’t Use Sequential Keys as Public Identifiers

Sequential keys, such as what Django provides as a default as model primary keys, can be
a security concern if used publicly. We cover this in-depth at Section 28.28: Never Display
Sequential Primary Keys.

In our example, we’re going to use the model’s UUID rather than the model’s primary key
to look up our records. We always try to avoid using sequential numbers for lookups.

18.2.2 Use Consistent API Module Naming

Just like anything else, how things are named needs to be consistent across a project. Here’s
two patterns we’ve explored.

The first is a simple GraphQL architecture suitable for most small-to-mid sized applications.
Because Ariadne’s API is lightweight, we’ve found it’s workable to write all queries and
mutations in one schemas.py module.

Example 18.1: Simple GraphQL Architecture

config/
├── schema.py # imports forms & models from flavors app
├── settings/
├── urls.py
flavors/

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 233

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 18: Building GraphQL APIs With Django

├── __init__.py
├── app.py
├── forms.py
├── models.py

Sometimes a single schemas.py module gets too big for comfort. In which case breaking it
up across apps has worked well. Queries and mutations are moved to individual apps, then
imported into the core schemas.py module where make_executable_schema() can be
called on it.

Example 18.2: Complex GraphQL Architecture

config/
├── schema.py # imports queries/mutations from flavors app
├── settings/
├── urls.py
flavors/
├── __init__.py
├── app.py
├── api/
│   ├── queries.py # imports models
│   ├── mutations.py # imports forms and models
├── forms.py
├── models.py

Please observe the following about both architectures:

ä We’re still leaning on Django forms and models

ä We stick to intuitive, consistent naming patterns.

18.2.3 Try to Keep Business Logic Out of API Views

Regardless if your API is small or large, it’s a good idea to keep your logic where it belongs.
Validation should be in forms (or DRF serializers if you prefer them) and database handling
should be in models.

It’s easy with Ariadne to start dumping business logic into function or class definitions.
Avoid this anti-pattern, it makes it harder to test, upgrade, or reuse logic when it comes
time to release APIv2.

234 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 18.3: Shutting Down an External API

18.2.4 Test Your API

We’ve found that the best way to test our Ariadne-powered GraphQL is to use
Django’s built-in RequestFactory system. Here is how Ariadne tests it’s Django integra-
tion: github.com/mirumee/ariadne/blob/master/tests/django/test_query_
execution.py

18.2.5 Version Your API

In progress

18.2.6 Be Careful With Customized Authentication Schemes

In progress

18.3 Shutting Down an External API

In progress

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 235

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 18: Building GraphQL APIs With Django

236 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

19 | JavaScript and Django

This chapter is about JavaScript in the context of Django. Here are the most common ways
JavaScript is used in tandem with Django:

ä To provide the entire frontend and consume REST/GraphQL APIs served by

Django. This is the Single Page App approach often executed with React, Vue, and
other frameworks.
ä To enhance templates written in either DTL or Jinja2 served by Django.
ä To make the Django admin useful and efficient. This is important to remember when
APIs created with Django are serving native apps, even then JavaScript is often still
part of projects.

This chapter will address the following topics:

ä Section 19.1: Popular JavaScript Approaches

ä Section 19.2: Consuming Django-served APIs with JavaScript
ä Section 19.3: Real-Time Woes a.k.a. Latency
ä Section 19.4: Using JavaScript with Templates Served by Django
ä ??: ??

19.1 Popular JavaScript Approaches

With the advent of faster JavaScript engines and a maturation of the associated community,
there has been a rise in new JavaScript frameworks that are designed for integration with
REST or GraphQL APIs. The three most popular ones in 2020 seem to be:

React.js facebook.github.io/react/
A JavaScript framework and ecosystem created and maintained by Facebook. De-
signed for creation of HTML, iOS, and Android applications. React is typically used
for Single Page Apps, but can be used to enhance existing pages.
Vue.js vuejs.org
Vue is designed to be incrementally adoptable. While its ecosystem isn’t as large as

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 237

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 19: JavaScript and Django

React.js, the simpler data model is considered easier to learn. In the Django world,
Vue is frequently used to enhance existing pages or for building of Single Page Apps
frontends.
Angular angular.io
Angular is a typescript-only framework open sourced by Google. Within the Django
community it is not as popular as React or Vue.

Reactisso2019.HTML+minimumJSis2020

– David Heinemeier Hansson, Founder of Rails, Basecamp, and Hey

In addition to these comprehensive Single Page Application frameworks the approach of

using serverside rendered HTML templates remains popular. While this might seem like a
step backwards, new approaches to this older architecure has demonstrated that it remains
a viable approach. Sites like hey.com, basecamp.com, and the forthcoming Writernaut
project demonstrate SPA-like speed and power using traditional HTML templates.

Vanilla JavaScript developer.mozilla.org/en-US/docs/Web/JavaScript

Vanilla JavaScript is plain JavaScript without any additional libraries like React,
Vue, or jQuery. There was a time when such libraries were absolutely necessary,
but JavaScript in the browser has become a powerful, feature-rich tool for building
projects.
Small, Focused Libraries + Vanilla JavaScript While Vanilla JavaScript’s interaction with
the DOM has grown in recent years, there are a number of useful libraries that can
empower developers and reduce code complexity. Examples include:
ä Alpine.js github.com/alpinejs/alpine is a rugged, minimal framework
for composing JavaScript behavior in markup. The syntax is Intentionally almost
identical to Vue, giving the power of Vue without the associated heavyweight
architecture.
ä Turbolinks github.com/turbolinks/turbolinks Uses HTML to render
views on the server side and link to pages as usual. When users follow a link,
Turbolinks automatically fetches the page, swaps in its <body>, and merges its
<head>, all without incurring the cost of a full page load. Getting it running
Django isn’t trivial, but can dramatically speed up traditional server side ren-
dered template sites.
jQuery jquery.com
Once the backbone of the JavaScript world, so much of functionality JQuery provides
has now been factored into Vanilla JavaScript it is rarely used in new projects. How-
ever, legions of existing Django projects use it extensively, and will continue to need
to be maintained or upgraded to other tools for years to come.

238 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 19.1: Popular JavaScript Approaches

These five options can really improve what we like to call the ‘immediate user experience’.
However, with every good thing there are always things to consider and things to do. Here
are a number of anti-patterns that we’ve discovered when it comes to frontend projects
consuming APIs for content.

19.1.1 Building Single Page Apps When Multi-Page Apps Suffice

Single-page apps with frameworks like React, Vue, and Angular are fun to build, but does a
traditional CMS-site need to be one? Certainly the content pages can include API-powered
editing controls, but when building this kind of site, there is something to be said for tradi-
tional HTML pages.

For example, our health provider in 2017 had a SPA-style site. It was lovely. The way every-
thing moved together was a marvel. And it was completely useless when we had to do any
kind of comparative research.

The worst example of the site was when the search system returned a list of doctors we
couldn’t easily compare them. When we clicked on one for more information, their data was
in a sliding modal. We couldn’t right-click and open several on independant tabs as doing
so just took us to the root search page. We could print or email yourself the information on
individual doctors, but PDFs and email are awful comparison tools compared to hopping
between tabs.

What the site should have provided is individual domain references (i.e. URLs) for each
doctor. Either parsed by the server on the back end or even by JavaScript URL management
on the front end. This isn’t hard to do, yet it remained a painfully common issue until we
went to another healthcare provider.

19.1.2 Upgrading Legacy Sites

Unless the entire site is being scrapped for a new version, don’t upgrade the whole front-end
at once.

When working with legacy projects, it’s often easier to add new features as single-page
apps. This allows for the maintainers of the project to deliver improved experiences with
new features, while preserving the stability of the existing code base. A good example of
this might be adding a calendar application to an existing project. This is easily possible
with Vue and to a lesser extent with React.

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 239

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 19: JavaScript and Django

19.1.3 Not Writing Tests

When we first begin working in a new language or framework, including client-side
JavaScript, it’s tempting to skip the tests. In a word, don’t. Working in the client is get-
ting more complicated and sophisticated every year. Between evolving client-side standards,
things are simply not as readable there as on the server side.

We cover Django/Python testing in Chapter 24: Testing Stinks and Is a Waste of Money!.

A good reference for JavaScript testing is stackoverflow.com/questions/300855/
javascript-unit-test-tools-for-tdd

19.1.4 Not Understanding JavaScript Memory Management

Single-page apps are great, but the complex implementations where users keep them open
constantly will hold objects in the browser for a very long time. Eventually, if not managed,
this can cause browser slowdowns and crashes. Each JavaScript framework comes with tools
or advice on how to handle this potential problem, and it’s a good idea to know the recom-
mended approach.

19.1.5 Storing Data in the DOM When It’s Not jQuery

After years of using jQuery, some of us have grown used to using DOM elements to store
data (especially Daniel). However, when using other JavaScript frameworks this isn’t ideal.
They have their own mechanisms for handling client data, and by not following them we
risk losing out on some of the features promised by these frameworks.

We recommend looking up the data management methods for your chosen JavaScript frame-
work and embracing them as deeply as possible.

19.2 Consuming Django-served APIs with JavaScript

Now that we’ve covered both creating REST APIs and template best practices, let’s combine
them. In other words, these are best practices for using Django-powered tools to display
content to the end user in the browser using content managed by REST/GraphQL APIs
and presented by modern JavaScript frameworks.

19.2.1 Learn How to Debug the Client

Debugging client-side JavaScript is a lot more than simply writing console.log() and
console.dir() statements. There are a number of tools for debugging and finding errors,
and some of them are specifically written for particular JavaScript frameworks. Once a tool
is chosen, it’s an excellent idea to take a day to learn how to write client-side tests.

Reference material:

240 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 19.2: Consuming Django-served APIs with JavaScript

ä developers.google.com/web/tools/chrome-devtools
ä developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_
JavaScript

Figure 19.1: Server-side vs. client-side ice cream.

19.2.2 When Possible, Use JavaScript-Powered Static Asset Preproces-

sors
Up until about 2017, we used Python everywhere, including JavaScript and CSS minifica-
tion. However, the JavaScript community is maintaining their versions of these tools better
than the Python community. That’s perfectly okay, because since they’ve done the work on
this part of the toolchain, we can focus on other things.

As we write this, the most commonly used tool for this kind of work is webpack. Used by
both React, Vue, and every major framework, webpack bundles browser-based scripts to be
served as assets. While we prefer to use the stock webpack setup provided by Vue or React
CLI tools, understanding webpack is a very powerful skill to have.

References:

ä webpack.js.org
ä github.com/owais/django-webpack-loader - Owais Lone’s Django package
for transparently using webpack with Django

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 241

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 19: JavaScript and Django

19.3 Real-Time Woes a.k.a. Latency

Let’s say we’ve put together a well-designed, well-indexed, well-cached real-time project
with the widest bandwidth piping content to the world. We can handle any load, and our
test users applaud the speed and functionality of the project. Things look great, and we look
forward to bonuses and raises.

Then the complaints from the other side of the planet start coming in about the slow speed
of the application. Our effort isn’t ‘real-time’ to any of a potentially large block of users and
our client/boss is really unhappy.

Congratulations, we’ve just hit the speed of light!

This isn’t a joke, it’s a very real problem. Here, Django isn’t the problem. Instead, it’s physics.
The time it takes for HTTP requests to transmit back and forth across half the circumference
of the planet is noticeable to human beings. Add in server-side and client-side processing,
and we risk alienating potential or existing users.

Also, keep in mind that even the fastest local connections have hiccups and slow-downs. So
it’s not uncommon for ‘real-time’ applications to have ways to handle this sort of behavior.

19.3.1 Solution: Mask the Latency With Animations

One of the more common fixes is to have JavaScript-powered animation distract the user
from latency issues. We encounter this every time we use a single page app with an attractive
interface, including all modern web-based email clients.

19.3.2 Solution: Fake Successful Transactions

Another solution involves processing the request on the client-side as if the request success-
fully made it to the server. We’ll need to include client-side logic to handle failures, but
JavaScript frameworks handling HTTP requests are asynchronous, making this feasible,
albeit possibly complicated.

If you’ve ever suddenly discovered that your cloud-based spreadsheet hadn’t saved the data
entered for the past 30 seconds, you’ve uncovered this kind of JavaScript powered trickery
in action. As this can be very frustrating, some online tools upon detecting a connection
failure, disallow further data entry.

19.3.3 Solution: Geographically Based Servers

Geographically-based servers across all seven continents is an option. However, for Django
this is not trivial to implement, not at the programming or database level. It requires a

242 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 19.3: Real-Time Woes a.k.a. Latency

significant volume of skills and expertise that’s outside the scope of this book.

If you have the time and budget, this can be an exciting avenue to explore and we encour-
age it. However, unless you’ve done this before there is a good chance you are going to
underestimate the effort involved.

19.3.4 Solution: Restrict Users Geographically

Sometimes we just don’t have a choice. Perhaps our application is too reliant on ‘real-time’
performance and geolocating servers might be outside the budget. We might make some
people unhappy, but that can be mitigated to some degree by saying things like, ‘Support in
your country is coming soon!’

19.3.5 AJAX and the CSRF Token

Django’s CSRF protection appears to be an inconvenience when writing AJAX calls from
the browser to the server. If you use AJAX with Django, especially with REST APIs,
you may discover that triggering the CSRF token validation blocks your ability to POST,
PATCH, or DELETE data to your API. However, it’s part of what makes Django secure,
don’t disable it!

Our answer to overcoming this hurdle to read and understand the AJAX portion of the
CSRF documentation of Django, listed in the resources below. It comes with sample
JavaScript code that you can put into your projects.

References:

ä docs.djangoproject.com/en/3.2/ref/csrf/
ä github.com/jazzband/dj-rest-auth This is a proven authentication library for
DRF that depending on circumstances can be used without CSRF.
ä ??: ??

WARNING: Don’t Use AJAX as an Excuse to Turn Off CSRF

Django core developer Aymeric Augustin says, “...CSRF protection is almost always
disabled because the developers couldn’t quite figure out how to make it work... if
it accepts cookie authentication.”
Unless using planning to build a machine-to-machine API, don’t build a site with
disabled CSRF. If it’s not possible to figure out how to make it work, ask for help.
No one’s going to make fun of someone trying to make their site more secure.

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 243

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 19: JavaScript and Django

19.4 Using JavaScript with Templates Served by Django

19.4.1 JavaScript Can Go In the Header Again
For years it was advantageous to put links to scripts and <script> tags at the bottom of a
page body. Modern browsers are much better at parsing how pages load, so this technique
is no longer necessary.

19.4.2 Use JSON Encoding for Data Consumed by JavaScript

Rely on JSON encoding rather than finding ways to dump Python structures directly to
templates. It’s not just easier to integrate into client-side JavaScript, it’s safer. To do this,
always use the json_script filter.

Example 19.1: Properly JSON encoding Data for JavaScript

{{ page_data|json_script:"page-data" }}
<script>
var data = JSON.parse(document.getElementById('page').textContent);
injectNameIntoDiv('scoopName', data.scoop.name);
</script>

Don’t do this, which must we admit to having done for years.

Example 19.2: Improperly JSON encoding Data for JavaScript

// Rendering a value from the Django template

// Hoping no one injected anything malignant
injectNameIntoDiv('scoopName', {{ scoop.name }});

Reference: docs.djangoproject.com/en/3.2/ref/templates/builtins/
#json-script

19.5 Strengthening JavaScript Skills

One of the best things we can do when implementing the consumption of REST APIs
on the client side is to ensure our JavaScript skills are up to par. While Python developers
sometimes like to grumble about JavaScript, it is a very capable language in its own right.
Any responsible web developer will take the time to ramp up their skills so they can reap
the benefits of modern JavaScript.

TODO

244 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 19.6: Follow JavaScript Coding Standards

19.5.1 Learn More JavaScript!

There are plenty of resources available for improving your basic JavaScript skills. We list our
favorites at the end of Appendix C: Additional Resources.

19.6 Follow JavaScript Coding Standards

In the case of JavaScript, we advocate the following guides for both front- and back-end
work:

ä Felix’s Node.js Style Guide

nodeguide.com/style.html
ä idiomatic.js
github.com/rwaldron/idiomatic.js

19.7 Summary
TODO

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 245

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 19: JavaScript and Django

246 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

20 | Tradeoffs of Replacing
Core Components

Some people advocate swapping out core parts of Django’s stack for other pieces. Should it
be done?

Short Answer: Don’t do it. Even one of the founders of Instagram (Kevin Systrom) said
to Forbes.com that it’s completely unnecessary (bit.ly/2pZxOBO).
Long Answer: It’s certainly possible, since Django modules are simply just Python mod-
ules. Is it worth it? Well, it’s worth it only if:
ä You are okay with sacrificing some or all of your ability to use third-party Django
packages.
ä You have no problem giving up the powerful Django admin.
ä You have already made a determined effort to build your project with core
Django components, but you are running into walls that are major blockers.
ä You have already analyzed your own code to find and fix the root causes of your
problems. For example, you’ve done all the work you can to reduce the numbers
of queries made in your templates.
ä You’ve explored all other options including caching, denormalization, etc.
ä Your project is a real, live production site with tons of users. In other words,
you’re certain that you’re not just optimizing prematurely.
ä You’ve looked at and rejected adopting a Service Oriented Approach (SOA) for
those cases Django has problems dealing with.
ä You’re willing to accept the fact that upgrading Django will be extremely painful
or impossible going forward.

That doesn’t sound so great anymore, does it?

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 247

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 20: Tradeoffs of Replacing Core Components

20.1 The Temptation to Build FrankenDjango

Every few years, a new fad leads waves of developers to replace some particular core Django
component. Here’s a summary of some of the fads we’ve seen come and go.

Figure 20.1: Replacing more core components of cake with ice cream seems like a good idea.
Which cake would win? The one on the right!

20.2 Non-Relational Databases vs. Relational

Databases
Even Django projects that use relational databases for persistent data storage rely on non-
relational databases. If a project relies on tools like Memcached for caching and Redis for
queuing, then it’s using non-relational databases.

The problem occurs when NoSQL solutions are used to completely replace Django’s rela-
tional database functionality without considering in-depth the long-term implications.

20.2.1 Not All Non-Relational Databases Are ACID Compliant

ACID is an acronym for:

Atomicity means that all parts of a transaction work or it all fails. Without this, you risk
data corruption.
Consistency means that any transaction will keep data in a valid state. Strings remain
strings and integers remain integers. Without this, you risk data corruption.
Isolation means that concurrent execution of data within a transaction will not collide or
leak into another transaction. Without this, you risk data corruption.
Durability means that once a transaction is committed, it will remain so even if the
database server is shut down. Without this, you risk data corruption.

248 Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322

 20.2: Non-Relational Databases vs. Relational
Databases

Fad Reasons
For performance reasons, replacing the Not okay: “I have an idea for a social net-
database/ORM with a NoSQL database work for ice cream haters. I just started
and corresponding ORM replacement. building it last month. It must scale to bil-
lions!

Okay: “Our site has 50M users and I’m

hitting the limits of what I can do with
indexes, query optimization, caching, etc.
We’re also pushing the limits of our Post-
gres cluster. I’ve done a lot of research on
this and am going to try storing a simple
denormalized view of data in Cassandra to
see if it helps. I’m aware of the CAP the-
orem (en.wikipedia.org/wiki/CAP_
theorem), and for this view, eventual con-
sistency is fine.”
For data processing reasons, replacing the Not okay: “SQL Sucks! We’re going with
database/ORM with a NoSQL database a document-oriented database like Mon-
and corresponding ORM replacement. goDB!”

Okay: “PostgreSQL’s and MySQL’s

JSON datatypes replicate nearly every
aspect of MongoDB’s data storage system.
Yes, MongoDB’s has built-in MapReduce
functionality, but it’s easier to execute that
in a task queue.”
Replacing Django’s template engine with Not okay: “I read that Jinja2 is faster. I
Jinja2, Mako, or something else. don’t know anything about caching or op-
timization, but I need Jinja2!”

Not okay: “I hate having logic in Python

modules. I want logic in my templates!”

Okay: “I have a small number of views

which generate 1MB+ HTML pages de-
signed for Google to index. I’ll use
Django’s native support for multiple tem-
plate languages to render the 1MB+ sized
pages with Jinja2, and serve the rest with
Django Template Language.”

Table 20.1: Fad-based Reasons to Replace Components of Django

Please submit issues to github.com/feldroy/two- scoops- of- django- 3.x/issues 249

Prepared exclusively for LAKLAK LAKLAK ([email protected]) Transaction: 11322
Chapter 20: Tradeoffs of Replacing Core Components

Did you notice how each of those descriptions ended with ‘Without this, you risk data corrup-
tion.’? This is because for many NoSQL engines, there is little-to-no mechanism for ACID
compliance. It’s much easier to corrupt the data, which is mostly a non-issue for things like
caching but another thing altogether for projects handling processing of persistent medical
or e-commerce data.

20.2.2 Don’t Use Non-Relational Databases for Relational Tasks

Imagine if we were to use a non-relational database to track the sale of properties, prop-
erty owners, and how property laws worked for them in 50 US states. There are a lot of
unpredictable details, so wouldn’t a schemaless datastore be perfect for this task?

Perhaps...

We would need to track the relationship between properties, property owners, and laws of
50 states. Our Python code would have to maintain the referential integrity between all the
components. We would also need to ensure that the right data goes into the right place.

For a task like this, stick with a relational database.

20.2.3 Ignore the Hype and Do Your Own Research

It’s often said that non-relational databases are faster and scale better than relational
databases. Whether or not this is true, don’t blindly swallow the marketing hype of the
companies behind any particular alternative database solution.

Instead, do as we do: search for benchmarks, read case studies describing when things went
right or wrong, and form opinions as independently as possible.

Also, experiment with unfamiliar NoSQL databases on small hobby side projects before
you make major changes to your main project infrastructure. Your main codebase is not a
playground.

Lessons learned by companies and individuals:

ä Pinterest: medium.com/@Pinterest_Engineering/
stop-using-shiny-3e1613c2ce14
ä Dan McKinley while at Etsy: mcfunley.com/
why-mongodb-never-worked-out-at-etsy
ä When to use MongoDB with Django daniel.feldroy.com/
when-to-use-mongodb-with-django.html

250 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 20.3: What About Replacing the Django Template Language?

20.2.4 How We Use Non-Relational Databases With Django

This is how we prefer to do things:

ä If we use a non-relational data store, limit usage to short-term things like caches,
queues, and sometimes denormalized data. But avoid it if possible, to reduce the
number of moving parts.
ä Use relational data stores for long-term, relational data and sometimes denormalized
data (PostgreSQL’s array and JSON fields work great for this task).

For us, this is the sweet spot that makes our Django projects shine.

20.3 What About Replacing the Django Template

Language?
We advocate the practice of sticking entirely to the Django Template Language (DTL) with
the exception of rendered content of huge size. However, as this use case is now covered by
Django’s native support of alternate template systems, we’ve moved discussion of this topic
to Chapter 16: Django Templates and Jinja2.

20.4 Summary
Always use the right tool for the right job. We prefer to go with stock Django components,
just like we prefer using a scoop when serving ice cream. However, there are times when
other tools make sense.

Just don’t follow the fad of mixing vegetables into your ice cream. You simply can’t replace
the classic strawberry, chocolate, and vanilla with supposedly “high-performance” flavors
such as broccoli, corn, and spinach. That’s taking it too far.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 251
Chapter 20: Tradeoffs of Replacing Core Components

252 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
21 | Working With the Django Admin

When people ask, “What are the benefits of Django over other web frameworks?” the admin is
what usually comes to mind.

Imagine if every gallon of ice cream came with an admin interface. You’d be able to not
just see the list of ingredients, but also add/edit/delete ingredients. If someone was messing
around with your ice cream in a way that you didn’t like, you could limit or revoke their
access.

Figure 21.1: Chocolate chip ice cream with an admin interface.

Pretty surreal, isn’t it? Well, that’s what web developers coming from another background
feel like when they first use the Django admin interface. It gives you so much power over
your web application automatically, with little work required.

21.1 It’s Not for End Users

The Django admin interface is designed for site administrators, not end users. It’s a place
for your site administrators to add/edit/delete data and perform site management tasks.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 253
Chapter 21: Working With the Django Admin

Although it’s possible to stretch it into something that your end users could use, you really
shouldn’t. It’s just not designed for use by every site visitor.

21.2 Admin Customization vs. New Views

It’s usually not worth it to heavily customize the Django admin. Sometimes, creating a
simple view or form from scratch results in the same desired functionality with a lot less
work. We’ve always had better results with creating custom management dashboards for
client projects than we have with modifying the admin to fit the need of the client.

21.3 Viewing String Representations of Objects

The default admin page for a Django app shows a list of generic looking objects like this:

Figure 21.2: Admin list page for an ice cream bar app.

That’s because the default string representation of an IceCreamBar object is “IceCreamBar

object”. Wouldn’t it be helpful to display something better?

Figure 21.3: What? An admin interface for ice cream bars?

254 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 21.3: Viewing String Representations of Objects

21.3.1 Using __str__()

Implementing __str__() is straightforward:

Example 21.1: String Representation of Objects

from django.db import modelss

class IceCreamBar(models.Model):
name = models.CharField(max_length=100)
shell = models.CharField(max_length=100)
filling = models.CharField(max_length=100)
has_stick = models.BooleanField(default=True)

def __str__(self):
return self.name

The result is as follows:

Figure 21.4: Improved admin list page with better string representation of our objects.

It’s more than that. When you’re in the shell, you see the better string representation:

Example 21.2: List of Ice Cream Bar Types

>>> IceCreamBar.objects.all()
[<IceCreamBar: Vanilla Crisp>, <IceCreamBar: Mint Cookie Crunch>,
<IceCreamBar: Strawberry Pie>]

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 255
Chapter 21: Working With the Django Admin

The __str__() method is called whenever you call str() on an object. This occurs in the
Django shell, templates, and by extension the Django admin. Therefore, try to make the
results of __str__() nice, readable representation of Django model instances.

21.3.2 Using list_display

If you want to change the admin list display in a way that isn’t quite a string representation
of the object, then use list_display.

Example 21.3: Admin List Display

from django.contrib import admin

from .models import IceCreamBar

@admin.register(IceCreamBar)
class IceCreamBarModelAdmin(admin.ModelAdmin):
list_display = ('name', 'shell', 'filling')

The result with the specified fields:

Figure 21.5: Further improvements to the admin list page.

21.4 Adding Callables to ModelAdmin Classes

You can use callables such as methods and functions to add functionality to the Django
django.contrib.admin.ModelAdmin class. This allows you to really modify the list
and display screens to suit your ice cream project needs.

256 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 21.4: Adding Callables to ModelAdmin Classes

For example, it’s not uncommon to want to see the exact URL of a model instance in
the Django admin. If you define a get_absolute_url() method for your model, what
Django provides in the admin is a link to a redirect view whose URL is very different from
the actual object URL. Also, there are cases where the get_absolute_url() method is
meaningless (REST APIs come to mind).

In the example below, we demonstrate how to use a simple callable to provide a link to our
target URL:

Example 21.4: Adding Callables to ModelAdmin Classes

# icecreambars/admin.py
from django.contrib import admin
from django.urls import reverse, NoReverseMatch
from django.utils.html import format_html

from .models import IceCreamBar

@admin.register(IceCreamBar)
class IceCreamBarModelAdmin(admin.ModelAdmin):
list_display = ('name', 'shell', 'filling')
readonly_fields = ('show_url',)

def show_url(self, instance):

url = reverse('icecreambars:ice_cream_bar_detail',
,→ kwargs={'pk': instance.pk})
response = format_html("""<a href="{0}">{0}</a>""", url)
return response

show_url.short_description = 'Ice Cream Bar URL'

Since a picture is worth a thousand words, here is what our callable does for us:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 257
Chapter 21: Working With the Django Admin

Figure 21.6: Displaying URL in the Django Admin.

WARNING: Always Use format_html When Presenting Data

From Users
Any time data is presented in the admin that includes HTML tags, use
format_html to ensure that arguments are escaped during the rendering process.
This is explained more at Section 28.9.1: Use format_html Over mark_safe.

21.5 Be Aware of the Complications of Multiuser Environ-

ments
Nothing in the Django admin locks records to a particular staff- or admin-level user. While
this is fine for a project with a single person with admin-level access, on a multi-user project
it can be a very serious problem. Here is what happens:

1 Via the Django admin, Daniel edits the record for “Peppermint Sundae” ice cream
bar. He starts to make changes. He gets a phone call from the marketing officer of
Icecreamlandia and leaves his screen open.
2 In the meantime, Audrey decides to modify “Peppermint Sundae” ice cream bar. She
spends ten minutes making her changes, then saves the her data.
3 Daniel gets off the phone and finally saves his changes. He overwrites Audrey’s
changes.

If you have multiple users with access to the Django admin, you need to be aware of this

258 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 21.6: Django’s Admin Documentation Generator

possibility.

21.6 Django’s Admin Documentation Generator

One of the more interesting developer tools that Django provides is the
django.contrib.admindocs package. Created in an era before the advent of the
documentation tools that we cover in Chapter 25: Documentation: Be Obsessed, it
remains a useful tool.

It’s useful because it introspects the Django framework to display docstrings for project
components like models, views, custom template tags, and custom filters. Even if a project’s
components don’t contain any docstrings, simply seeing a list of harder-to-introspect items
like oddly named custom template tags and custom filters can be really useful in exploring
the architecture of a complicated, existing application.

Using django.contrib.admindocs is easy, but we like to reorder the steps described in

the formal documentation:

1 pip install docutils into your project’s virtualenv.

2 Add django.contrib.admindocs to your INSTALLED_APPS.
3 Add path('admin/doc/', include('django.contrib.admindocs.urls'))
to your root URLConf. Make sure it’s included before the admin/ entry, so that
requests to /admin/doc/ don’t get handled by the latter entry.
4 Optional: Using the admindocs bookmarklets requires the XViewMiddleware to be
installed.

Once you have this in place, go to /admin/doc/ and explore. You may notice a lot of
your project’s code lacks any sort of documentation. This is addressed in the formal doc-
umentation on django.contrib.admindocs: docs.djangoproject.com/en/3.2/
ref/contrib/admin/admindocs/ and our own chapter on Chapter 25: Documenta-
tion: Be Obsessed.

21.7 Using Custom Skins With the Django Admin

Over the years there have been a number of efforts to reskin or theme the Django Admin.
These range from the venerable, stable, and very popular django-grappelli to more recent
up-and-comers. They allow easy-to-hard customization.

PACKAGE TIP: Custom django.contrib.admin Skins

Here are some of the more popular custom skins:
ä django-grappelli is the grand-daddy of all custom Django skins. Stable, ro-
bust, and with a unique but friendly style.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 259
Chapter 21: Working With the Django Admin

ä django-suit is built using the familiar Bootstrap front-end framework.

ä jet-admin is an API-based Admin Panel Framework.
A more complete list can be found at
djangopackages.org/grids/g/admin-styling/.

Django has a gigantic community, so why aren’t there more skins?

It turns out that besides the most basic CSS-based modifications, creating custom Django
themes is very challenging. For anyone who has delved into the source code for these
projects, it’s clear that custom admin skins require arcane code to account for some of the
idiosyncrasies of django.contrib.admin.

Patrick Kranzlmueller, maintainer of django-grappelli, goes into great detail in his article
on the subject, ‘A Frontend Framework for the Django Admin Interface’, which you can
read at the link below:

ä sehmaschine.net/blog/django-admin-frontend-framework.

Here are some tips when working with custom django.contrib.admin skins:

21.7.1 Evaluation Point: Documentation is Everything

As mentioned earlier, writing a custom skin for django.contrib.admin is hard. While
the successful skins are relatively easy to add to a project, it’s the edge cases (invariably
involved in extending the ModelAdmin object) that can hurt.

Therefore, when evaluating one of these projects for use on a project, check to see how far
the documentation goes beyond installation instructions.

21.7.2 Write Tests for Any Admin Extensions You Create

For our purposes, we’ve found that while clients enjoy the more modern themes, you
have to be careful of how far you extend these admin skins. What works great in vanilla
django.contrib.admin can break in a custom skin. Since the custom skins have to wrap
portions of django.contrib.admin abstractions in curious ways, debugging these prob-
lems can prove to be a mind-numbing nightmare.

Therefore, if you use a custom skin, the best practice is to write tests of the admin, especially
for any customization. Yes, it is a bit of work up front, but it means catching these bugs much,
much earlier.

260 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 21.8: Secure the Django Admin

For more on testing, see our writings on testing in Chapter 24: Testing Stinks and Is a
Waste of Money!.

21.8 Secure the Django Admin

Since the Django admin gives your site admins special powers that ordinary users don’t have,
it’s good practice to make it extra secure.

21.8.1 Change the Default Admin URL

By default, the admin URL is yoursite.com/admin/. Change it to something that’s long and
difficult to guess.

TIP: Jacob Kaplan-Moss Talks About Changing the Admin URL

Django project co-leader Jacob Kaplan-Moss says (paraphrased) that it’s an easy ad-
ditional layer of security to come up with a different name (or even different domain)
for the admin.

It also prevents attackers from easily profiling your site. For example, attackers can
tell which version of Django you’re using, sometimes down to the point-release level,
by examining the HTML content of the login screen for admin/ .

21.8.2 Use django-admin-honeypot

TODO - Ask Derek to update this package

If you’re particularly concerned about people trying to break into your Django site, django-
admin-honeypot is a package that puts a fake Django admin login screen at admin/ and
logs information about anyone who attempts to log in.

See github.com/dmpayton/django-admin-honeypot for more information.

21.8.3 Only Allow Admin Access via HTTPS

This is already implied in Section 28.6: HTTPS Everywhere, but we want to especially em-
phasize here that your admin needs to be TLS-secured. If your site allows straight HTTP
access, you will need to run the admin on a properly-secured domain, adding to the complex-
ity of your deployment. Not only will you need a second deployment procedure, but you’ll
need to include logic in your URLConf in order to remove the admin from HTTP access.
In the experience of the authors, it’s much easier to put the whole site on TLS/HTTPS.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 261
Chapter 21: Working With the Django Admin

Without TLS, if you log into your Django admin on an open WiFi network, it’s trivial for
someone to sniff your admin username/password.

21.8.4 Limit Admin Access Based on IP

Configure your web server to only allow access to the Django admin to certain IP addresses.
Look up the instructions for your particular web server.

ä Nginx instructions tech.marksblogg.com/django-admin-logins.html

An acceptable alternative is to put this logic into middleware. It’s better to do it at the
web server level because every middleware component adds an extra layer of logic wrapping
your views, but in some cases this can be your only option. For example, your platform-as-
a-service might not give you fine-grain control over web server configuration.

21.9 Securing the Admin Docs

Since the Django admin docs give your site admins a view into how the project is con-
structed, it’s good practice to keep them extra-secure just like the Django admin. Borrowing
from the previous section on the Django admin, we advocate the following:

ä Changing the admin docs URL to something besides yoursite.com/admin/doc/.

ä Only allowing admin docs access via HTTPS.
ä Limiting admin docs access based on IP.

21.10 Summary
In this chapter we covered the following:

ä Who should be using the Django admin.

ä When to use the Django admin and when to roll a new dashboard.
ä String representation of objects.
ä Adding callables to Django admin classes.
ä Using Django’s admin docs.
ä Encouraging you to secure the Django admin.
ä Advised on working with custom Django skins.

262 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
22 | Dealing With the User Model

Django comes with a built-in support for user records. It’s a useful feature, doubly so once
you learn how to extend and expand on the basic functionality. So let’s go over the best
practices:

22.1 Use Django’s Tools for Finding the User Model

The advised way to get to the user class is as follows:

Example 22.1: Using get_user_model to the User Record

# Stock user model definition

>>> from django.contrib.auth import get_user_model
>>> get_user_model()
<class django.contrib.auth.models.User>

# When the project has a custom user model definition

>>> from django.contrib.auth import get_user_model
>>> get_user_model()
<class profiles.models.UserProfile>

It is possible to get two different User model definitions depending on the project configu-
ration. This doesn’t mean that a project can have two different User models; it means that
every project can customize its own User model.

22.1.1 Use settings.AUTH_USER_MODEL for Foreign Keys to User

In Django, the official preferred way to attach ForeignKey, OneToOneField, or
ManyToManyField to User is as follows:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 263
Chapter 22: Dealing With the User Model

Example 22.2: Using settings.AUTH_USER_MODEL to Define Model Rela-

tions

from django.conf import settings

from django.db import models

class IceCreamStore(models.Model):

owner = models.OneToOneField(settings.AUTH_USER_MODEL)
title = models.CharField(max_length=255)

Yes, it looks a bit strange, but that’s what the official Django docs advise.

Figure 22.1: This looks strange too.

WARNING: Don’t Change settings.AUTH_USER_MODEL!

Once set in a project, changing settings.AUTH_USER_MODEL requires changing
your database schema accordingly. It’s one thing to add or modify User model fields,
it’s another thing to create a whole new User object.

22.1.2 Don’t Use get_user_model() for Foreign Keys to User

This is bad, as it tends to create import loops.

Example 22.3: Using get_user_model() Improperly

# DON'T DO THIS!
from django.contrib.auth import get_user_model
from django.db import models

class IceCreamStore(models.Model):

264 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 22.2: Custom User Fields for Django Projects

# This following line tends to create import loops.

owner = models.OneToOneField(get_user_model())
title = models.CharField(max_length=255)

22.2 Custom User Fields for Django Projects

In Django, as long as we incorporate the required methods and attributes, we can create our
own user model with its own fields.

PACKAGE TIP: Libraries for Defining Custom User Models

django-authtools is a library that makes defining custom user models easier. Of
particular use are the AbstractEmailUser and AbstractNamedUser models.
Even if you don’t end up using django-authtools, the source code is well worth
examining.
At this time django-authtools doesn’t support Django 3, even though there is a pull
request adding it. Nevertheless, as mentioned before the code is worth a look.

22.2.1 Option 1: Subclass AbstractUser

Choose this option if you like Django’s User model fields the way they are, but need extra
fields. For what it’s worth, this is the first approach that we look at anytime we start a new
project. When using django-authtools’ base models, forms, and admin objects, we find that
it’s the quickest and easiest way to implement custom user models.

Here’s an example of how to subclass AbstractUser:

Example 22.4: Subclassing of AbstractUser

# profiles/models.py
from django.contrib.auth.models import AbstractUser
from django.db import models

class KarmaUser(AbstractUser):
karma = models.PositiveIntegerField(verbose_name='karma',
default=0,
blank=True)

The other thing you have to do is set this in your settings:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 265
Chapter 22: Dealing With the User Model

Example 22.5: Setting AUTH_USER_MODEL

AUTH_USER_MODEL = 'profiles.KarmaUser'

22.2.2 Option 2: Subclass AbstractBaseUser

AbstractBaseUser is the bare-bones option with only 3 fields: password, last_login,
and is_active.

Choose this option if:

ä You’re unhappy with the fields that the User model provides by default, such as
first_name and last_name.
ä You prefer to subclass from an extremely bare-bones clean slate but want to take
advantage of the AbstractBaseUser sane default approach to storing passwords.

If you want to go down this path, we recommend the following reading:

Official Django Documentation Example

docs.djangoproject.com/en/3.2/topics/auth/customizing/
#a-full-example
Source code of django-authtools (Especially admin.py, forms.py, and models.py)
github.com/fusionbox/django-authtools

22.2.3 Option 3: Linking Back From a Related Model

This code is very similar to the pre-Django 1.5 project technique of creating ‘Profile’ models.
Before discarding this approach as legacy, consider the following use cases:

Use Case: Creating a Third Party Package

ä We are creating a third-party package for publication on PyPI.

ä The package needs to store additional information per user, perhaps a Stripe ID or
another payment gateway identifier.
ä We want to be as unobtrusive to the existing project code as possible. Loose coupling!

Use Case: Internal Project Needs

ä We are working on our own Django project.

ä We want different types of users to have different fields.
ä We might have some users with a combination of different user types.

266 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 22.2: Custom User Fields for Django Projects

ä We want to handle this at the model level, instead of at other levels.

ä We want this to be used in conjunction with a custom user model from options #1 or
#2.

Either of these use cases provide motive for the continued use of this technique.

To make this technique work, we continue to use django.contrib.models.User (called

preferably via django.contrib.auth.get_user_model()) and keep your related fields
in separate models (e.g. Profiles). Here’s an example:

Example 22.6: Custom User Profile Examples

# profiles/models.py

from django.conf import settings

from django.db import models

from flavors.models import Flavor

class EaterProfile(models.Model):

# Default user profile

# If you do this you need to either have a post_save signal or
# redirect to a profile_edit view on initial login.
user = models.OneToOneField(settings.AUTH_USER_MODEL)
favorite_ice_cream = models.ForeignKey(Flavor, null=True,
,→ blank=True)

class ScooperProfile(models.Model):

user = models.OneToOneField(settings.AUTH_USER_MODEL)
scoops_scooped = models.IntegerField(default=0)

class InventorProfile(models.Model):

user = models.OneToOneField(settings.AUTH_USER_MODEL)
flavors_invented = models.ManyToManyField(Flavor, null=True,
,→ blank=True)

Using this approach, we can query for any user’s favorite ice cream trivially with the ORM:
user.eaterprofile.favorite_ice_cream. In addition, Scooper and Inventor

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 267
Chapter 22: Dealing With the User Model

profiles provide individual data that only applies to those users. Since that data is isolated
into dedicated models, it’s much harder for accidents between user types to occur.

The only downside to this approach is that it’s possible to take it too far in complexity of
profiles or in the supporting code. As always, keep your code as simple and clean as possible.

WARNING: Third-Party Libraries Should Not Be Defining the

User Model
Unless the express purpose of the library is to define custom user models for a project
(à la django-authtools), third-party libraries shouldn’t be using options #1 or #2 to
add fields to user models. Instead, they should rely on option #3.

22.3 Handling Multiple User Types

A common question from Django beginners is how to handle multiple user types. For ex-
ample with ice cream stores, dealing with customer, employee, and owner user types. Often
they think about defining two different user models, which is a dangerous approach to take
- Django isn’t designed to work that way.

Instead, just use one model and mark it appropriately. There’s two primary ways to do this:

22.3.1 Add a User Type Field

This approach assumes different roles have the same data and methods available to them.
In the user model, add a choices field distingishing between types of users. This allows for
role checks across a Django project.

Example 22.7: A Choices-Based User Type Field

class User(AbstractUser):
class Types(models.TextChoices):
EATER = "EATER", "Eater"
SCOOPER = "SCOOPER", "Scooper"
INVENTOR = "INVENTOR", "Inventor"

# What type of user are we?

type = models.CharField(
_("Type"), max_length=50, choices=Types.choices,
,→ default=Types.EATER
)

268 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 22.3: Handling Multiple User Types

Please note that we don’t rely on the multiple use of BooleanField to control roles. That
approach works only in limited circumstances, and becomes confusing as a site grows in size
and complexity over time. Better to use a single field to control the role of a User.

Furthermore, the approach we’re using relies on a CharField. In practice, we tend to use
ForeignKey relations to a Role model. If users need multiple roles, then we will either make
that a ManyToManyField or lean on Django’s built-in Group system. The disadvantage of
using Django’s built-in Group system is the minimal documentation for it.

22.3.2 Add a User Type Field Plus Proxy Models

It is typical for different types of users to have different methods and properties. For example,
a SCOOPER would have a scoop_icecream() method and an EATER would have a
consume() method.

Let’s also make querying for types of user as obvious and explicit as possible. It is easy to
forget to include a filter on a type of user, granting them access to things they should not
have.

Proxy models makes implementing this easy. First, in our model, add a base_type property
and extend the built-in save() method:

Example 22.8: ”The base_type Property”

class User(AbstractUser):
class Types(models.TextChoices):
EATER = "EATER", "Eater"
SCOOPER = "SCOOPER", "Scooper"
INVENTOR = "INVENTOR", "Inventor"

# Ensures that creating new users through proxy models works

base_type = Types.EATER

# What type of user are we?

type = models.CharField(
_("Type"), max_length=50,
choices=Types.choices,
default=Types.EATER
)

# ...

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 269
Chapter 22: Dealing With the User Model

def save(self, *args, **kwargs):

# If a new user, set the user's type based off the
# base_type property
if not self.pk:
self.type = self.base_type
return super().save(*args, **kwargs)

In our users/models.py module, underneath the User mode:

Example 22.9: ”Adding the Inventor Proxy Model”

class InventorManager(BaseUserManager):
def get_queryset(self, *args, **kwargs):
results = super().get_queryset(*args, **kwargs)
return results.filter(type=User.Types.INVENTOR)

class Inventor(User):
# This sets the user type to INVENTOR during record creation
base_type = User.Types.INVENTOR

# Ensures queries on the Inventor model return only Inventors

objects = InventorManager()

# Setting proxy to "True" means a table WILL NOT be created

# for this record
class Meta:
proxy = True

# Only inventors get to invent new flavors!

def invent(self):
# Magical custom logic goes Here
return "Delicious!"

As mentioned in the comments, proxy models don’t add fields. What they do create is refer-
ences to a model object on which we can hang custom managers, methods, and properties.
The example queries below will show the power of this approach:

270 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 22.3: Handling Multiple User Types

Example 22.10: Demonstrating the Power of Proxy Models

>>> from users.models import User, Inventor

>>> User.objects.count() # Over 300 million users!
323482357
>>> Inventor.objects.count() # But only 3 inventors
3
>>> # Calling someone as both a User and an Inventor
>>> user = User.objects.get(username='umafeldroy')
>>> user
<User: uma>
>>> inventor = Inventor.objects.get(username='umafeldroy')
>>> inventor
<Inventor: uma>
>>> # Calling a method that's only for inventors
>>> user.invent()
AttributeError
>>> inventor.invent()
'Delicious'

The proxy approach gives us multiple types of users without creating a new User table or
being forced to dramatically extend django.contrib.auth in such a way that we can’t
use third-party libraries.

That proxy models can have their own model managers means that we can have more ex-
plicit queries. This helps prevent mistakes in granting permission. Compare these two code
examples for clarity:

Example 22.11: Explicitly Filtering on Inventor Type

>>> User.objects.filter(type=User.Types.INVENTOR)
>>> Inventor.objects.filter() # Our preference

22.3.3 Adding Extra Data Fields

We’ve found there two ways to handle extra data fields for different user types:

1 Use OneToOneField relations to profile models as described in Section 22.2.3: Op-

tion 3: Linking Back From a Related Model.
2 Put all the fields in the base User model. This approach is simple, but with enough
users and user-specific data can result in the User table slowing. Even without perfor-

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 271
Chapter 22: Dealing With the User Model

mance considerations in, there is significant risk of innappropiate data being held by
the wrong user type.

Our preference is the first option, to link back from a related model and combine with proxy
models. Here’s how we do it using all the ”Profile” models from Section 22.2.3: Option 3:
Linking Back From a Related Model.

class Inventor(User):
# ...
objects = InventorManager()

class Meta:
proxy = True

@property
def extra(self):
return self.inventorprofile

class Scooper(User):
# ...
objects = ScooperManager()

class Meta:
proxy = True

@property
def extra(self):
return self.scooperprofile

class Eater(User):
# ...
objects = EaterManager()

class Meta:
proxy = True

@property
def extra(self):
return self.eaterprofile

Now, no matter the user type, the Profile from the One-to-One relationship can be accessed

272 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 22.4: Summary

thus:

ä eater.extra.favorite_ice_cream
ä scooper.extra.scoops_scooped
ä invented.extra.flavors_invented

We really like this approach, it is easy to remember that the relation to the profile table is
accessible via the extra property. It even works with architectures that allow users to have
more than one role.

Proxy models are something typically used sparingly as complex implementations can be
confusing. However, under the right circumstances such as multiple user types, they are
incredibly useful.

22.3.4 Additional Resources on Multiple User Types

ä Daniel Feldroy’s Youtube tutorial on this topic: feld.to/multiple-user-types
ä Official documentation on Proxy Models: docs.djangoproject.com/en/3.2/
topics/db/models/#proxy-models
ä Vitor Freitas’ of ‘Simple is better than Complex article’ on the topic of multiple user
types. Very useful article, even if he doesn’t use proxy models: feld.to/2VSi3My

22.4 Summary
In this chapter we covered the new method to find the User model and define our own
custom ones. Depending on the needs of a project, they can either continue with the current
way of doing things or customize the actual user model. We also covered the use case of
multiple user model types through proxy models.

The next chapter is a dive into the world of third-party packages.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 273
Chapter 22: Dealing With the User Model

274 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
23 | Django’s Secret Sauce:
Third-Party Packages

TODO - Add mention of https://www.python.org/dev/peps/pep-0517/

The real power of Django is more than just the framework and documentation available at
djangoproject.com. It’s the vast selection of third-party Django and Python packages
provided by the open source community. There are many, many third-party packages avail-
able for your Django projects which can do an incredible amount of work for you. These
packages have been written by people from all walks of life, and they power much of the
world today.

Figure 23.1: A jar of Django’s mysterious secret sauce. Most don’t have a clue what this is.

Much of professional Django and Python development is about the incorporation of third-
party packages into Django projects. If you try to write every single tool that you need from
scratch, you’ll have a hard time getting things done.

This is especially true for us in the consulting world, where client projects consist of many
of the same or similar building blocks.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 275
Chapter 23: Django’s Secret Sauce: Third-Party Packages

Figure 23.2: The secret is out. It’s just hot fudge.

23.1 Examples of Third-Party Packages

Chapter 37: Appendix A: Packages Mentioned In This Book covers all of the packages
mentioned throughout Two Scoops of Django. This list is a great starting point if you’re
looking for highly-useful packages to consider adding to your projects.

Note that not all of those packages are Django-specific, that means that you can use some of
them in other Python projects. (Generally, Django-specific packages have names prefixed
with “django-” or “dj-”, but there are many exceptions.)

23.2 Know About the Python Package Index

The Python Package Index (PyPI), located at pypi.org/, is a repository of software for
the Python programming language. As of the time this sentence was written, it lists over
100,000 packages, including Django itself.

For the vast majority of Python community, no open source project release is considered
official until it occurs on the Python Package Index.

The Python Package Index is much more than just a directory. Think of it as the world’s
largest center for Python package information and files. Whenever you use pip to install a
particular release of Django, pip downloads the files from the Python Package Index. Most
Python and Django packages are downloadable from the Python Package Index in addition
to pip.

23.3 Know About DjangoPackages.org

Django Packages (djangopackages.org) is a directory of reusable apps, sites, tools and
more for your Django projects. Unlike PyPI, it doesn’t store the packages themselves, in-
stead providing a mix of hard metrics gathered from the Python Package Index, GitHub,
ReadTheDocs, and “soft” data entered by users.

Django Packages is best known as a comparison site for evaluating package features. On

276 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 23.4: Know Your Resources

Django Packages, packages are organized into handy grids so they can be compared against
each other.

Django Packages also happens to have been created by the authors of this book, with con-
tributions from many, many people in the Python community. Thanks to many volunteers
it is continually maintained and improved as a helpful resource for Django users.

23.4 Know Your Resources

Django developers unaware of the critical resources of Django Packages and the Python
Package Index are denying themselves one of the most important advantages of using
Django and Python. If you are not aware of these tools, it’s well worth the time you spend
educating yourself.

As a Django (and Python) developer, make it your mission to use third-party libraries in-
stead of reinventing the wheel whenever possible. The best libraries have been written, doc-
umented, and tested by amazingly competent developers working around the world. Stand-
ing on the shoulders of these giants is the difference between amazing success and tragic
downfall.

As you use various packages, study and learn from their code. You’ll learn patterns and tricks
that will make you a better developer.

On the other hand, it’s very important to be able to identify the good packages from the
bad. It’s well worth taking the time to evaluate packages written by others the same way we
evaluate our own work. We cover this later in this chapter in Section 23.10: What Makes a
Good Django Package?

23.5 Tools for Installing and Managing Packages

To take full advantage of all the packages available for your projects, having virtualenv and
pip (or Conda or Poetry) installed isn’t something you can skip over. It’s mandatory.

Refer to Chapter 2: The Optimal Django Environment Setup for more details.

23.6 Package Requirements

As we mentioned earlier in Chapter 5: Settings and Requirements Files, we manage our
Django/Python dependencies with requirements files. These files go into the requirements/
directory that exists in the root of our projects.

23.7 Wiring Up Django Packages: The Basics

When you find a third-party package that you want to use, follow these steps:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 277
Chapter 23: Django’s Secret Sauce: Third-Party Packages

23.7.1 Step 1: Read the Documentation for the Package

Are you sure you want to use it? Make sure you know what you’re getting into before you
install any package.

23.7.2 Step 2: Add Package and Version Number to Your

Requirements
If you recall from Chapter 5: Settings and Requirements Files, a requirements/base.txt file
looks something like this (but probably longer):

Example 23.1: Adding Packages with Version Numbers to Requirements

Django==1.11
coverage==4.3.4
django-extensions==1.7.6
django-braces==1.11

Note that each package is pinned to a specific version number. Always pin your package
dependencies to version numbers.

What happens if you don’t pin your dependencies? You are almost guaranteed to run into
problems at some point when you try to reinstall or change your Django project. When new
versions of packages are released, you can’t expect them to be backwards-compatible.

Our sad example: Once we followed a software-as-a-service platform’s instructions for us-
ing their library. As they didn’t have their own Python client, but an early adopter had a
working implementation on GitHub, those instructions told us to put the following into
our requirements/base.txt:

Example 23.2: How Not To List Requirements

-e git+https://github.com/erly-adptr/py-junk.git#egg=py-jnk

Our mistake. We should have known better and pinned it to a particular git revision number.

Not the early adopter’s fault at all, but they pushed up a broken commit to their repo. Once
we had to fix a problem on a site very quickly, so we wrote a bug fix and tested it locally in
development. It passed the tests. Then we deployed it to production in a process that grabs

278 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 23.8: Troubleshooting Third-Party Packages

all dependency changes; unfortunately the broken commit was interpreted as a valid change.
Which meant, while fixing one bug, we crashed the site.

Not a fun day.

The purpose of using pinned releases is to add a little formality and process to our published
work. Especially in Python, GitHub and other repos are a place for developers to pub-
lish their work-in-progress, not the final, stable work upon which our production-quality
projects depend.

One more thing, when pinning dependencies, try to pin the dependencies of dependencies.
It just makes deployment and testing that much more predictable.

23.7.3 Step 3: Install the Requirements Into Your Virtualenv

Assuming you are already in a working virtualenv and are at the <repo_root> of your
project, you pip install the appropriate requirements file for your setup, e.g. require-
ments/dev.txt.

If this is the first time you’ve done this for a particular virtualenv, it’s going to take a while
for it to grab all the dependencies and install them.

23.7.4 Step 4: Follow the Package’s Installation Instructions

Exactly
Resist the temptation to skip steps unless you’re very familiar with the package. Since open
source Django package developers tend to take pride in their documentation and love to get
people to use their packages, most of the time the installation instructions they’ve authored
make it easy to get things running.

23.8 Troubleshooting Third-Party Packages

Sometimes you run into problems setting up a package. What should you do?

First, make a serious effort to determine and solve the problem yourself. Pour over the
documentation and make sure you didn’t miss a step. Search online to see if others have run
into the same issue. Be willing to roll up your sleeves and look at the package source code,
as you may have found a bug.

If it appears to be a bug, see if someone has already reported it in the package repository’s
issue tracker. Sometimes you’ll find workarounds and fixes there. If it’s a bug that no one
has reported, go ahead and file it.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 279
Chapter 23: Django’s Secret Sauce: Third-Party Packages

If you still get stuck, try asking for help in all the usual places: StackOverflow, IRC #django,
the project’s IRC channel if it has its own one, and your local Python user group. Be as
descriptive and provide as much context as possible about your issue.

23.9 Releasing Your Own Django Packages

Whenever you write a particularly useful Django app, consider packaging it up for reuse in
other projects.

The best way to get started is to follow Django’s Advanced Tutorial: How to Write Reusable
Apps, for the basics: docs.djangoproject.com/en/3.2/intro/reusable-apps/

In addition to what is described in that tutorial, we recommend that you also:

ä Create a public repo containing the code. Most Django packages are hosted on
GitHub these days, so it’s easiest to attract contributors there, but various alterna-
tives exist (Gitlab, Bitbucket, Assembla, etc.).
ä Release the package on the Python Package Index (pypi.org). Follow
the submission instructions at packaging.python.org/distributing/
#uploading-your-project-to-pypi.
ä Add the package to Django Packages (djangopackages.org).
ä Use Read the Docs (readthedocs.io) or similar service to host your Sphinx docu-
mentation.

TIP: Where Should I Create a Public Repo?

There are websites that offer free source code hosting and version control for open
source projects. As mentioned in Chapter 2: The Optimal Django Environment
Setup, GitHub or GitLab are two popular options.

When choosing alternative hosted version control services, keep in mind that pip
only supports Git, Mercurial, Bazaar, and Subversion.

23.10 What Makes a Good Django Package?

Here’s a checklist for you to use when creating a new open source Django package. Much of
this applies to Python packages that are not Django-specific. This checklist is also helpful
for when you’re evaluating a Django package to use in any of your projects.

23.10.1 Purpose
Your package should do something useful and do it well. The name should be descriptive.
The package repo’s root folder should be prefixed with ‘django-’ or ‘dj-’ to help make it easier
to find.

280 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 23.10: What Makes a Good Django Package?

If part of the package’s purpose can be accomplished with a related Python package that
doesn’t depend on Django, then create a separate Python package and use it as a dependency.

23.10.2 Scope
Your package’s scope should be tightly focused on one small task. This means that your
application logic will be tighter, and users will have an easier time patching or replacing the
package.

23.10.3 Documentation
A package without documentation is a pre-alpha package. Docstrings don’t suffice as docu-
mentation.

As described in Chapter 25: Documentation: Be Obsessed, your docs should be written in

Markdown. A nicely-formatted version of your docs should be generated with MkDocs,
Sphinx, or other tools and hosted publicly. We encourage you to use readthedocs.io
with webhooks so that your formatted documentation automatically updates whenever you
make a change.

If your package has dependencies, they should be documented. Your package’s installation
instructions should also be documented. The installation steps should be bulletproof.

23.10.4 Tests
Your package should have tests. Tests improve reliability, make it easier to advance
Python/Django versions, and make it easier for others to contribute effectively. Write up
instructions on how to run your package’s test suite. If you or any contributor can run your
tests easily before submitting a pull request, then you’re more likely to get better quality
contributions.

23.10.5 Templates
In the past, some Django packages provided instructions for creating templates in their docs
in lieu of actual template files. However, nowadays it’s pretty standard for Django packages
to come with a set of barebones templates that demonstrate basic functionality. Typically
these templates contain only minimalist HTML, any needed JavaScript, and no CSS. The
exception is for packages containing widgets that require CSS styling.

23.10.6 Activity
Your package should receive regular updates from you or contributors if/when needed.
When you update the code in your repo, you should consider uploading a minor or ma-

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 281
Chapter 23: Django’s Secret Sauce: Third-Party Packages

jor release to the Python Package Index.

23.10.7 Community
Great open source packages, including those for Django, often end up receiving contribu-
tions from other developers in the open source community. All contributors should receive
attribution in a CONTRIBUTORS.rst or AUTHORS.rst file.

Be an active community leader if you have contributors or forks of your package. If your
package is forked by other developers, pay attention to their work. Consider if there are
ways that parts or all of their work can be merged into your fork. If the package’s function-
ality diverges a lot from your package’s purpose, be humble and consider asking the other
developer to give their fork a new name.

23.10.8 Modularity
Your package should be as easily pluggable into any Django project that doesn’t replace
core components (templates, ORM, etc) with alternatives. Installation should be minimally
invasive. Be careful not to confuse modularity with over-engineering, though.

23.10.9 Availability on PyPI

All major and minor releases of your package should be available for download from the
Python Package Index. Developers who wish to use your package should not have to go to
your repo to get a working version of it. Use proper version numbers per the next section.

23.10.10 Uses the Broadest Requirements Specifiers Possible

Your third-party package should specify in setup.py the install_requires argument
what other libraries your package requires in the broadest terms possible. However, this
is a terrible way to define a package’s requirements:

Example 23.3: Narrow Requirements for a Package

# DON'T DO THIS!
# requirements for django-blarg

Django==3.0
requests==2.11.0

The reason is dependency graphs. Every so often something that you absolutely pin to a
specific version of Django or another library will break on someone else’s site project. For

282 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 23.10: What Makes a Good Django Package?

example, what if icecreamratings.com was our site and this was its deployed project’s require-
ments.txt file, and we installed django-blarg?

Example 23.4: Requirements.txt for icecreamratings.com

# requirements.txt for the mythical web site 'icecreamratings.com'

Django==3.1
requests==2.13.0
django−blarg==1.0

# Note that unlike the django−blarg library , we explicitly pin

# the requirements so we have total control over the environment

What would happen if Bad Example 21.3 were installed to a project with Example 21.4
requirements is that the Django 3.0 requirement would overwrite the Django 3.1 specifica-
tion during installation of icecreamratings.com requirements. As there are several backwards
incompatibilities between Django 3.0 and 3.1, django-blarg could make icecreamratings.com
site simply throw HTTP 500 errors.

Your third-party package should specify what other libraries your package requires in the
broadest terms possible:

Example 23.5: Broadly Defined Package Requirements

# requirements for django-blarg

Django>=3.1,<3.0
requests>=2.13.0,<=3.0.0

Additional Reading:

ä pip.pypa.io/en/stable/reference/pip_install/
#requirement-specifiers
ä nvie.com/posts/pin-your-packages/

23.10.11 Proper Version Numbers

Like Django and Python, we prefer to adhere to the strict version of PEP 386 naming
schema. In fact we follow the ‘A.B.C’ pattern. Let’s go through each element:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 283
Chapter 23: Django’s Secret Sauce: Third-Party Packages

‘A’ represents the major version number. Increments should only happen with large changes
that break backwards compatibility from the previous major version. It’s not uncom-
mon to see large API changes between versions.
‘B’ is the minor version number. Increments include less breaking changes, or deprecation
notices about forthcoming changes.
‘C’ represents bug fix releases, and purists call this the ‘micro’ release. It’s not uncommon
for developers to wait until a project has its first release at this level before trying the
latest major or minor release of an existing project.

For alpha, beta, or release-candidates for a project, the convention is to place this informa-
tion as a suffix to the upcoming version number. So you might have:

ä Django 3.2-alpha
ä django-crispy-forms 1.9.1-beta

WARNING: Don’t Upload Unfinished Code to PyPI

PyPI, the Python Package Index, is meant to be the place where dependable, stable
packages can be harnessed to build Python projects. PyPI is not the place for Alpha,
Beta, or Release Candidate code, especially as pip and other tools will fetch the latest
release by default.

Be nice to other developers and follow the convention of only placing proper
releases on PyPI.

Note: While modern versions of pip no longer install pre-releases by default, it’s
dangerous to expect users of code to have the latest pip version installed.

Additional Reading:

ä python.org/dev/peps/pep-0386
ä semver.org

23.10.12 Name
The name of the project is absolutely critical. A well-named project makes it easy to discover
and remember, a poor name hides it from potential users, can scare off its use from some
developer shops, and even block it from being listed on PyPI, Django Packages, and other
resources.

284 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 23.10: What Makes a Good Django Package?

We did cover the basics in Section 4.2: What to Name Your Django Apps, but here are tips
that apply to open source Django packages:

ä Check to see that the name isn’t already registered on PyPI. Otherwise, it won’t be trivial
to install with pip.
ä Check to see that the name isn’t on Django Packages. This applies only to packages de-
signed for use with Django.
ä Don’t use names that include obscenity. While you might find them funny, it’s unfortu-
nate for others. For example a noted developer once created a library that couldn’t be
used at NASA until he agreed to change the name.

23.10.13 License
Your package needs a license. Preferably, for individuals it should be licensed under the MIT
licenses, which are generally accepted for being permissive enough for most commercial or
noncommercial uses. If you are worried about patents, then go with the Apache license.

Create a LICENSE.rst file in your repo root, mention the license name at the top, and paste
in the appropriate text from the (OSI) approved list at choosealicense.com.

TIP: Licenses Protect You and the World

In this era of casual litigation and patent trolls adding a software license isn’t just
a matter of protecting your ownership of the code. It’s much, much more. If you
don’t license your code, or use an unapproved license not vetted by real lawyers, you
run the risk of your work being used as a weapon by a patent troll, or in the case of
financial or medical disaster, you could be held liable.
OSI-approved licenses all include a couple critical statements on copyright, redis-
tribution, disclaimer of warranty, and limitation of liability.

23.10.14 Clarity of Code

The code in your Django package should be as clear and simple as possible. Don’t use weird,
unusual Python/Django hacks without explaining what you are doing.

23.10.15 Use URL Namespaces

Described in Section 8.4: Use URL Namespaces, URL namespaces allow for greater inter-
operability. Using means it’s easier to manage collisions between projects, or even prepare
for it ahead of time.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 285
Chapter 23: Django’s Secret Sauce: Third-Party Packages

If there is concern about future collisions, settings-based URL namespace systems can be
implemented. This is where the project defines its URL namespace as a setting, then pro-
vides a Django context processor and detailed instructions on use. While it’s not hard to
implement, it does create a level of abstraction that can make a project a little bit harder to
maintain.

23.11 Creating Your Own Packages the Easy Way

Releasing your own bit of code can be a wonderfully rewarding experience. Everyone should
do it!

That said, putting all the pieces together in order to make a reusable Django package is a lot
of work, and it’s common to get things wrong. Fortunately, Cookiecutter makes this easy.

PACKAGE TIP: Cookiecutter: Project Templates Made Easy

In 2013 Audrey created the popular Cookiecutter utility for generating project tem-
plates. It’s easy to use and very powerful. Numerous templates exist for Python and
Django packages. Even better, many IDEs such as PyCharm and Visual Studio
Code now provide support for Cookiecutter-based templates.
ä github.com/cookiecutter/cookiecutter
ä cookiecutter.readthedocs.io

In the Cookiecutter templates referenced below, we have vetted them by aggressively asking
for them to be reviewed by leaders in both the Django and Python communities. Just use
the following bash example at the command-line:

Example 23.6: Using Cookiecutter to Jumpstart Packages

# Only if you haven't installed cookiecutter yet

\$ pip install cookiecutter

# Creating a Django Package from scratch

\$ cookiecutter
,→ https://github.com/pydanny/cookiecutter-djangopackage.git

# Creating a Python Package from scratch

\$ cookiecutter
,→ https://github.com//ionelmc/cookiecutter-pylibrary.git

You’ll be prompted to provide information. The generated result will be an implementation

of a base Django/Python/etc. package template that includes code, documentation, tests,

286 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 23.12: Maintaining Your Open Source Package

license, and much more.

23.12 Maintaining Your Open Source Package

WARNING: Open Source Burnout and Giving Too Much
Unless you are getting paid professionally to do open source work, remember that
this is volunteer work done for pleasure. Do what you can at your own pace, and
just try your best.

The open source packages that you create have a life of their own. They mature over time,
changing as their needs and the development standards grow over time. Here are some
things we should do when maintaining open source projects:

23.12.1 Give Credit for Pull Requests

When someone submits a pull request that’s accepted, treat them right. Make sure to add
the contributor to a project’s author document called something like CONTRIBUTORS.md
or AUTHORS.md.

23.12.2 Handling Bad Pull Requests

Sometimes you get pull requests that you just have to reject. Be nice and positive about it,
since a well-handled rejected pull request can make a friend for life.

Here are problematic pull requests that should be considered for rejection:

ä Any pull request that fails the tests. Ask for fixes. See Chapter 24: Testing Stinks and
Is a Waste of Money!.
ä Any added code that reduces test coverage. Again, see chapter 24.
ä Pull requests should change/fix as little as possible. Large, wide-sweeping changes in a
pull request should be rejected, with comments to isolate changes in smaller, atomic
pull requests.
ä Overly complex code submissions should be carefully considered. There is nothing wrong
with asking for simplification, better comments, or even rejecting an overly complex
pull request.
ä Code that breaks PEP-8 needs to be resubmitted. The Django world follows PEP-8 very
closely, and so should your project. Submissions that violate PEP 8 can be requested
to be improved.
ä Code changes combined with major whitespace cleanup. If someone submits a change of
two lines of code and corrects 200 lines of whitespace issues, the diff on that pull

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 287
Chapter 23: Django’s Secret Sauce: Third-Party Packages

request is functionally unreadable and should be rejected. Whitespace cleanups need

to be in their own pull request.

WARNING: Code Changes Combined With Major Whitespace

Cleanup
We’re adding a warning because this is arguably a form of code obfuscation by a
third party. One could argue it’s potentially a security risk. What better way to inject
malignant code than through a pull request?

23.12.3 Do Formal PyPI Releases

In the Python community, it’s considered irresponsible to force developers to rely on a ‘stable’
master or trunk branch of critical open source projects because the PyPI version is out of
date. This can cause problems as open source code repositories are not considered to be good
sources of production quality code. For example, which particular commit or tag should be
used? On the other hand, PyPI, is a known resource designed to securely provide valid
installable packages.

In the Python world, the accepted best practice is to release when significant (or even minor)
changes or bug fixes happen on trunk or master. In fact, minor bug fix releases are a part of
every ongoing software project and no one faults anyone for these kinds of things (except
in US government IT contracts, but that’s outside the scope of this book).

If you aren’t sure how this works, please look at python-request’s change history, it be-
ing one of Python’s most popular projects: github.com/psf/requests/blob/master/
HISTORY.md

To create and upload your distribution, use the following steps:

Example 23.7: Using Twine to Upload Package Distributions

pip install twine

python setup.py sdist
twine upload dist/*

PACKAGE TIP: What is Twine?

Twine is the preferred library for uploading packages to PyPI. The problem with
python setup.py is that it sends files over a non secure connection, exposing

288 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 23.12: Maintaining Your Open Source Package

your library to a man-in-the-middle attack. In contrast, twine uses only verified

TLS to upload your package.

That’s not all! Twine works better at uploading Wheels (see the next subsection),
doesn’t require executing the setup.py, and even pre-signs your releases. If you are
seriously security minded, it’s the tool of choice.

23.12.4 Create and Deploy Wheels to PyPI

According to PEP 427, Wheels are the new standard of python distribution. They are in-
tended to replace eggs and provide a number of advantages including faster installation and
allow secure digital signing. Support is offered in pip >= 1.4 and setuptools >= 0.8.

Example 23.8: Installing Wheel

pip install wheel

Then, after you’ve deployed your package to PyPI, run the following commands:

Example 23.9: Creating Wheel Distributions and Uploading Them

python setup.py bdist_wheel

twine upload dist/*

Twine makes universal wheels when the optional setup.cfg file is at the same level as setup.py
and includes this snippet:

Example 23.10: Configuring Universal Wheels

# setup.cfg
[wheel]
universal = 1

Wheel Resources:

Specification: PEP 427 python.org/dev/peps/pep-0427

Wheel Package on PyPI https://feld.to/3eTOdyS/project/wheel
Documentation wheel.readthedocs.io
Advocacy pythonwheels.com

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 289
Chapter 23: Django’s Secret Sauce: Third-Party Packages

23.12.5 Add Git Tags to the Repo

We also like to git tag our releases in the repo. This is a snapshot of the code at the time the
release was submitted to PyPI (see previous subsection). In addition to serving as a historical
record, it can also location for installing packages from. For example, if an organization
doesn’t want to stand up a private package server, they can rely on git tags (or releases for
those repo hosts like GitHub that provide this option).

Example 23.11: Create and Push Tag to repo

git tag -a v1.4 -m "my version 1.4"

git push origin v1.4

Reference: git-scm.com/book/en/v2/Git-Basics-Tagging

23.12.6 Upgrade the Package to New Versions of Django

Every once in awhile, Django is updated with a minor release. Approximately once a year
there is a major Django release. When this happens, it’s very important to run our package’s
test suite in a virtualenv that contain Django’s latest release.

If for no other reason, this is an excellent reason to include tests in your project.

23.12.7 Follow Good Security Practices

We discuss security in-depth in Chapter 28: Security Best Practices. However, core Django,
Python, and PyPy developer Alex Gaynor has an incredibly useful article for maintainers
of any open source project:
alexgaynor.net/2013/oct/19/security-process-open-source-projects

TIP: Alex Gaynor on Security for Open Source Projects

“Security vulnerabilities put your users, and often, in turn, their users at
risk. As an author and distributor of software, you have a responsibility
to your users to handle security releases in a way most likely to help
them avoid being exploited.”

23.12.8 Provide Sample Base Templates

Always include some basic templates for views using your project. We prefer to write either
incredibly simple HTML or use a common front-end frameworks such as Bootstrap or Tail-
wind. This makes ‘test-driving’ the project much easier for developers who are considering

290 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 23.13: Additional Reading

using it to solve their problems. Invariably they’ll modify the templates inside their own
templates/ directory, but this just makes everything so much easier.

In addition, include a templates/myapp/base.html to increase interoperability. You can see a

description and example of this in cookiecutter-djangopackage:

bit.ly/2onSzCV

23.12.9 Give the Package Away

Sometimes, life takes you away from maintaining a package. It might be family or a new
job, but sometimes you just have no need for a particular open source project. Time consid-
erations might mean that you don’t have the ability to review pull requests or explore ideas
for new features. If you’re the creator of a project it can be extremely challenging to let it go.

However, by giving a project away to active maintainers, it can be reborn and prove more
useful. It also earns the respect of the developer community at large.

Some notable giveaways in the Django and Python communities include:

ä Ian Bicking and pip/virtualenv.

ä Daniel and Audrey Feldroy and djangopackages.org
ä Daniel Feldroy and django-uni-form, dj-stripe, and django-mongonaut
ä Audrey Feldroy and Cookiecutter
ä Rob Hudson and django-debug-toolbar.

23.13 Additional Reading

The following are links to useful articles for anyone contributing to, creating, or maintaining
open source libraries:

ä djangoappschecklist.com A personal favorite of ours, the Django Apps Check-

list is a checklist for everything in this chapter.
ä alexgaynor.net/2013/sep/26/effective-code-review
ä hynek.me/articles/sharing-your-labor-of-love-pypi-quick-and-dirty

23.14 Summary
Django’s real power is in the vast selection of third-party packages available to you for use
in your Django projects.

Make sure that you have pip and virtualenv (or Poetry as an alternative) installed and know
how to use them, since they’re your best tools for installing packages on your system in a
manageable way.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 291
Chapter 23: Django’s Secret Sauce: Third-Party Packages

Get to know the packages that exist. The Python Package Index and Django Packages are
a great starting point for finding information about packages.

Package maturity, documentation, tests, and code quality are good starting criteria when
evaluating a Django package.

Installation of stable packages is the foundation of Django projects big and small. Being
able to use packages means sticking to specific releases, not just the trunk or master of a
project. Barring a specific release, you can rely on a particular commit. Fixing problems that
a package has with your project takes diligence and time, but remember to ask for help if
you get stuck.

We also covered how to create your own third-party package, and provided basic instruction
on how to use cookiecutter to jump-start you on your way to releasing something on the
Python Package Index. We also included instructions on using the new Wheel format.

Finally, we provided guidance on how to maintain a package.

292 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
24 | Testing Stinks and Is a Waste of
Money!

There, got you to this chapter.

Now you have to read it.

We’ll try and make this chapter interesting.

24.1 Testing Saves Money, Jobs, and Lives

Daniel’s Story: Ever hear the term “smoke test”?

Gretchen Davidian, a Management and Program Analyst at NASA, told me that when she
was still an engineer, her job as a tester was to put equipment intended to get into space
through such rigorous conditions that they would begin emitting smoke and eventually
catch on fire.

That sounds exciting! Employment, money, and lives were on the line, and knowing
Gretchen’s attention to detail, I’m sure she set a lot of hardware on fire.

Keep in mind that for a lot of us, as software engineers, the same risks are on the line
as NASA. I recall in 2004 while working for a private company how a single miles-vs-
kilometers mistake cost a company hundreds of thousands of dollars in a matter of hours.
Quality Assurance (QA) staff lost their jobs, which meant that money and health benefits
were gone. In other words, employment, money, and possibly lives can be lost without ad-
equate tests. While the QA staff were very dedicated, everything was done via manually
clicking through projects, and human error simply crept into the testing process.

Today, as Django moves into a wider and wider set of applications, the need for automated
testing is just as important as it was for Gretchen at NASA and for the poor QA staff in 2004.
Here are some cases where Django is used today that have similar quality requirements:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 293
Chapter 24: Testing Stinks and Is a Waste of Money!

ä Applications that handle medical information.

ä Applications that provide life-critical resources to people in need.
ä Applications working with user’s financial information.

PACKAGE TIP: Useful Library for Testing Django Projects

We like to use coverage.py.

This tool provides clear insight into what parts of your code base are covered by tests,
and what lines haven’t been touched by tests. You also get a handy percentage of how
much of your code is covered by tests. Even 100% test coverage doesn’t guarantee a
bug-free application, but it helps.

We want to thank Ned Batchelder for his incredible work in maintaining cover-
age.py. It’s a superb project and is useful for any Python-related project.

24.2 How to Structure Tests

Let’s say we’ve just created a new Django app. The first thing we do is delete the default but
useless tests.py module that python manage.py startapp creates. In it’s place we create
a directory called tests and place an empty __init__.py within.

Inside that new directory, because most apps need them, we create test_forms.py,
test_models.py, test_views.py modules. Tests that apply to forms go into test_forms.py, model
tests go into test_models.py, and so on.

Here’s what it looks like:

Example 24.1: How to Structure Tests

popsicles/
__init__.py
admin.py
forms.py
models.py
tests/
__init__.py
test_forms.py
test_models.py
test_views.py
views.py

294 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 24.3: How to Write Unit Tests

Also, if we have other files besides forms.py, models.py and views.py that need testing, we
create corresponding test files.

We like this approach because while it does add an extra layer of nesting, the alternative is
to have an app structure with a painful number of modules to navigate.

TIP: Prefix Test Modules With test_

It’s critically important that we always prefix test modules with test_, otherwise
Django’s test runner can’t trivially discover our test files. Also, it’s a nice convention
that allows for greater flexibility for viewing filenames in IDEs and text editors. As
always, never lean on your preferred IDE for how to name things.

24.3 How to Write Unit Tests

It’s not uncommon for programmers to feel at the top of their game at the moment they are
writing code. When they revisit that same code in months, weeks, days, or even hours and
it’s not uncommon for programmers to feel as if that same code is of poor quality.

The same applies to writing unit tests.

Over the years, we’ve evolved a number of practices we like to follow when writing tests,
including unit tests. Our goal is always to write the most meaningful tests in the shortest
amount of time. Hence the following:

24.3.1 Each Test Method Tests One Thing

A test method must be extremely narrow in what it tests. A single unit test should never
assert the behavior of multiple views, models, forms, or even multiple methods within a
class. Instead, a single test should assert the behavior of a single view, model, form, method,
or function.

Therein lies a conundrum. How does one run a test for a view, when views often require the
use of models, forms, methods, and functions?

The trick is to be absolutely minimalistic when constructing the environment for a particular
test, as shown in the example below:

Example 24.2: Testing Just One Thing

# flavors/tests/test_api.py
import json

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 295
Chapter 24: Testing Stinks and Is a Waste of Money!

from django.test import TestCase

from django.urls import reverse

from flavors.models import Flavor

class FlavorAPITests(TestCase):

def setUp(self):
Flavor.objects.get_or_create(title='A Title',
,→ slug='a-slug')

def test_list(self):
url = reverse('flavors:flavor_object_api')
response = self.client.get(url)
self.assertEquals(response.status_code, 200)
data = json.loads(response.content)
self.assertEquals(len(data), 1)

In this test, taken from code testing the API we presented in Section 17.2: Illustrating
Design Concepts With a Simple API, we use the setUp() method to create the minimum
possible number of records needed to run the test.

Here’s a much larger example, one based on the REST API example that we provided in
Chapter 17: Building REST APIs With Django REST Framework.

Example 24.3: Testing API Code

# flavors/tests/test_api.py
import json

from django.test import TestCase

from django.urls import reverse

from flavors.models import Flavor

class DjangoRestFrameworkTests(TestCase):

def setUp(self):
Flavor.objects.get_or_create(title='title1', slug='slug1')

296 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 24.3: How to Write Unit Tests

Flavor.objects.get_or_create(title='title2', slug='slug2')

self.create_read_url = reverse('flavors:flavor_rest_api')
self.read_update_delete_url = \
reverse('flavors:flavor_rest_api', kwargs={'slug':
,→ 'slug1'})

def test_list(self):
response = self.client.get(self.create_read_url)

# Are both titles in the content?

self.assertContains(response, 'title1')
self.assertContains(response, 'title2')

def test_detail(self):
response = self.client.get(self.read_update_delete_url)
data = json.loads(response.content)
content = {'id': 1, 'title': 'title1', 'slug': 'slug1',
'scoops_remaining': 0}
self.assertEquals(data, content)

def test_create(self):
post = {'title': 'title3', 'slug': 'slug3'}
response = self.client.post(self.create_read_url, post)
data = json.loads(response.content)
self.assertEquals(response.status_code, 201)
content = {'id': 3, 'title': 'title3', 'slug': 'slug3',
'scoops_remaining': 0}
self.assertEquals(data, content)
self.assertEquals(Flavor.objects.count(), 3)

def test_delete(self):
response = self.client.delete(self.read_update_delete_url)
self.assertEquals(response.status_code, 204)
self.assertEquals(Flavor.objects.count(), 1)

24.3.2 For Views, When Possible Use the Request Factory

The django.test.client.RequestFactory provides a way to generate a request in-
stance that can be used as the first argument to any view. This provides a greater amount of
isolation than the standard Django test client, but it does require a little bit of extra work on

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 297
Chapter 24: Testing Stinks and Is a Waste of Money!

the part of the test writer. This is because the request factory doesn’t support middleware,
including session and authentication.

See docs.djangoproject.com/en/3.2/topics/testing/advanced/

Unfortunately the documentation doesn’t cover when you want to test a view wrapped with
a single middleware class. For example, if a view required sessions, this is how we would
do it:

Example 24.4: How to Add Middleware to Requests and Responses

from django.contrib.auth.models import AnonymousUser

from django.contrib.sessions.middleware import SessionMiddleware
from django.test import TestCase, RequestFactory

from .views import cheese_flavors

def add_middleware_to_request(request, middleware_class):

middleware = middleware_class()
middleware.process_request(request)
return request

def add_middleware_to_response(request, middleware_class):

middleware = middleware_class()
middleware.process_response(request)
return request

class SavoryIceCreamTest(TestCase):
def setUp(self):
# Every test needs access to the request factory.
self.factory = RequestFactory()

def test_cheese_flavors(self):
request = self.factory.get('/cheesy/broccoli/')
request.user = AnonymousUser()

# Annotate the request object with a session

request = add_middleware_to_request(request,
,→ SessionMiddleware)
request.session.save()

# process and test the request

298 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 24.3: How to Write Unit Tests

response = cheese_flavors(request)
self.assertContains(response, 'bleah!')

24.3.3 Don’t Write Tests That Have to Be Tested

Tests should be written as simply as possible. If the code in a test (or the code called to help
run a test) feels complicated or abstracted, then we have a problem. In fact, we ourselves
are guilty of writing overly complicated utility test functions that required their own tests
in the past. As one can imagine, this made debugging the actual tests a nightmare.

24.3.4 Don’t Repeat Yourself Doesn’t Apply to Writing Tests

The setUp() method is really useful for generating reusable data across all test methods in
a test class. However, sometimes we need similar but different data between test methods,
which is where we often fall into the trap of writing fancy test utilities. Or worse, we decide
that rather than write 20 similar tests, we can write a single method that when passed certain
arguments will handle all the work for us.

Our favorite method of handling these actions is to just dig in and write the same or sim-
ilar code multiple times. In fact, we’ll quietly admit to copy/pasting code between tests to
expedite our work.

TIP: Our Problem With the Django Testing Tutorial #5

The official Django beginner tutorial on testing demonstrates the implementation
of a create_question() utility method, which is designed to take some of the
repetition out of the process of created questions.
We think its inclusion in the tutorial is a mistake. Simple examples in the official
documentation that follow poor practice provide encouragement that often play
out in terrible ways. We’ve encountered projects inspired by this example to add
deep layers of abstraction to their testing code, abstraction that makes it incredibly
challenging to correct and enhance existing tests.
Again, Don’t Repeat Yourself doesn’t apply to tests.

24.3.5 Don’t Rely on Fixtures

We’ve learned over time that using fixtures is problematic. The problem is that fixtures are
hard to maintain as a project’s data changes over time. Modifying JSON-formatted files to
match your last migration is hard, especially as it can be difficult to identify during the JSON

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 299
Chapter 24: Testing Stinks and Is a Waste of Money!

load process where your JSON file(s) is either broken or a subtly inaccurate representation
of the database.

Rather than wrestle with fixtures, we’ve found it’s easier to write code that relies on the
ORM. Other people like to use third-party packages.

PACKAGE TIP: Tools to Generate Test Data

The following are popular tools for test data generation:
ä factory boy A package that generates model test data.
ä faker This package generates test data, but rather than a random jumble of
text, it creates localized names, addresses, and text. It even comes with instruc-
tions on how to integrate it with factory boy: faker.readthedocs.io/en/
master/
#how-to-use-with-factory-boy
ä model bakery Another package that generates model test data.
ä mock Not explicitly for Django, this allows us to replace parts of our system
with mock objects. This project made its way into the standard library as of
Python 3.3.

24.3.6 Things That Should Be Tested

Everything! Seriously, test whatever is possible to test, including:

Views: Viewing of data, changing of data, and custom class-based view methods.
Models: Creating/updating/deleting of models, model methods, model manager methods.
Forms: Form methods, clean() methods, and custom fields.
Validators: Really dig in and write multiple test methods against each custom validator
you write. Pretend you are a malignant intruder attempting to damage the data in the
site.
Signals: Since they act at a distance, signals can cause grief especially if you lack tests on
them.
Filters: Since filters are essentially just functions accepting one or two arguments, writing
tests for them should be easy.
Template Tags: Since template tags can do anything and can even accept template context,
writing tests often become much more challenging. This means you really need to test
them, since otherwise you may run into edge cases.
Miscellany: Context processors, middleware, email, and anything else not covered in this
list.

300 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 24.3: How to Write Unit Tests

Failure What happens when any of the above fail? Testing for system error is as important
as testing for system success.

The only things that shouldn’t be tested are parts of your project that are already covered by
tests in core Django and third-party packages. For example, a model’s fields don’t have to be
tested if you’re using Django’s standard fields as-is. However, if you’re creating a new type
of field (e.g. by subclassing FileField), then you should write detailed tests for anything
that could go wrong with your new field type.

Figure 24.1: Test as much of your project as you can, as if it were free ice cream.

24.3.7 Test for Failure

Let’s say we have a view that allows users to edit their own ice cream shop reviews. Typical
tests involve logging in, attempting to change the review, and then checking whether they’ve
actually changed. Test success, coverage 100%. Right?

However, this only tests part of the scenario. What if the user isn’t logged in? What if
the user is trying to edit someone else’s review? Does the view produce an error and most
importantly: is the object left unchanged? It has been argued that this test is even more
important than the success scenario: a failure in the success scenario will cause inconvenience
for users but will be reported. A failure in the fail scenario will cause a silent security hole
that could go undetected until it’s too late.

This is only a sampling of the things that can go wrong when we don’t test for what happens
when our systems break down. It is up to us to learn how to test for the exceptions our code

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 301
Chapter 24: Testing Stinks and Is a Waste of Money!

may throw:

ä docs.python.org/2/library/unittest.html#unittest.TestCase.
assertRaises
ä bit.ly/2pAxLtm PyTest assertion docs

24.3.8 Use Mock to Keep Unit Tests From Touching the World
Unit tests are not supposed to test things external to the function or method they are call-
ing. Which means that during tests we should not access external APIs, receive emails or
webhooks, or anything that is not part of the tested action. Alas, this causes a conundrum
when you are trying to write a unit test for a function that interacts with an external API.

At this point you have two choices:

ä Choice #1: Change the unit test to be an Integration Test.

ä Choice #2: Use the Mock library to fake the response from the external API.

The Mock library, created by Michael Foord, has as one of its features the capability to
briefly monkey-patch libraries in order to make them return the exact value that we want.
This way we are testing not the availability of the external API, but instead just the logic of
our code.

In the example displayed below, we are monkey-patching a function in a mythical Ice Cream
API library so our test code doesn’t access anything external to our application.

Example 24.5: Using Mock to Keep Unit Tests From Touching the World

from unittest import mock, TestCase

import icecreamapi

from flavors.exceptions import CantListFlavors

from flavors.utils import list_flavors_sorted

class TestIceCreamSorting(TestCase):

# Set up monkeypatch of icecreamapi.get_flavors()

@mock.patch.object(icecreamapi, 'get_flavors')
def test_flavor_sort(self, get_flavors):
# Instructs icecreamapi.get_flavors() to return an
,→ unordered list.

302 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 24.3: How to Write Unit Tests

get_flavors.return_value = ['chocolate', 'vanilla',

,→ 'strawberry', ]

# list_flavors_sorted() calls the icecreamapi.get_flavors()

# function. Since we've monkeypatched the function, it
,→ will always
# return ['chocolate', 'strawberry', 'vanilla', ]. Which
,→ the.
# list_flavors_sorted() will sort alphabetically
flavors = list_flavors_sorted()

self.assertEqual(
flavors,
['chocolate', 'strawberry', 'vanilla', ]

Now let’s demonstrate how to test the behavior of the list_flavors_sorted() function
when the Ice Cream API is inaccessible.

Example 24.6: Testing For When API is Unavailable

@mock.patch.object(icecreamapi, 'get_flavors')
def test_flavor_sort_failure(self, get_flavors):
# Instructs icecreamapi.get_flavors() to throw a FlavorError.
get_flavors.side_effect = icecreamapi.FlavorError()

# list_flavors_sorted() catches the icecreamapi.FlavorError()

# and passes on a CantListFlavors exception.
with self.assertRaises(CantListFlavors):
list_flavors_sorted()

As an added bonus for API authors, here’s how we test how code handles two different
python-requests connection problems:

Example 24.7: Testing python-requests Connection Failures

@mock.patch.object(requests, 'get')
def test_request_failure(self, get):
"""Test if the target site is inaccessible."""

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 303
Chapter 24: Testing Stinks and Is a Waste of Money!

get.side_effect = requests.exception.ConnectionError()

with self.assertRaises(CantListFlavors):
list_flavors_sorted()

@mock.patch.object(requests, 'get')
def test_request_failure_ssl(self, get):
"""Test if we can handle SSL problems elegantly."""
get.side_effect = requests.exception.SSLError()

with self.assertRaises(CantListFlavors):
list_flavors_sorted()

24.3.9 Use Fancier Assertion Methods

Comparing two lists (or tuples) is a very common use case. However, if the lists are al-
lowed to have different sort orders, then we have to sort the lists to match, then run
self.assertEqual(control_list, candidate_list) right?

Not if we know to use unittest’s assertItemsEqual() assertion method! In fact, Python

and Django’s unittest documentation includes handy links to the very useful assertion types
we get for free:

ä docs.python.org/3/library/unittest.html#assert-methods
ä docs.djangoproject.com/en/3.2/topics/testing/tools/#assertions

We’ve found the following assert methods extremely useful:

ä assertRaises
ä assertRaisesMessage()
ä assertCountEqual()
ä assertDictEqual()
ä assertFormError()
ä assertContains() Check status 200, checks in response.content.
ä assertHTMLEqual() Amongst many things, ignores whitespace differences.
ä assertInHTML() Great for confirming a tiny bit of HTML in a giant HTML page.
ä assertJSONEqual()
ä assertURLEqual()

304 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 24.4: What About Integration Tests?

24.3.10 Document the Purpose of Each Test

Just as it is a good idea to document the purpose of a class, method, or function with doc-
strings, it is also a good idea to document the purpose of the test analogs of these items.
If undocumented code makes a project somewhat harder to maintain, undocumented test
code can make a project impossible to test. To remedy this, a little bit of docstring can go a
long way.

If one thinks this is boring, well, we’ve found that a good way to deal with an impossible-
to-debug problem is to document the related tests. By the time the tests are documented,
we have either figured out the problem or we have documented tests. Either case is a win!

Resources:

ä hynek.me/articles/document-your-tests/
ä interrogate.readthedocs.io/ - A library that checks code bases for missing
docstrings

24.4 What About Integration Tests?

Integration testing is when individual software modules are combined and tested as a group.
This is best done after unit tests are complete. Examples of integration tests can include:

ä Selenium tests to confirm that an application works in the browser.

ä Actual testing against a third-party API instead of mocking responses. For example,
Django Packages conducts periodic tests against GitHub and the PyPI API to ensure
that its interaction with those systems is valid.
ä Interacting with httpbin.org to confirm the validity of outbound requests.
ä Using runscope.com or postman.com to validate that our API is working as ex-
pected.

Integration tests are a great way to confirm that ‘all the pieces’ are working. We can confirm
that our users will see what they are supposed to see and our APIs are functioning correctly.

The downside of integration tests are:

ä Setting up integration tests can take a lot of time.

ä Compared to unit tests, integrations are extremely slow. That’s because instead of
testing the smallest components, integration tests are, by definition, testing the whole
system.
ä When errors are thrown by integration tests, uncovering the problem is harder than
unit tests. For example, a problem affecting a single type of browser might be caused
by a unicode transformation happening at the database level.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 305
Chapter 24: Testing Stinks and Is a Waste of Money!

ä Integration tests are fragile compared to unit tests. A small change in a component
or setting can break them. We’ve yet to work on a significant project where at least
one person wasn’t forever blocked from running them successfully.

Even with these problems, integration tests are useful and worth considering adding to our
testing stack.

24.5 Continuous Integration

For projects of any size, we recommend setting up a continuous integration (CI) server to
run the project’s test suite whenever code is committed and pushed to the project repo. See
Chapter 34: Continuous Integration for more details.

24.6 Who Cares? We Don’t Have Time for Tests!

“Tests are the Programmer’s stone, transmuting fear into boredom.”

–Kent Beck

Let’s say you are confident of your coding skill and decide to skip testing to increase your
speed of development. Or maybe you feel lazy. It’s easy to argue that even with test genera-
tors and using tests instead of the shell, they can increase the time to get stuff done.

Oh, really?

What about when it’s time to upgrade?

That’s when the small amount of work you did up front to add tests saves you a lot of work.

For example, in the summer of 2010, Django 1.2 was the standard when we started Django
Packages (djangopackages.org). Since then the project has stayed current with new
Django versions, which has been really useful. Because of its pretty good test coverage,
moving it up a version of Django (or the various dependencies) has been easy. Our path to
upgrade:

ä Upgrade the version in a local instance of Django Packages.

ä Run the tests.
ä Fix any errors that are thrown by the tests.
ä Do some manual checking.

If Django Packages didn’t have tests, any time we upgraded anything we would have to click
through dozens and dozens of scenarios manually, which is error-prone. Having tests means
we can make changes and dependency upgrades with the confidence that our users (i.e. the
Django community) won’t have to deal with a buggy experience.

This is the benefit of having tests.

306 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 24.7: The Game of Test Coverage

24.7 The Game of Test Coverage

A great, fun game to play is trying get test coverage as high as possible. Every day that we
increase our test coverage is a victory, and every day that the coverage goes down is a loss.

24.8 Setting Up the Test Coverage Game

Yes, we call test coverage a game. It’s a good tool for developers to push themselves. It’s
also a nice metric that both developers and their clients/employers/investors can use to help
evaluate the status of a project.

We advocate following these steps because most of the time we want to only test our own
project’s apps, not all of Django, and the myriad of third-party libraries that are the building
blocks of our project. Testing those ‘building blocks’ takes an enormous amount of time,
which is a waste because most are already tested or require additional setup of resources.

24.8.1 Step 1: Start Writing Tests

We’ve done that already, right?

24.8.2 Step 2: Run Tests and Generate Coverage Report

Let’s try it out! In the command-line, at the <project_root>, type:

Example 24.8: Running Django tests using coverage.py

$ coverage run manage.py test --settings=twoscoops.settings.test

If we have nothing except for the default tests for two apps, we should get a response that
looks like:

Example 24.9: Positive Test Results

Creating test database for alias "default"...

..
-----------------------------------------------
Ran 2 tests in 0.008s

OK

Destroying test database for alias "default"...

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 307
Chapter 24: Testing Stinks and Is a Waste of Money!

This doesn’t look like much, but what it means is that we’ve constrained our application to
only run the tests that you want. Now it’s time to go and look at and analyze our embarrass-
ingly low test coverage numbers.

24.8.3 Step 3: Generate the Report!

coverage.py provides a very useful method for generating HTML reports that don’t just
provide percentage numbers of what’s been covered by tests, it also shows us the places
where code is not tested. In the command-line, at the <project_root>:

Example 24.10: Test Results Without admin.py

$ coverage html --omit="admin.py"

Ahem...don’t forget to change <project-root> to match the development machine’s structure!

For example, depending on where one does things, the <path-to-project-root> could be:

ä /Users/audreyr/code/twoscoops/twoscoops/
ä /Users/pydanny/projects/twoscoops/twoscoops/
ä c:\ twoscoops

After this runs, in the <project_root> directory there is a new directory called htmlcov/ . In
the htmlcov/ directory, open the index.html file using any browser.

What is seen in the browser is the test results for our test run. Unless we already wrote some
tests, the total on the front page will be in the single digits, if not at 0%. Click into the
various modules listed and we should see lots of code that’s red-colored. Red is bad.

Let’s go ahead and admit that our project has a low coverage total. If your project has a
low coverage total, you need to admit it as well. It’s okay just so long as we also resolve to
improve the coverage total.

In fact, there is nothing wrong in saying publicly that you are working to improve a project’s
test coverage. Then, other developers (including ourselves) will cheer you on!

24.9 Playing the Game of Test Coverage

The game has a single rule:

Mandate that no commit can lower test coverage.

308 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 24.10: Alternatives to unittest

If we add a feature or bug fix and coverage is 65% when we start, we can’t merge our code
in until coverage is at least 65% again. At the end of each day, if test coverage goes up by
any amount, it means we’re winning.

Keep in mind that the gradual increase of test coverage can be a very good thing over huge
jumps. Gradual increases can mean that we developers aren’t putting in bogus tests to bump
up coverage numbers; instead, we are improving the quality of the project.

24.10 Alternatives to unittest

All the examples in this chapter thus far have used the unittest library. While every known
authority on testing agrees that unittest is a very powerful, useful tool, not all of them like it.
The specific reason, and one we fully comprehend, is that it requires too much boilerplate.

Fortunately, there is the pytest alternatives that requires a lot less boilerplate: pypi.org/
project/pytest-django/

This library is a wrapper around the pytest library. In return for a little bit of extra setup,
this allows for not just the running of unittest-based tests, but also for running any function
(and class/directory/module) prefixed with “test_”. For example, you could write a simple
test that looks like this:

Example 24.11: py.test example

# test_models.py
from pytest import raises

from cones.models import Cone

def test_good_choice():
assert Cone.objects.filter(type='sugar').count() == 1

def test_bad_cone_choice():
with raises(Cone.DoesNotExist):
Cone.objects.get(type='spaghetti')

While this example is based on pytest, similar functionality can be used with nose and it’s
nose.tools.raises decorator.

A possible downside of the simplicity of these function-based tests is the lack of inheritance.
If a project needs to have similar behavior over a lot of different tests, then writing tests this
way may not make sense.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 309
Chapter 24: Testing Stinks and Is a Waste of Money!

24.11 Summary
All of this might seem silly, but testing can be very serious business. In a lot of developer
groups this subject, while gamified, is taken very seriously. Lack of stability in a project can
mean the loss of clients, contracts, and even employment.

In the next chapter we cover a common obsession of Python developers: documentation.

310 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
25 | Documentation: Be Obsessed

Given a choice between ice cream and writing great documentation, most Python developers
would probably choose to write the documentation. That being said, writing documentation
while eating ice cream is even better.

When you have great documentation tools like Markdown , MkDocs, Sphinx, Myst-
Parser for Sphinx, and you actually can’t help but want to add docs to your projects.

25.1 Use GitHub-Flavored Markdown for Docs

You’ll want to learn and follow the standard best practices for documentation. These days,
GitHub-Flavored Markdown (GFM) is the most common markup language used for doc-
umenting Python projects. Other tools and companies have adopted GFM, for example,
GitLab uses GFM in their READMEs and issue trackers.

Here is the formal GFM specification and a sample project which benefits from using it:

ä github.github.com/gfm/
ä django-rest-framework.org/

While it’s possible to study the formal documentation for GFM and learn the basics, here
is a quick primer of some very useful commands you should learn.

Example 25.1: Markdown Primer

# H1 Header

**emphasis (bold/strong)**

*italics*

_underline_

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 311
Chapter 25: Documentation: Be Obsessed

link: [Two Scoops Press](https://twoscoopspress.com)

Note: The secret to links in Markdown is think of it like a

,→ function call. The link in the paranthesis is the value being
,→ called.

## Section Header

#. An enumerated list item

#. Second item

- First bullet

- Second bullet

- Indented Bullet

- Note carriage return and indents

Literal code block:

``` python
def like():
print("I like Ice Cream")

for i in range(10):
like()
```

JavaScript colored code block:

``` js
console.log("Don't use alert()");
```

312 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 25.2: Use MkDocs or Sphinx with Myst to Generate Documentation From Markdown

25.2 Use MkDocs or Sphinx with Myst to Generate Docu-

mentation From Markdown
MkDocs and Sphinx with Myst are static site generators which render nice-looking docs
from your .md files. Output formats include HTML, LaTeX, manual pages, and plain text.

MkDocs should just work per the usage instructions at mkdocs.org/#getting-started

For Sphinx, you’ll need to follow the Myst instructions to generate markdown docs: https:
//myst-parser.readthedocs.io/en/latest/using/intro.html.

TIP: Build Your Documentation at Least Weekly

You never know when bad cross-references or invalid formatting can break the doc-
umentation build. Rather than discover that the documentation is unbuildable at an
awkward moment, just make a habit of creating it on a regular basis. Our preference
is to make it part of our Continuous Integration processes, as covered in Chapter 34:
Continuous Integration.

PACKAGE TIP: Other Documentation Generators

We list the two main Python-based documentation site generators, but if one is
willing to explore non-python options, there are numerous other open source and
commercial documentation alternatives. Here is a quick list of some excellent ones
we’ve used in the past few years:
ä docusaurus.io
ä docsify.js.org
ä bookdown.org

25.3 What Docs Should Django Projects Contain?

Developer-facing documentation refers to notes and guides that developers need in order
to set up and maintain a project. This includes notes on installation, deployment, architec-
ture, how to run tests or submit pull requests, and more. We’ve found that it really helps to
place this documentation in all our projects, private or public.

Here we provide a table that describes what we consider the absolute minimum documen-
tation:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 313
Chapter 25: Documentation: Be Obsessed

Filename or Directory Reason Notes

README.md Every coding project you Provide at least a short
begin regardless of frame- paragraph describing what
work or languae should the project does. Also, link
have a README.md file to the installation instruc-
in the repository root. tions in the docs/ directory.
docs/ Your project documenta- A simple directory.
tion should go in one, con-
sistent location. This is the
Python community stan-
dard.
docs/deployment.md This file lets you take a day A point-by-point set of in-
off. structions on how to instal-
l/update the project into
production, even if it’s done
via something powered by
various devops tools that
in theory make it “simple”,
document it here.
docs/installation.md This is really nice for new A point-by-point set of in-
people coming into a structions on how to on-
project or when you get a board yourself or another
new laptop and need to set developer with the soft-
up the project. ware setup for a project.
docs/architecture.md A guide for understanding This is how you imagine a
what things evolved from project to be in simple text
as a project ages and grows and it can be as long or
in scope. short as you want. Good
for keeping focused at the
beginning of an effort.

Table 25.1: Documentation Django Projects Should Contain

314 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 25.4: Additional Markdown Documentation Resources

Figure 25.1: Even ice cream could benefit from documentation.

25.4 Additional Markdown Documentation Resources

ä python.org/dev/peps/pep-0257 Official specification on docstrings.
ä readthedocs.io Read the Docs is a free service that can host your Sphinx or Mk-
Docs documentation.
ä pythonhosted.org Python Hosted is another free service for documentation host-
ing.
ä en.wikipedia.org/wiki/Markdown
ä documentup.com will host README documents written in Markdown format.

25.5 The ReStructuredText Alternative

ReStructuredText is a plain text formatting syntax not too dissimilar to Markdown. It has
a lot more built-in features than Markdown but is harder to learn and slower to write. It is
used by core tools such as Django, Python and many older third-party libraries .

25.5.1 ReStructuredText Resources

ä docutils.sourceforge.net/docs/ref/rst/restructuredtext.html is
the formal specification for reStructuredText.
ä docs.readthedocs.io/en/stable/intro/getting-started-with-sphinx.
html - ReadTheDoc’s starting instructions for Sphinx.
ä sphinx-doc.org/ is the Sphinx project home page. While usable with Markdown,
Sphinx really shines when used with ReStructuredText.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 315
Chapter 25: Documentation: Be Obsessed

25.6 When Documentation Needs to Be Convert to/from

Markdown or ReStructuredText
Pandoc is a command-line tool that allows us to convert files from one markup format
into another. We can write in one format, and quickly convert it to another. Here’s how to
convert files using this tool:

Example 25.2: Using Pandoc to convert between formats

\$ # To convert a ReStructuredText document to GitHub-Flavored

,→ Markdown
\$ pandoc -t gfm README.rst -o README.md
\$ # To convert a Markdown document to ReStructuredText
\$ pandoc -f gfm README.md -o README.rst

Pandoc’s documentation is found at pandoc.org.

25.7 Wikis and Other Documentation Methods

For whatever reason, if you can’t place developer-facing documentation in the project itself,
you should have other options. While wikis, online document stores, and word processing
documents don’t have the feature of being placed in version control, they are better than no
documentation.

Please consider creating documents within these other methods with the same names as the
ones we suggested in the table on the previous page.

25.8 Ensuring that Code is Documented

Even with unambigious naming patterns and type hints, it can be hard to determine what
a particular class, method, or function does. This is where docstrings have saved the day
for so many over the years. To make enforcing this level of documentation easier, use the
Interrogate library.

interrogate.readthedocs.io/

25.9 Summary
In this chapter we went over the following:

ä The use of Markdown to write documentation in plaintext format.

ä The use of static site generators to render your documentation in HTML. Some tools
enable rendering docs as epub, mobi, or PDF.
ä Advice on the documentation requirements for any Django project.

316 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 25.9: Summary

ä Using ReStructuredText as a documentation alternative.

ä Pandoc as a tool to convert between formats.
ä Enforcing documentation with Interrogate.

Next, we’ll take a look at common bottlenecks in Django projects and ways to deal with
them.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 317
Chapter 25: Documentation: Be Obsessed

318 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
26 | Finding and Reducing
Bottlenecks

WARNING: This Chapter is in Progress

We are in the midst of working on this chapter and will expand on it in the days to
come. We are open to suggestions on topics and items to cover, please submit them
to github.com/feldroy/two-scoops-of-django-3.x/issues

This chapter covers a few basic strategies for identifying bottlenecks and speeding up your
Django projects.

26.1 Should You Even Care?

Remember, premature optimization is bad. If your site is small- or medium-sized and the
pages are loading fine, then it’s okay to skip this chapter.

On the other hand, if your site’s user base is growing steadily or you’re about to land a
strategic partnership with a popular brand, then read on.

26.2 Speed Up Query-Heavy Pages

This section describes how to reduce bottlenecks caused by having too many queries, as well
as those caused by queries that aren’t as snappy as they could be.

We also urge you to read up on database access optimization in the official Django docs:
docs.djangoproject.com/en/3.2/topics/db/optimization/

26.2.1 Find Excessive Queries With Django Debug Toolbar

You can use django-debug-toolbar to help you determine where most of your queries are
coming from. You’ll find bottlenecks such as:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 319
Chapter 26: Finding and Reducing Bottlenecks

ä Duplicate queries in a page.

ä ORM calls that resolve to many more queries than you expected.
ä Slow queries.

You probably have a rough idea of some of the URLs to start with. For example, which
pages don’t feel snappy when they load?

Install django-debug-toolbar locally if you don’t have it yet. Look at your project in a web
browser, and expand the SQL panel. It’ll show you how many queries the current page
contains.

PACKAGE TIP: Packages for Profiling and Performance Analysis

django-debug-toolbar is a critical development tool and an invaluable aid in page-
by-page analysis. We also recommend adding django-cache-panel to your project,
but only configured to run when settings/local.py module is called. This will increase
visibility into what your cache is doing.

django-extensions comes with a tool called RunProfileServer that starts

Django’s runserver command with hotshot/profiling tools enabled.

silk (github.com/mtford90/silk) Silk is a live profiling Django app that inter-

cepts and stores HTTP requests and database queries before presenting them in a
user interface for further inspection.

26.2.2 Reduce the Number of Queries

Once you know which pages contain an undesirable number of queries, figure out ways to
reduce that number. Some of the things you can attempt:

ä Try using select_related() in your ORM calls to combine queries. It follows

ForeignKey relations and combines more data into a larger query. If using CBVs,
django-braces makes doing this trivial with the SelectRelatedMixin. Beware of
queries that get too large by explicitly passing the related field names you are interested
in. Only the specified relations will be followed. Combine that with careful testing!
ä For many-to-many and many-to-one relationships that can’t be optimized with
select_related(), explore using prefetch_related() instead.
ä If the same query is being generated more than once per template, move the query
into the Python view, add it to the context as a variable, and point the template ORM
calls at this new context variable.

320 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 26.2: Speed Up Query-Heavy Pages

ä Implement caching using a key/value store such as Memcached or Redis. Then

write tests to assert the number of queries run in a view. See
docs.djangoproject.com/en/3.2/topics/testing/tools/#django.
test.TransactionTestCase.assertNumQueries for instructions.
ä Use the django.utils.functional.cached_property decorator to cache in
memory the result of method call for the life of an object instance. This is incred-
ibly useful, so please see Section 31.3.5: django.utils.functional.cached_property in
chapter 31.

26.2.3 Speed Up Common Queries

The length of time it takes for individual queries can also be a bottleneck. Here are some
tips, but consider them just starting points:

ä Make sure your indexes are helping speed up your most common slow queries. Look
at the raw SQL generated by those queries, and index on the fields that you filter/sort
on most frequently. Look at the generated WHERE and ORDER_BY clauses.
ä Understand what your indexes are actually doing in production. Development ma-
chines will never perfectly replicate what happens in production, so learn how to
analyze and understand what’s really happening with your database.
ä Look at the query plans generated by common queries.
ä Turn on your database’s slow query logging feature and see if any slow queries occur
frequently.
ä Use django-debug-toolbar in development to identify potentially-slow queries defen-
sively, before they hit production.

Once you have good indexes, and once you’ve done enough analysis to know which queries
to rewrite, here are some starting tips on how to go about rewriting them:

1 Rewrite your logic to return smaller result sets when possible.

2 Re-model your data in a way that allows indexes to work more effectively.
3 Drop down to raw SQL in places where it would be more efficient than the generated
query.

TIP: Use EXPLAIN ANALYZE / EXPLAIN

If you’re using PostgreSQL, you can use EXPLAIN ANALYZE to get an extremely
detailed query plan and analysis of any raw SQL query. For more information, see:
ä revsys.com/writings/postgresql-performance.html
ä craigkerstiens.com/2013/01/10/more-on-postgres-performance/

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 321
Chapter 26: Finding and Reducing Bottlenecks

The MySQL equivalent is the EXPLAIN command, which isn’t as detailed but is still
helpful. For more information, see:
ä dev.mysql.com/doc/refman/5.7/en/explain.html

A nice feature of django-debug-toolbar is that the SQL pane has an EXPLAIN fea-
ture.

26.2.4 Switch ATOMIC_REQUESTS to False

The clear, vast majority of Django projects will run just fine with the setting of
ATOMIC_REQUESTS to True. Generally, the penalty of running all database queries in a
transaction isn’t noticeable. However, if your bottleneck analysis points to transactions caus-
ing too much delay, it’s time to change the project run as ATOMIC_REQUESTS to False. See
Section 7.7.2: Explicit Transaction Declaration for guidelines on this setting.

26.3 Get the Most Out of Your Database

You can go a bit deeper beyond optimizing database access. Optimize the database itself !
Much of this is database-specific and already covered in other books, so we won’t go into
too much detail here.

26.3.1 Know What Doesn’t Belong in the Database

Frank Wiles of Revolution Systems taught us that there are two things that should never
go into any large site’s relational database:

Logs. Don’t add logs to the database. Logs may seem OK on the surface, especially in devel-
opment. Yet adding this many writes to a production database will slow their performance.
When the ability to easily perform complex queries against your logs is necessary, we rec-
ommend third-party services such as Splunk or Loggly, or use of document-based NoSQL
databases.

Ephemeral data. Don’t store ephemeral data in the database. What this means is data that
requires constant rewrites is not ideal for use in relational databases. This includes examples
such as django.contrib.sessions, django.contrib.messages, and metrics. Instead, move this
data to things like Memcached, Redis, and other non-relational stores.

TIP: Frank Wiles on Binary Data in Databases

Actually, Frank says that there are three things to never store in a database, the
third item being binary data. Storage of binary data in databases is addressed by

322 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 26.4: Cache Queries With Memcached or Redis

django.db.models.FileField, which does the work of storing files on file

servers like AWS CloudFront or S3 for you. Exceptions to this are detailed in Sec-
tion 6.4.5: When to Use BinaryField.

26.3.2 Getting the Most Out of PostgreSQL

If using PostgreSQL, be certain that it is set up correctly in production. As this is outside
the scope of the book, we recommend the following articles:

ä wiki.postgresql.org/wiki/Detailed_installation_guides
ä wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
ä revsys.com/writings/postgresql-performance.html
ä craigkerstiens.com/2012/10/01/understanding-postgres-performance
ä craigkerstiens.com/2013/01/10/more-on-postgres-performance

26.3.3 Getting the Most Out of MySQL

It’s easy to get MySQL running, but optimizing production installations requires experience
and understanding. As this is outside the scope of this book, we recommend the following
links to help you:

ä TODO - list good resources

26.4 Cache Queries With Memcached or Redis

You can get a lot of mileage out of simply setting up Django’s built-in caching system with
Memcached or Redis. You will have to install one of these tools, install a package that
provides Python bindings for them, and configure your project.

You can easily set up the per-site cache, or you can cache the output of individual views or
template fragments. You can also use Django’s low-level cache API to cache Python objects.

Reference material:

ä docs.djangoproject.com/en/3.2/topics/cache/
ä github.com/niwinz/django-redis

26.5 Identify Specific Places to Cache

Deciding where to cache is like being first in a long line of impatient customers at Ben and
Jerry’s on free scoop day. You are under pressure to make a quick decision without being
able to see what any of the flavors actually look like.

Here are things to think about:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 323
Chapter 26: Finding and Reducing Bottlenecks

ä Which views/templates contain the most queries?

ä Which URLs are being requested the most?
ä When should a cache for a page be invalidated?

Let’s go over the tools that will help you with these scenarios.

26.6 Consider Third-Party Caching Packages

Third-party packages will give you additional features such as:

ä Caching of QuerySets.
ä Cache invalidation settings/mechanisms.
ä Different caching backends.
ä Alternative or experimental approaches to caching.

A few of the popular Django packages for caching are:

ä django-cacheops
ä django-cachalot

See djangopackages.org/grids/g/caching/ for more options.

WARNING: Third-Party Caching Libraries Aren’t Always the

Answer
Having tried many of the third-party Django cache libraries, we have to ask our
readers to test them very carefully and be prepared to drop them. They are cheap,
quick wins, but can lead to some hair-raising debugging efforts at the worst possible
times.

Cache invalidation is hard, and in our experience, magical cache libraries are better
for projects with more static content. By-hand caching is a lot more work, but leads
to better performance in the long run and doesn’t risk those terrifying moments.

26.7 Compression and Minification of HTML, CSS, and

JavaScript
When a browser renders a web page, it usually has to load HTML, CSS, JavaScript, and im-
age files. Each of these files consumes the user’s bandwidth, slowing down page loads. One
way to reduce bandwidth consumption is via compression and minification. Django even
provides tools for you: GZipMiddleware and the {% spaceless %} template tag. Through
the at-large Python community, we can even use WSGI middleware that performs the
same task.

324 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 26.8: Use Upstream Caching or a Content Delivery Network

The problem with making Django and Python do the work is that compression and minifica-
tion take up system resources, which can create bottlenecks of their own. A better approach
is to use Nginx or Apache web servers configured to compress the outgoing content. If you
are maintaining your own web servers, this is absolutely the way to go.

A common approach is to use a third-party compression module or Django library to com-

press and minify the HTML, CSS, and JavaScript in advance. Our preference is django-
pipeline which comes recommended by Django core developer Jannis Leidel.

For CSS and JavaScript, most people use JavaScript-powered tools for minification. Tools
like django-webpack-loader manage the JavaScript libraries within the Django context.
The advantage of this approach is the greater mindshare of tools and solved problems in
this domain space.

Tools and libraries to reference:

ä Apache and Nginx compression modules

ä django-webpack-loader
ä django-pipeline
ä django-compressor
ä django-htmlmin
ä Django’s built-in spaceless tag: docs.djangoproject.com/en/3.2/ref/
templates/builtins/spaceless
ä djangopackages.org/grids/g/asset-managers/

26.8 Use Upstream Caching or a Content Delivery

Network
Upstream caches such as Varnish are very useful. They run in front of your web server and
speed up web page or content serving significantly. See varnish-cache.org.

Content Delivery Networks (CDNs) like Fastly, Akamai, and Amazon Cloudfront serve
static media such as images, video, CSS, and JavaScript files. They usually have servers all
over the world, which serve out your static content from the nearest location. Using a CDN
rather than serving static content from your application servers can speed up your projects.

26.9 Other Resources

Advanced techniques on scaling, performance, tuning, and optimization are beyond the
scope of this book, but here are some starting points.

On general best practices for web performance:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 325
Chapter 26: Finding and Reducing Bottlenecks

ä TODO - add useful links

On scaling large Django sites:

ä “The Temple of Django Database Performance” is a book that dives deep into opti-
mizing Django projects for speed and scalability. It’s a delightful book full of fan-
tasy and RPG references and worth every penny. spellbookpress.com/books/
temple-of-django-database-performance/
ä Written with a focus on scaling Django, the book “High Performance Django” espouses
many good practices. Full tricks and tips, as well as questions in each section that
force you to think about what you are doing. Dated in places, but still full of useful
information. highperformancedjango.com
ä Watch videos of presentations from past DjangoCons and Py-
Cons about different developers’ experiences. Scaling prac-
tices vary from year to year and from company to company:
https://www.youtube.com/results?search_query=django+scaling

Figure 26.1: With your site running smoothly, you’ll be feeling as cool as a cone.

326 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 26.10: Summary

TIP: For Sites With High Volume: High Performance Django

We want to reiterate that “High Performance Django” is worth getting if your site
has enough traffic to cause issues. While it’s getting old, Peter Baumgartner and
Yann Malet wrote the book more at the conceptual level, making it a volume that
you should consider purchasing.
ä highperformancedjango.com
ä amazon.com/High-Performance-Django/dp/1508748128

26.10 Summary
In this chapter, we explored a number of bottleneck reduction strategies including:

ä Whether you should even care about bottlenecks in the first place.
ä Profiling your pages and queries.
ä Optimizing queries.
ä Using your database wisely.
ä Caching queries.
ä Identifying what needs to be cached.
ä Compression of HTML, CSS, and JavaScript.
ä Exploring other resources.

In the next chapter, we’ll cover various practices involving asynchronous task queues, which
may resolve our bottleneck problems.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 327
Chapter 26: Finding and Reducing Bottlenecks

328 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
27 | Asynchronous Task Queues

WARNING: This Chapter is in Progress

We are in the midst of working on this chapter and will expand on it in the days to
come. We are open to suggestions on topics and items to cover, please submit them
to github.com/feldroy/two-scoops-of-django-3.x/issues

An asynchronous task queue is one where tasks are executed at a different time from when
they are created, and possibly not in the same order they were created. Here is an example
of a human-powered asynchronous task queue:

1 In their spare time, Audrey and Daniel make ice cream cakes, taking orders from
friends and family. They use an issue tracker to track their tasks for scooping, spread-
ing, and decorating each cake.
2 Every so often, when they have spare time, they review the list of tasks and pick
one to do. Audrey prefers scooping and decorating, always doing those tasks first.
Daniel prefers scooping and spreading, finishing those before decorating. The result
is asynchronous completion of cake-making tasks.
3 As a cake-making task is completed and delivered, they mark the issue as closed.

TIP: Task Queue vs Asynchronous Task Queue

In the Django world, both terms are used to describe asynchronous task queue.
When someone writes task queue in the context of Django, they usually mean asyn-
chronous task queue.

Before we get into best practices, let’s go over some definitions:

Broker The storage for the tasks themselves. This can be implemented using any sort of

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 329
Chapter 27: Asynchronous Task Queues

persistence tool, although in Django the most common ones in use are RabbitMQ
and Redis. In the human-powered example, the storage is an online issue tracker.
Producer The code that adds tasks to the queue to be executed later. This is application
code, the stuff that makes up a Django project. In the human-powered example, this
would be Audrey and Daniel, plus anyone they can get to pitch in to help.
Worker The code that takes tasks from the broker and performs them. Usually there is more
than one worker. Most commonly each worker runs as a daemon under supervision.
In the human-powered example, this is Audrey and Daniel.
Serverless Usually provided by services such as AWS Lambda, this is, to paraphrase Mar-
tin Fowler, “where some amount of server-side logic is written by us but unlike tra-
ditional architectures is run in stateless compute containers that are event-triggered,
ephemeral (only last for one invocation), and fully managed by a 3rd party.” Server-
less takes over the role of Broker and Worker. In the human-powered example, it’s
as if Daniel and Audrey use a third-party service to take the orders and then follow
their precise instructions on doing the work.

27.1 Do We Need a Task Queue?

It depends. They add complexity but can improve user experience. Arguably it comes down
to whether a particular piece of code causes a bottleneck and can be delayed for later when
more free CPU cycles are available.

Here is a useful rule of thumb for determining if a task queue should be used:

Results take time to process: Task queue should probably be used.

Users can and should see results immediately: Task queue should not be used.

Let’s go over some possible use cases:

Issue Use Task Queue?

Sending bulk email Yes
Modifying files (including images) Yes
Fetching large amounts of data from third- Yes
party Ice Cream APIs
Inserting or updating a lot of records into Yes
a table
Updating a user profile No
Adding a blog or CMS entry No
Performing time-intensive calculations Yes
Sending or receiving of webhooks Yes

Table 27.1: Should a Project Have a Task Queue?

Please keep in mind there are site-traffic driven exceptions to all of these use cases:

330 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 27.1: Do We Need a Task Queue?

ä Sites with small-to-medium amounts of traffic may never need a task queue for any
of these actions.
ä Sites with larger amounts of traffic may discover that nearly every user action requires
use of a task queue.

Determining whether or not a site or action needs a task queue is a bit of an art. There is no
easy answer we can provide. However, knowing how to use them is a really powerful tool in
any developer’s toolkit.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 331
Chapter 27: Asynchronous Task Queues

332 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
28 | Security Best Practices

When it comes to security, Django has a pretty good record. This is due to security tools
provided by Django, solid documentation on the subject of security, and a thoughtful team
of core developers who are extremely responsive to security issues. However, it’s up to indi-
vidual Django developers such as ourselves to understand how to properly secure Django-
powered applications.

This chapter contains a list of things helpful for securing your Django application. This list
is by no means complete. Consider it a starting point.

TIP: What to Do if You Have a Security Breach

If you’re in the midst of a security crisis, please go to Chapter 37: Appendix G:
Handling Security Failures.

28.1 Reference Security Sections in Other Chapters

A number of other chapters in this book contain dedicated security sections, or touch on
security matters. These are found at the following locations:

ä Section 5.3: Separate Configuration From Code

ä Section 13.3: Always Use CSRF Protection With HTTP Forms That Modify Data
ä ??: ??
ä Section 28.28: Never Display Sequential Primary Keys
ä Section 21.8: Secure the Django Admin
ä Chapter 37: Appendix G: Handling Security Failures

28.2 Harden Your Servers

Search online for instructions and checklists for server hardening. Server hardening mea-
sures include but are not limited to things like setting up firewalls (help.ubuntu.com/
community/UFW), changing your SSH port, and disabling/removing unnecessary services.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 333
Chapter 28: Security Best Practices

28.3 Know Django’s Security Features

Django’s security features include:

ä Cross-site scripting (XSS) protection.

ä Cross-site request forgery (CSRF) protection.
ä SQL injection protection.
ä Clickjacking protection.
ä Support for TLS/HTTPS/HSTS, including secure cookies.
ä Automatic HTML escaping.
ä An expat parser hardened against XML bomb attacks.
ä Hardened JSON, YAML, and XML serialization/deserialization tools.
ä Secure password storage, using the PBKDF2 algorithm with a SHA256 hash by de-
fault. That said, please upgrade per Section 28.29: Upgrade Password Hasher to Ar-
gon2

Most of Django’s security features “just work” out of the box without additional configura-
tion, but there are certain things that you’ll need to configure. We’ve highlighted some of
these details in this chapter, but please make sure that you read the official Django documen-
tation on security as well: docs.djangoproject.com/en/3.2/topics/security/

28.4 Turn Off DEBUG Mode in Production

Your production site should not be running in DEBUG mode. Attackers can find out
more than they need to know about your production setup from a helpful DEBUG mode
stack trace page. For more information, see docs.djangoproject.com/en/3.2/ref/
settings/#debug.

Keep in mind that when you turn off DEBUG mode, you will need to set AL-
LOWED_HOSTS or risk raising a SuspiciousOperation error, which generates a 400
BAD REQUEST error that can be hard to debug. For more information on setting/debug-
ging ALLOWED_HOSTS see:

ä Section 28.7: Use Allowed Hosts Validation

ä ??: ??

28.5 Keep Your Secret Keys Secret

If the SECRET_KEY setting is not secret, depending on project setup, we risk an at-
tacker gaining control of other people’s sessions, resetting passwords, and more. Our API
keys and other secrets should be carefully guarded as well. These keys should not even be
kept in version control.

We cover the mechanics of how to keep your SECRET_KEY out of version control

334 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.6: HTTPS Everywhere

in Chapter 5: Settings and Requirements Files, Section 5.3: Separate Configuration From

Code, and Section 5.4: When You Can’t Use Environment Variables.

28.6 HTTPS Everywhere

Sites must always be deployed behind HTTPS. Not having HTTPS means that malicious
network users will sniff authentication credentials between your site and end users. In fact,
all data sent between your site and end users is up for grabs.

There is also no guarantee that any of your users are seeing what you expect them to see: an
attacker could manipulate anything in the request or the response. So HTTPS makes sense
even if all your information is public, but you do care about the integrity of the information.

Your entire site must be behind HTTPS. Your site’s static resources must also be served via
HTTPS, otherwise visitors will get warnings about “insecure resources” which will rightly
scare them away from your site. For reference, these warnings exist because they are a po-
tential man-in-the-middle vector.

TIP: Jacob Kaplan-Moss on HTTPS vs HTTP

Django co-leader Jacob Kaplan-Moss says, “Your whole site should only be available
via HTTPS, not HTTP at all. This prevents getting “firesheeped” (having a session
cookie stolen when served over HTTP). The cost is usually minimal.”

If visitors try to access your site via HTTP, they must be redirected to HTTPS. This
can be done either through configuration of your web server or with Django middleware.
Performance-wise, it’s better to do this at the web server level, but if you don’t have control
over your web server settings for this, then redirecting via Django middleware is fine.

In the modern era obtaining SSL certificates is cheap and easy. Depending on your plat-
form, it’s even provided for free. To set it up, follow the instructions for your particular
web server or platform-as-a-service. Our preferred services are the very reputable (and free)
letsencrypt.org and cloudflare.com. These services makes it trivial to secure sites
with proper SSL certicates.

Large cloud providers such as AWS, Google, and Microsoft also provide options.

WARNING: The Most Secure Option is to Set Up Your Own SSL

While services like CloudFlare, AWS, Google Cloud, and Azure make it easy to
set up SSL, they aren’t as secure as letsencrypt.org. We can best explain this
through the Principle of Least Privilege (POLP). This is the practice of limiting

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 335
Chapter 28: Security Best Practices

access rights for users to the bare minimum permissions they need to perform their
work.
While it is unlikely that a penetration will occur on these systems, either from an
outside or employee, it is not outside the realm of possibility.
For many systems the time benefit of using someone else’s SSL is worth it. However,
sites containing critical privacy information (HIPAA comes to mind) should set up
their own SSL.

WARNING: Not having HTTPS is Inexcusable

At the publication of this book it is 2020 and getting HTTPS/SSL into place is
relatively easy. Furthermore, browsers give users bold unavoidable warnings about
lack of SSL. Users of all ages and technical acumen are becoming knowledgeable of
what these warnings mean. Search engines harshly penalize sites without it.
Going forward there is no excuse for having a production site not on HTTPS. Don’t
do it, even for demos or MVPs.

TIP: Use django.middleware.security.SecurityMiddleware

The tool of choice for projects on Django for enforcing HTTPS/SSL across an
entire site through middleware is built right in. To activate this middleware just
follow these steps:
1 Add django.middleware.security.SecurityMiddleware to the

settings.MIDDLEWARE definition.
2 Set settings.SECURE_SSL_REDIRECT to True .

WARNING: django.middleware.security.SecurityMiddleware
Does Not Include static/media
Even if all Django requests are served over HTTPS, omitting HTTPS for resources
like javascript would still allow attackers to compromise your site.
As JavaScript, CSS, images, and other static assets are typically served directly by
the web server (nginx, Apache), make certain that serving of such content is done
via HTTPS. Providers of static assets such as Amazon S3 now do this by default.

336 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.6: HTTPS Everywhere

28.6.1 Use Secure Cookies

Your site should inform the target browser to never send cookies unless via HTTPS. You’ll
need to set the following in your settings:

Example 28.1: Securing Cookies

SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True

Read docs.djangoproject.com/en/3.2/topics/security/#ssl-https for more

details.

28.6.2 Use HTTP Strict Transport Security (HSTS)

HSTS can be configured at the web server level. Follow the instructions for your web server,
platform-as-a-service, and Django itself (via settings.SECURE_HSTS_SECONDS ).

If you have set up your own web servers, Wikipedia has sample HSTS configuration snippets
that you can use: en.wikipedia.org/wiki/HTTP_Strict_Transport_Security

When you enable HSTS, your site’s web pages include a HTTP header that tells HSTS-
compliant browsers to only connect to the site via secure connections:

ä HSTS-compliant browsers will redirect HTTP links to HTTPS.

ä If a secure connection isn’t possible (e.g. the certificate is self-signed or expired), an
error message will be shown and access will be disallowed.

To give you a better idea of how this works, here’s an example of what a HTTP Strict
Transport Security response header might look like:

Example 28.2: HSTS Response Header

Strict-Transport-Security: max-age=31536000; includeSubDomains

Some HSTS configuration advice:

1 Set max-age to a small value like 300 (5 minutes) during initial deployment of a
secured site to make sure you haven’t screwed something up or forgotten to make
some portion of the site available via HTTPS. We suggest this small value because
once you set max-age, we can’t unset it for users; their browsers control expiration,
not us.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 337
Chapter 28: Security Best Practices

2 Once it’s been confirmed the site is properly secured, set max-age to incrementally
larger values. Go to 1 hour (3600), 1 week (604800), 1 month (2592000), and finally
1 year (31536000). This allows us to correct issues that may not turn up in initial
checks.
3 Once the project’s max-age is set to 1 year or greater (31536000), submit the project
for HSTS preloading, which can be done at hstspreload.org/. This aids in block-
ing active attackers from intercepting the first request from a user to a site.

WARNING: Choose Your HSTS Policy Duration Carefully

Remember that HSTS is a one-way switch. It’s a declaration that for the next N
seconds, your site will be HTTPS-only. Don’t set a HSTS policy with a max-age
longer than you are able to maintain. Browsers do not offer an easy way to unset it.

Note that HSTS should be enabled in addition to redirecting all pages to HTTPS as de-
scribed earlier.

WARNING: Additional Warning for includeSubDomains

We recommend everyone to use HSTS with a long duration and to use
includeSubDomains. However, especially in projects with lots of legacy compo-
nents, the combination requires great care when configuring.

Example: Imagine we create a new Django website called example.com. The site
is HTTPS with HSTS enabled. We test the HSTS settings, which work fine,
and then increase the duration to a year. Alas, after a month, someone realises
legacy.example.com is still a production service and does not support HTTPS. We
remove includeSubdomains from the header, but by now it’s already too late: all
clients inside the company have the old HSTS header remembered.

In short, before even considering includeSubdomains, one should be entirely aware

of what might be hosted under the domain that HSTS is configured on.

28.6.3 HTTPS Configuration Tools

Mozilla provides a SSL configuration generator at the mozilla.github.io/
server-side-tls/ssl-config-generator/, which can provide a starting point for
your own configuration. While not perfect, it expedites setting up HTTPS. As our security
reviewers say, “In general, any HTTPS is better than plain HTTP.”

338 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.7: Use Allowed Hosts Validation

Once you have a server set up (preferably a test server), use the Qualys SSL Labs server test
at ssllabs.com/ssltest/index.html to see how well you did. A fun security game is
trying to score an A+. Especially as the official Two Scoops of Django reward for getting
that good of a grade is a trip to the local favorite ice cream saloon.

28.7 Use Allowed Hosts Validation

In production, you must set ALLOWED_HOSTS in your settings to a list of allowed
host/domain names in order to avoid raising SuspiciousOperation exceptions. This is a
security measure to prevent the use of fake HTTP host headers to submit requests.

We recommend that you avoid setting wildcard values here. For more information, read the
Django documentation on ALLOWED_HOSTS and the get_host() method:

ä docs.djangoproject.com/en/3.2/ref/settings/#allowed-hosts
ä docs.djangoproject.com/en/3.2/ref/request-response/#django.
http.HttpRequest.get_host

28.8 Always Use CSRF Protection With HTTP Forms

That Modify Data
Django comes with easy-to-use cross-site request forgery protection (CSRF) built in, and
by default it is activated across the site via the use of middleware. Make sure that your data
changing forms use the POST method, which by default will activate CSRF protection on
the server side. The only valid exception to use GET method is for search forms, where
it is useful for the user to have the argument visible in the URL. We have some strong
recommendations discussed in Section 13.3: Always Use CSRF Protection With HTTP
Forms That Modify Data.

28.9 Prevent Against Cross-Site Scripting (XSS) Attacks

XSS attacks usually occur when users enter malignant JavaScript that is then rendered into
a template directly. This isn’t the only method, but it is the most common. Fortunately for
us, Django by default escapes <, >, ’, ”, and &, which is all that is needed for proper HTML
escaping.

The following are recommended by the Django security team:

28.9.1 Use format_html Over mark_safe

Django gives developers the ability to mark content strings as safe, meaning that Django’s
own safeguards are taken away. A better alternative is django.utils.html.format_html ,
which is like Python’s str.format() method, except designed for building up HTML

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 339
Chapter 28: Security Best Practices

fragments. All args and kwargs are escaped before being passed to str.format() which
then combines the elements.

Reference: docs.djangoproject.com/en/3.2/ref/utils/#django.utils.html.
format_html

28.9.2 Don’t Allow Users to Set Individual HTML Tag Attributes

If you allow users to set individual attributes of HTML tags, that gives them a venue for
injecting malignant JavaScript.

28.9.3 Use JSON Encoding for Data Consumed by JavaScript

Rely on JSON encoding rather than finding ways to dump Python structures directly to
templates. It’s not just easier to integrate into client-side JavaScript, it’s safer. To do this,
always use the json_script filter.

Reference: docs.djangoproject.com/en/3.2/ref/templates/builtins/
#json-script

28.9.4 Beware Unusual JavaScript

Due to JavaScript’s weird semantics, it’s possible to construct syntactically-valid, executable
programs from a very tiny subset of characters. Per feld.to/unusual-javascript, it’s
possible to transform regular-looking JavaScript into an alphabet of only six characters (plus
sign, exclamation mark, open/close bracket and open/close parenthesis).

WARNING: NSFW: feld.to/unusual-javascript not safe for work

The URL that feld.to/unusual-javascript links to has a URL that may vi-
olate educational or corporate viewing policies.

28.9.5 Add Content Security Policy Headers

Also known as CSP, Content Security Policy provides a standard method to declare ap-
proved origins of content that browsers should be allowed to load on a website. Covered
types are JavaScript, CSS, HTML frames, web workers, fonts, images, embeddable objects
such as Java applets, ActiveX, audio and video files, and other HTML5 features. Option-
ally, CSP can be used with a violation-report URL collected by a project or managed by a
third-party service.

ä en.wikipedia.org/wiki/Content_Security_Policy

340 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.10: Defend Against Python Code Injection Attacks

ä github.com/mozilla/django-csp

28.9.6 Additional Reading

There are other avenues of attack that can occur, so educating yourself is important.

ä docs.djangoproject.com/en/3.2/ref/templates/builtins/#escape
ä en.wikipedia.org/wiki/Cross-site_scripting

28.10 Defend Against Python Code Injection Attacks

We once were hired to help with a project that had some security issues. The requests coming
into the site were being converted from django.http.HttpRequest objects directly into
strings via creative use of the str() function, then saved to a database table. Periodically,
these archived Django requests would be taken from the database and converted into Python
dicts via the eval() function. This meant that arbitrary Python code could be run on the
site at any time.

Needless to say, upon discovery the critical security flaw was quickly removed. This just
goes to show that no matter how secure Python and Django might be, we always need to
be aware that certain practices are incredibly dangerous.

28.10.1 Python Built-Ins That Execute Code

Beware of the eval() , exec() , and execfile() built-ins. If your project allows arbitrary
strings or files to be passed into any of these functions, you are leaving your system open to
attack.

For more information, read “Eval Really Is Dangerous” by Ned Batchelder:

nedbatchelder.com/blog/201206/eval_really_is_dangerous.html

28.10.2 Python Standard Library Modules That Can Execute Code

“Never unpickle data that could have come from an untrusted source, or that could
have been tampered with.”

– docs.python.org/3/library/pickle.html

You should not use the Python standard library’s pickle module to deserialize anything
which could have been modified by the user. As a general rule, avoid accepting pickled values
from user for any reason. Specific warnings about pickle and security are listed below:

ä lincolnloop.com/blog/playing-pickle-security/
ä blog.nelhage.com/2011/03/exploiting-pickle/

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 341
Chapter 28: Security Best Practices

28.10.3 Third-Party Libraries That Can Execute Code

When using PyYAML, only use safe_load() . While the use of YAML in the Python
and Django communities is rare outside of continuous integration, it’s not uncommon to
receive this format from other services. Therefore, if you are accepting YAML documents,
only load them with the yaml.safe_load() method.

For reference, the yaml.load() method will let you create Python objects,
which is really bad. As Ned Batchelder says, yaml.load() should be renamed to
yaml.dangerous_load() : nedbatchelder.com/blog/201302/war_is_peace.
html

28.10.4 Be Careful With Cookie-Based Sessions

Typically most Django sites use either database- or cache-based sessions. These function
by storing a hashed random value in a cookie which is used as a key to the real session
value, which is stored in the database or cache. The advantage of this is that only the key
to the session data is sent to the client, making it very challenging for malignant coders to
penetrate Django’s session mechanism.

However, Django sites can also be built using cookie-based sessions, which place the session
data entirely on the client’s machine. While this means slightly less storage needs for the
server, it comes with security issues that justify caution. Specifically:

1 It is possible for users to read the contents of cookie-based sessions.

2 If an attacker gains access to a project’s SECRET_KEY and your session serializer
is JSON-based, they gain the ability to falsify session data and therefore, if authenti-
cation is used, impersonate any user.
3 If an attacker gains access to a project’s SECRET_KEY and your session serializer
is pickle-based, they gain the ability to not only falsify session data, but also to execute
arbitrary code. In other words, not only can they assume new rights and privileges,
they can also upload working Python code. If you are using pickle-based sessions or
are considering using them, please read the tip below.
4 Another disadvantage of this configuration is that sessions can’t be invalidated in a
guaranteed way (except when they expire): you can try to override the cookie in the
browser with a new value, but you can’t enforce an attacker to use it: if they continue
sending requests with the old cookie, the session backend won’t know the difference.

342 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.11: Validate All Incoming Data With Django Forms

TIP: Use JSON for Cookie-Based Sessions

The default cookie serializer for Django is JSON-based, meaning that even if an
attacker discovers a project’s SECRET_KEY , they can’t execute arbitrary code.
If you decide to write your own cookie serializer, stick to using JSON as the format.
Never, ever use the optional pickle serializer.

Resources on the subject:

ä docs.djangoproject.com/en/3.2/topics/http/sessions/
#session-serialization
ä docs.djangoproject.com/en/3.2/ref/settings/
#session-serializer

Another thing to consider is that cookie-based sessions are a potential client-side perfor-
mance bottleneck. Transmitting the session data server-to-client is generally not an issue,
but client-to-server transmissions are much, much slower. This is literally the difference
between download and upload speeds all internet users encounter.

In general, we try to avoid cookie-based sessions.

Additional reading:

ä docs.djangoproject.com/en/3.2/topics/http/sessions/
#using-cookie-based-sessions
ä http://bit.ly/2plfHqU Threatpost.com article on cookies
ä yuiblog.com/blog/2007/03/01/performance-research-part-3/

28.11 Validate All Incoming Data With Django Forms

Django forms should be used to validate all data being brought into your project, including
from non-web sources. Doing so protects the integrity of our data and is part of securing
your application. We cover this in Section 13.1: Validate All Incoming Data With Django
Forms.

TIP: Using DRF Serializers Instead of Django Forms

Django REST Framework’s validation is as well constructed and secure as Django’s
form libraries. If you are more familiar with DRF, then using serializers to validate
all incoming data is perfectly okay.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 343
Chapter 28: Security Best Practices

28.12 Disable the Autocomplete on Payment Fields

You should disable the HTML field autocomplete browser feature on fields that are gate-
ways to payment. This includes credit card numbers, CVVs, PINs, credit card dates, etc.
The reason is that a lot of people use public computers or their personal computers in public
venues.

For reference, Django forms make this easy:

Example 28.3: Disabling Autocomplete in Form Fields

from django import forms

class SpecialForm(forms.Form):
my_secret = forms.CharField(
widget=forms.TextInput(attrs={'autocomplete': 'off'}))

For any site that might be used in a public area (an airport for example), consider changing
the form field itself to PasswordInput:

Example 28.4: Changing Public Widget to PasswordInput

from django import forms

class SecretInPublicForm(forms.Form):

my_secret = forms.CharField(widget=forms.PasswordInput())

28.13 Handle User-Uploaded Files Carefully

The only way to completely safely serve user-provided content is from a completely sepa-
rate domain. For better or worse, there are an infinite number of ways to bypass file type
validators. This is why security experts recommend the use of content delivery networks
(CDNs): they serve as a place to store potentially dangerous files.

If you must allow upload and download of arbitrary file types, make sure that the server
uses the “Content-Disposition: attachment” header so that browsers won’t display
the content inline.

28.13.1 When a CDN Is Not an Option

When this occurs, uploaded files must be saved to a directory that does not allow them to
be executed. In addition, at the very least make sure the HTTP server is configured to serve

344 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.13: Handle User-Uploaded Files Carefully

images with image content type headers, and that uploads are restricted to a whitelisted
subset of file extensions.

Take extra care with your web server’s configuration here, because a malicious user can try to
attack your site by uploading an executable file like a CGI or PHP script and then accessing
the URL. This won’t solve every problem, but it’s better than the defaults.

Consult your web server’s documentation for instructions on how to configure this, or con-
sult the documentation for your platform-as-a-service for details about how static assets and
user-uploaded files should be stored.

28.13.2 Django and User-Uploaded Files

Django has two model fields that allow for user uploads: FileField and ImageField.
They come with some built-in validation, but the Django docs also strongly advise you to
“pay close attention to where you’re uploading them and what type of files they are, to avoid
security holes.”

If you are only accepting uploads of certain file types, do whatever you can do to ensure that
the user is only uploading files of those types. For example, you can:

ä Use the python-magic library to check the uploaded file’s headers:

github.com/ahupp/python-magic
ä Validate the file with a Python library that specifically works with that file type. Unfor-
tunately this isn’t documented, but if you dig through Django’s ImageField source
code, you can see how Django uses PIL to validate that uploaded image files are in
fact images.
ä Use defusedxml instead of native Python XML libraries or lxml. See Section 28.21:
Guard Against XML Bombing With defusedxml.

WARNING: Custom Validators Aren’t the Answer Here

Don’t just write a custom validator and expect it to validate your uploaded files be-
fore dangerous things happen. Custom validators are run against field content after
they’ve already been coerced to Python by the field’s to_python() method.

If the contents of an uploaded file are malicious, any validation happening after
to_python() is executed may be too late.

Further research:

ä docs.djangoproject.com/en/3.2/ref/models/fields/#filefield

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 345
Chapter 28: Security Best Practices

ä youtube.com/watch?v=HS8KQbswZkU Tom Eastman’s PyCon AU talk on the

“The dangerous, exquisite art of safely handing user-uploaded files” is required watch-
ing if building a site handling file uploads.

28.14 Don’t Use ModelForms.Meta.exclude

When using ModelForms, always use Meta.fields. Never use Meta.exclude. The use
of Meta.exclude is considered a grave security risk, specifically a Mass Assignment Vul-
nerability. We can’t stress this strongly enough. Don’t do it.

One common reason we want to avoid the Meta.exclude attribute is that its behavior
implicitly allows all model fields to be changed except for those that we specify. When using
the excludes attribute, if the model changes after the form is written, we have to remember
to change the form. If we forget to change the form to match the model changes, we risk
catastrophe.

Let’s use an example to show how this mistake could be made. We’ll start with a simple ice
cream store model:

Example 28.5: Sample Store Model

# stores/models.py
from django.conf import settings
from django.db import models

class Store(models.Model):
title = models.CharField(max_length=255)
slug = models.SlugField()
owner = models.ForeignKey(settings.AUTH_USER_MODEL)
# Assume 10 more fields that cover address and contact info.

Here is the wrong way to define the ModelForm fields for this model:

Example 28.6: Implicit Definition of Form Fields

# DON'T DO THIS!
from django import forms

from .models import Store

class StoreForm(forms.ModelForm):

class Meta:

346 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.14: Don’t Use ModelForms.Meta.exclude

model = Store
# DON'T DO THIS: Implicit definition of fields.
# Too easy to make mistakes!
excludes = ("pk", "slug", "modified", "created", "owner")

In contrast, this is the right way to define the same ModelForm’s fields:

Example 28.7: Explicit Definition of Form Fields

from django import forms

from .models import Store

class StoreForm(forms.ModelForm):

class Meta:
model = Store
# Explicitly specifying the fields we want
fields = (
"title", "address_1", "address_2", "email",
"usstate", "postal_code", "city",
)

The first code example, as it involves less typing, appears to be the better choice. It’s not, as
when you add a new model field you now you need to track the field in multiple locations
(one model and one or more forms).

Let’s demonstrate this in action. Perhaps after launch we decide we need to have a way of
tracking store co-owners, who have all the same rights as the owner. They can access account
information, change passwords, place orders, and specify banking information. The store
model receives a new field as shown on the next page:

Example 28.8: Added Co-Owners Field

# stores/models.py
from django.conf import settings
from django.db import models

class Store(models.Model):
title = models.CharField(max_length=255)

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 347
Chapter 28: Security Best Practices

slug = models.SlugField()
owner = models.ForeignKey(settings.AUTH_USER_MODEL)
co_owners = models.ManyToManyField(settings.AUTH_USER_MODEL)
# Assume 10 more fields that cover address and contact info.

The first form code example which we warned against using relies on us remembering to
alter it to include the new co_owners field. If we forget, then anyone accessing that store’s
HTML form can add or remove co-owners. While we might remember a single form,
what if we have more than one ModelForm for a model? In complex applications this is not
uncommon.

On the other hand, in the second example, where we used Meta.fields we know exactly
what fields each form is designed to handle. Changing the model doesn’t alter what the form
exposes, and we can sleep soundly knowing that our ice cream store data is more secure.

28.14.1 Mass Assignment Vulnerabilities

The problem we describe in this section is a Mass Assignment Vulnerability.

These occur when the patterns such as Active Record, designed to empower developers,
create security risks for web applications. The solution is the approach we advocate in this
section, which is explicit definition of fields that can be modified.

See n.wikipedia.org/wiki/Mass_assignment_vulnerability for more detail.

28.15 Don’t Use ModelForms.Meta.fields = ”__all__”

This includes every model field in your model form. It’s a shortcut, and a dangerous one. It’s
very similar to what we describe in Section 28.14: Don’t Use ModelForms.Meta.exclude,
and even with custom validation code, exposes projects to form-based Mass Assignment
Vulnerabilities. We advocate avoiding this technique as much as possible, as we feel that it’s
simply impossible to catch all variations of input.

28.16 Beware of SQL Injection Attacks

The Django ORM generates properly-escaped SQL which will protect your site from users
attempting to execute malignant, arbitrary SQL code.

Django allows you to bypass its ORM and access the database more directly through raw
SQL. When using this feature, be especially careful to escape your SQL code properly. This
is of concern in these specific components of Django:

ä The .raw() ORM method.

348 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.17: Don’t Store Unnecessary Data

ä The .extra() ORM method.

ä Directly accessing the database cursor.

Reference:

ä docs.djangoproject.com/en/3.2/topics/security/
#sql-injection-protection

28.17 Don’t Store Unnecessary Data

There is data we should avoid storing for financial and legal reasons.

28.17.1 Never Store Credit Card Data

Unless you have a strong understanding of the PCI-DSS security standards
(pcisecuritystandards.org) and adequate time/resources/funds to validate your
PCI compliance, storing credit card data is too much of a liability and should be avoided.

Instead, we recommend using third-party services like Stripe, Braintree, Adyen, PayPal,
and others that handle storing this information for you, and allow you to reference the data
via special tokens. Most of these services have great tutorials, are very Python and Django
friendly, and are well worth the time and effort to incorporate into your project.

TIP: Educate Yourself on PCI Compliance

Ken Cochrane has written an excellent blog post on PCI
compliance. Please read kencochrane.net/blog/2012/01/
developers-guide-to-pci-compliant-web-applications/

TIP: Read the Source Code of Open Source E-Commerce

Solutions
If you are planning to use any of the existing open source Django e-commerce so-
lutions, examine how the solution handles payments. If credit card data is being
stored in the database, even encrypted, then please use another solution.

28.17.2 Don’t Store PII or PHI Unless Required (By Law)

PII is the abbreviation for Personally Identifying Information and PHI is the abbreviation
for Protected Health Information. This data is a key part of our online identity and can
be exploited to gain access to various accounts or worse. Using the United States as an
example, storing Social Security Numbers (SSN) and state ID numbers should be avoided

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 349
Chapter 28: Security Best Practices

unless required by law. Even if required by law, if it is possible to store the data in one-way
hashes, then by all means do so.

If PII is discovered within a project that doesn’t have a legal obligation to store it, raise it
as an issue immediately with the project owners. Having this data stolen through either
technical or social methods can be disastrous legally and financially for an organization. In
the case of Payment cards can be easily cancelled, the same cannot be said for PII.

In the case of PHI, which is medical data, our warnings are more dire. Specifically, ev-
eryone involved is at risk for civil and criminal prosecution under Title II of HIPAA. See
en.wikipedia.org/wiki/HIPAA#Security_Rule. While that rule is for US-based
projects, many countries have similar regulations. For example, the authors know someone
called to trial in Kenya over a PHI leak.

28.18 Monitor Your Sites

Check your web servers’ access and error logs regularly. Install monitoring tools and check
on them frequently. Keep an eye out for suspicious activity.

28.19 Keep Your Dependencies Up-to-Date

You should always update your projects to work with the latest stable release of Django and
third-party dependencies. This is particularly important when a release includes security
fixes. For that, we recommend pyup.io, which automatically checks requirements files
against the latest versions that PyPI provides.

‘I’ve set up (these kinds of services) for distinct actions: it mails me once a week
for each project with any outdated dependencies, and if it finds an insecure ver-
sion it automatically creates a pull request in GitHub, so tests run automatically
and I can deploy quickly.’

– Sasha Romijn, Django core dev and security reviewer for Two Scoops of
Django 1.8

Useful links for updates to Django itself.

ä The official Django weblog at djangoproject.com/weblog/

ä The official django-announce mailing list at groups.google.com/forum/#!
forum/django-announce

28.20 Prevent Clickjacking

Clickjacking is when a malicious site tricks users to click on a concealed element of another
site that they have loaded in a hidden frame or iframe. An example is a site with a false
social media ‘login’ button that is really a purchase button on another site.

350 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.21: Guard Against XML Bombing With defusedxml

Django has instructions and components to prevent this from happening:

ä docs.djangoproject.com/en/3.2/ref/clickjacking/

28.21 Guard Against XML Bombing With defusedxml

Attacks against XML libraries are nothing new. For example, the amusingly titled but dev-
astating ‘Billion Laughs’ attack (http://en.wikipedia.org/wiki/Billion_laughs)
was discovered in 2003.

Unfortunately, Python, like many other programming languages, doesn’t account for this or
other venues of attack via XML. Furthermore, third-party Python libraries such as lxml are
vulnerable to at least 4 well-known XML-based attacks. For a list of Python and Python
library vulnerabilities see https://feld.to/2KKwuMq.

Fortunately for us, Christian Heimes created defusedxml, a Python library designed to
patch Python’s core XML libraries and some of the third-party libraries (including lxml).

For more information, please read:

ä https://pypi.org/project/defusedxml

28.22 Explore Two-Factor Authentication

Two-factor authentication (2FA) requires users to authenticate by combining two separate
means of identification.

For modern web applications, what that usually means is you enter your password as well
as a value sent to your mobile device. Another option for One-Time Passwords (OTP) is
to register them in a password manager.

The advantage of 2FA is that it adds another component, one that changes frequently, to
the authentication process, great for any site involving personal identity, finance, or medical
requirements.

The downside is that the user needs to have a charged mobile device with access to a network
in order to log in to your site, making it not so ideal for users who may not have a charged
mobile device or easy access to a network.

ä en.wikipedia.org/wiki/Two_factor_authentication
ä https://pypi.org/project/django-two-factor-auth

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 351
Chapter 28: Security Best Practices

PACKAGE TIP: Look for TOTP in 2FA Products and Packages

TOTP is short for en.wikipedia.org/wiki/Time-based_One-time_
Password_Algorithm, which is an open standard used by Google Authenticator
and many other services. TOTP does not require network access, which is useful
for building certain kinds of Django projects. However, SMS implementations
require cellular network access or third-party services such as twilio.com.

WARNING: The Issue of 2FA Recovery

An important issue is how people recover from loss of their 2FA token or phone
number. Passwords are generally recovered by sending an e-mail with a secret link.
However, if the 2FA token can also be reset by e-mail, access to the user’s e-mail
has basically become the single factor of authentication. Common methods include
offering TOTP authentication with SMS as a fallback, or offering a number of
recovery codes that need to be kept by the user. In some cases, organisations will
only reset these tokens after receiving a scan of an identity card belonging to the
account holder. In any case, a recovery process will be needed, so think of this in
advance.

28.23 Embrace SecurityMiddleware

We’ve mentioned Django’s built-in django.middleware.security.SecurityMiddleware
several times already in this chapter. We owe it to ourselves and our users to embrace and
use this feature of Django.

28.24 Force the Use of Strong Passwords

A strong password is one that more than just a list of characters. It is long and preferably
complex, including punctuation, digits, and both character cases. Let’s pledge to protect our
users by enforcing the use of such passwords.

So what makes the best password?

Our opinion is that at this point in time, length is more important than complexity. An 8
character length password mixing cases, numbers, and special characters is easier by several
orders of magnitude to break than a 50-character sentence of just lower cased letters. What’s
even better is if you have a 30-50 character sentence that includes numbers, mixed cases,
and special characters.

Reference: xkcd.com/936/

352 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.25: Don’t Prevent Copy/Pasting of Password

Quality Password Specification

Bad 6-10 characters of just the alphabet
Okay Minimum 8 characters, mixed case + nu-
meric + special characters
Better Minimum 30 characters of just the alpha-
bet
Best Minimum 30 characters, mixed case + nu-
meric + special characters

Table 28.1: Password Strength: Length vs Complexity

28.25 Don’t Prevent Copy/Pasting of Password

There are sites that force users to enter passwords manually, rather than allow for copy/past-
ing. This is a terrible anti-pattern, one that encourages users to rely on easy-to-remember
or repeatedly used passwords rather than strong, unique passwords for every site generated
by secure password managers such as 1Password or LastPass.

28.26 Give Your Site a Security Checkup

There are a number of services that provide automated checkups for sites. They aren’t security
audits, but they are great, free ways to make certain that your production deployment doesn’t
have any gaping security holes.

pyup.io’s Safety library (github.com/pyupio/safety) checks your installed depen-

dencies for known security vulnerabilities. By default it uses the open Python vulnerability
database Safety DB, but can be upgraded to use pyup.io’s Safety API using the --key
option.

TODO - consider adding snikt or other commercial tools

Mozilla also provides a similar, but non-Django specific service called Observatory
(observatory.mozilla.org).

28.27 Put Up a Vulnerability Reporting Page

It’s a good idea to publish information on your site about how users can report security
vulnerabilities to you.

GitHub’s “Responsible Disclosure of Security Vulnerabilities” page is a good example of

this and rewards reporters of issues by publishing their names:
help.github.com/articles/responsible-disclosure-of-security-vulnerabilities/

28.28 Never Display Sequential Primary Keys

Displaying sequential primary keys is to be avoided because:

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 353
Chapter 28: Security Best Practices

ä They inform potential rivals or hackers of your volume

ä By displaying these values we make it trivial to exploit insecure direct object references
ä We also provide targets for XSS attacks

Here are some patterns for looking up records without revealing sequential identifiers:

28.28.1 Lookup by Slug

In the Django world, this is incredibly common. There are literally hundreds of examples
available on how to do it. This is the goto method for many Django projects. However, it
becomes a little challenging when you have issues with duplicate slugs. In which case, one
of the other methods apply.

28.28.2 UUIDs
Django comes with very useful models.UUIDField. While a use case for using them as
primary keys for large distributed systems exists, they also serve nicely for public lookups.
Here is a sample model:

Example 28.9: Using UUID for Public Lookups

import uuid
from django.db import models

class IceCreamPayment(models.Model):
uuid = models.UUIDField(
unique=True,
default=uuid.uuid4,
editable=False)

def __str__(self):
return str(self.pk)

And here is how we call that model:

Example 28.10: Looking Up Payment By UUID

>>> from payments import IceCreamPayment

>>> payment = IceCreamPayment()
>>> IceCreamPayment.objects.get(id=payment.id)
<IceCreamPayment: 1>
>>> payment.uuid

354 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.28: Never Display Sequential Primary Keys

UUID('0b0fb68e-5b06-44af-845a-01b6df5e0967')
>>> IceCreamPayment.objects.get(uuid=payment.uuid)
<IceCreamPayment: 1>

TIP: Using UUIDs as the Primary Key

Some people like to use UUIDs as the primary key. In fact, we’ve done so on several
projects over the years. Here are observations on doing so by us and members of the
Django security team:
UUIDs as primary keys simplifies the security of data. Instead of two fields (id and
uuid), there is just ”id”.
On the other hand, sequential IDs are possible for humans to remember, UUIDs
are not. If the only access to a model is through UUIDs, that makes working with
data a bit harder. To grab a particular record one typically needs the entire UUID
string.
Also, there are performance considerations to be aware of when using UUIDs for
lookups. Security is being traded for speed: percona.com/blog/2019/11/22/
uuids-are-popular-but-bad-for-performance-lets-discuss/.
What all of this tells us is that UUIDs for primary keys (and lookups) is something
that should be considered carefully. If a project is expected to handle billions of
records, an alternative approach (including sequential IDs) should be considered.

WARNING: The Dangers of Obfuscating Sequential IDs

Slugs and UUIDs both have their disadvantages. The slug-based approach runs
into collisions quickly, causing things like, “vanilla”, “vanilla-2”, “vanilla-3” to occur.
UUIDs, to put it simply, are long and not memorizable by most humans. What can
we do?
You can obfuscate the sequential ID. But we don’t recommend it. Why not?
The short answer: Obfuscating is not an effective way to hide sequential IDs.
The long answer: There are any number of methods for obfuscating numbers rang-
ing from base64 encoding to using the hashids library. These approaches
work by converting a number to a alphanumeric code and back again. They
not only hide the number, they also shorten it. Sounds great, right?
The problem is that every method of obfuscating sequential IDs is funda-
mentally insecure. Base64 encoding is trivial to undo. Libraries like hashids
can be broken with brute-forceattacks or by anyone with a good un-
derstanding of cryographic knowledge ( carnage.github.io/2015/08/

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 355
Chapter 28: Security Best Practices

cryptanalysis-of-hashids).
To summarize: If you want to hide your sequential identifiers, don’t rely on obfus-
cation.

28.29 Upgrade Password Hasher to Argon2

...Django’s default password hasher is PBKDF2. While this is still acceptable as a
lowest-common-denominator option, there are and for many years have been better
options available.

– James Bennett, Django Core Developer and security reviewer of Two Scoops
of Django.

PBKDF2 is Django’s default because it’s supported in the Python standard library and thus
requires no third-party packages. Argon2 is the best option that is natively supported by
Django. Installation instructions and references:

ä https://docs.djangoproject.com/en/3.2/topics/auth/passwords/
#using-argon2-with-django - Instructions on installing and using Argon2
with Django
ä https://docs.djangoproject.com/en/3.2/topics/auth/passwords/
#argon2 - Instructions on customizing Argon2 arguments (advanced and not
necessary for most projects)
ä latacora.micro.blog/2018/04/03/cryptographic-right-answers.
html - Easy-to-understand list of answers to questions non-security focused
developers get asked about cryptographic questions.

28.30 Use SRI When Loading Static Assets From External

Sources
It is not uncommon for projects to load libraries from external sources. For example, JQuery,
Google Fonts, Bootstrap Tailwindcss, and even React and Vue are often loaded from CDNs
like CDNJS, unpkg, Google Hosted Libraries, and others. Unfortunately, without the use
of SRI (Subresource Integrity) to confirm the source, attackers can replace or modify assets
used by users. In other words, don’t do this:

Example 28.11: Naively Loading Bootstrap Assets

<!-- DON'T DO THIS - loading static assets without SRI -->

<link rel="stylesheet"

356 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.30: Use SRI When Loading Static Assets From External Sources

,→ href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css">

<script
,→ src="https://code.jquery.com/jquery-3.3.1.slim.min.js"></script>
<script
,→ src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"></s
<script
,→ src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"></scri

This is why we recommend the use of subresource integrity hashes when including CSS,
JavaScript, or web fonts from external projects. This is provided as a feature by CDNJS, the
JQuery project, the Bootstrap project, and a shockingly small number of other projects.

From the official Bootstrap site, this is what using SRI looks like:

Example 28.12: Using SRI When Loading Bootstrap Assets

<!-- Loading Static Assets with SRI -->

<link rel="stylesheet"

,→ href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"

,→ integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T"
crossorigin="anonymous">
<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js"

,→ integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo"
crossorigin="anonymous"></script>
<script
,→ src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"

,→ integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1"
crossorigin="anonymous"></script>
<script
,→ src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"

,→ integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM"
crossorigin="anonymous"></script>

If a project doesn’t include SRI hashes with their installation instructions, we suggest you

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 357
Chapter 28: Security Best Practices

load their assets using CDNJS, which includes it as a feature in their “copy” dropdown. Just
choose the “copy link tag” option.

Recommended tools and useful references:

ä cdnjs.com - CDN that provides SRI hashes for all libraries they host
ä developer.mozilla.org/en-US/docs/Web/Security/Subresource_
Integrity - Mozilla’s documetation of SRI
ä w3.org/TR/SRI/ - Official specification

28.31 Reference Our Security Settings Appendix

Keeping track of everything that relates to security and Django is challenging. This chapter
alone is nigh 30 pages long and at the beginning we make it very clear this is not an absolute
reference.

In order to add clarity, we’ve created Appendix G: Security Settings Reference. This is where
we put important and useful information on how to better configure the security settings of
a Django project.

28.32 Review the List of Security Packages

In the security section of Appendix A: Packages, we list over ten related security packages
that can make a difference to your site. While some are listed in this chapter, others are
unique to that section of this book.

28.33 Keep Up-to-Date on General Security Practices

We end this chapter with some common-sense advice.

First, keep in mind that security practices are constantly evolving, both in the
Django community and beyond. Subscribe to groups.google.com/forum/#!forum/
django-announce and check Twitter, Hacker News, and various security blogs regularly.

Second, remember that security best practices extend well beyond those practices specific to
Django. You should research the security issues of every part of your web application stack,
and you should follow the corresponding sources to stay up to date.

TIP: Good Books and Articles on Security

Paul McMillan, Django core developer, security expert, and Two Scoops reviewer,
recommends the following books:
ä “The Tangled Web: A Guide to Securing Modern Web Applications”:
amzn.to/1hXAAyx
ä “The Web Application Hacker’s Handbook”:

358 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
 28.34: Summary

amzn.to/1dZ7xEY

In addition, we recommend the following reference site:

ä wiki.mozilla.org/WebAppSec/Secure_Coding_Guidelines

28.34 Summary
Please use this chapter as a starting point for Django security, not the ultimate reference
guide. See the Django documentation’s list for additional security topics:
docs.djangoproject.com/en/3.2/topics/security/
#additional-security-topics

Django comes with a good security record due to the diligence of its community and at-
tention to detail. Security is one of those areas where it’s a particularly good idea to ask for
help. If you find yourself confused about anything, ask questions and turn to others in the
Django community for help.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 359
Chapter 28: Security Best Practices

360 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322
29 | Logging: What’s It For, Anyway?

Logging is like rocky road ice cream. Either you can’t live without it, or you forget about it
and wonder once in awhile why it exists.

Anyone who’s ever worked on a large production project with intense demands understands
the importance of using the different log levels appropriately, creating module-specific log-
gers, meticulously logging information about important events, and including extra detail
about the application’s state when those events are logged.

While logging might not seem glamorous, remember that it is one of the secrets to building
extremely stable, robust web applications that scale and handle unusual loads gracefully.
Logging can be used not only to debug application errors, but also to track interesting
performance metrics.

Logging unusual activity and checking logs regularly is also important for ensuring the
security of your server. In the previous chapter, we covered the importance of checking your
server access and error logs regularly. Keep in mind that application logs can be used in
similar ways, whether to track failed login attempts or unusual application-level activity.

29.1 Application Logs vs. Other Logs

This chapter focuses on application logs. Any log file containing data logged from your
Python web application is considered an application log.

In addition to your application logs, you should be aware that there are other types of logs,
and that using and checking all of your server logs is necessary. Your server logs, database
logs, network logs, etc. all provide vital insight into your production system, so consider
them all equally important.

29.2 Why Bother With Logging?

Logging is your go-to tool in situations where a stack trace and existing debugging tools
aren’t enough. Whenever you have different moving parts interacting with each other or the
possibility of unpredictable situations, logging gives you insight into what’s going on.

Please submit
Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322 361
Chapter 29: Logging: What’s It For, Anyway?

The different log levels available to you are DEBUG, INFO, WARNING, ERROR, and CRITICAL.
Let’s now explore when it’s appropriate to use each logging level.

29.3 When to Use Each Log Level

In places other than your production environment, you might as well use all the log lev-
els. Log levels are controlled in your project’s settings modules, so we can fine tune this
recommendation as needed to account for load testing and large scale user tests.

In your production environment, we recommend using every log level except for DEBUG.

Figure 29.1: Appropriate usage of CRITICAL/ERROR/WARNING/INFO logging in

ice cream.

Since the same CRITICAL, ERROR, WARNING, and INFO logs are captured whether in pro-
duction or development, introspection of buggy code requires less modification of code. This
is important to remember, as debug code added by developers working to fix one problem
can create new ones.

The rest of this section covers how each log level is used.

29.3.1 Log Catastrophes With CRITICAL

Use the CRITICAL log level only when something catastrophic occurs that requires urgent
attention.

For example, if your code relies on an internal web service being available, and if that web
service is part of your site’s core functionality, then you might log at the CRITICAL level
anytime that the web service is inaccessible.

362 Please submit

Prepared exclusively issues to github.com/feldroy/two-
for LAKLAK scoops- of- django- 3.x/issues
LAKLAK ([email protected]) Transaction: 11322