The Definitive Guide to Django: Web Development Done Right

This book is about Django, a Web development framework that saves you time and makes Web development a joy. Using Django, you can build and maintain high-quality Web applications with minimal fuss.

At its best, Web development is an exciting, creative act; at its worst, it can be a repetitive, frustrating nuisance. Django lets you focus on the fun stuff — the crux of your Web application — while easing the pain of the repetitive bits. In doing so, it provides high-level abstractions of common Web development patterns, shortcuts for frequent programming tasks, and clear conventions for how to solve problems. At the same time, Django tries to stay out of your way, letting you work outside the scope of the framework as needed.

The goal of this book is to make you a Django expert. The focus is twofold. First, we explain, in depth, what Django does and how to build Web applications with it. Second, we discuss higher-level concepts where appropriate, answering the question “How can I apply these tools effectively in my own projects?” By reading this book, you’ll learn the skills needed to develop powerful Web sites quickly, with code that is clean and easy to maintain.

In this chapter, we provide a high-level overview of Django.

What Is a Web Framework?

Django is a prominent member of a new generation of Web frameworks. So what exactly does that term mean?

To answer that question, let’s consider the design of a Web application written using the Common Gateway Interface (CGI) standard, a popular way to write Web applications circa 1998. In those days, when you wrote a CGI application, you did everything yourself — the equivalent of baking a cake from scratch. For example, here’s a simple CGI script, written in Python, that displays the ten most recently published books from a database:

#!/usr/bin/python

import MySQLdb

print "Content-Type: text/html"
print
print "<html><head><title>Books</title></head>"
print "<body>"
print "<h1>Books</h1>"
print "<ul>"

connection = MySQLdb.connect(user='me', passwd='letmein', db='my_db')
cursor = connection.cursor()
cursor.execute("SELECT name FROM books ORDER BY pub_date DESC LIMIT 10")
for row in cursor.fetchall():
    print "<li>%s</li>" % row[0]

print "</ul>"
print "</body></html>"

connection.close()

This code is straightforward. First, it prints a “Content-Type” line, followed by a blank line, as required by CGI. It prints some introductory HTML, connects to a database and executes a query that retrieves the latest ten books. Looping over those books, it generates an HTML unordered list. Finally, it prints the closing HTML and closes the database connection.

With a one-off dynamic page such as this one, the write-it-from-scratch approach isn’t necessarily bad. For one thing, this code is simple to comprehend — even a novice developer can read these 16 lines of Python and understand all it does, from start to finish. There’s nothing else to learn; no other code to read. It’s also simple to deploy: just save this code in a file called latestbooks.cgi, upload that file to a Web server, and visit that page with a browser.

But as a Web application grows beyond the trivial, this approach breaks down, and you face a number of problems:

What happens when multiple pages need to connect to the database? Surely that database-connecting code shouldn’t be duplicated in each individual CGI script, so the pragmatic thing to do would be to refactor it into a shared function.
Should a developer really have to worry about printing the “Content-Type” line and remembering to close the database connection? This sort of boilerplate reduces programmer productivity and introduces opportunities for mistakes. These setup- and teardown-related tasks would best be handled by some common infrastructure.
What happens when this code is reused in multiple environments, each with a separate database and password? At this point, some environment-specific configuration becomes essential.
What happens when a Web designer who has no experience coding Python wishes to redesign the page? Ideally, the logic of the page — the retrieval of books from the database — would be separate from the HTML display of the page, so that a designer could edit the latter without affecting the former.

These problems are precisely what a Web framework intends to solve. A Web framework provides a programming infrastructure for your applications, so that you can focus on writing clean, maintainable code without having to reinvent the wheel. In a nutshell, that’s what Django does.

The MVC Design Pattern

Let’s dive in with a quick example that demonstrates the difference between the previous approach and that undertaken using a Web framework. Here’s how you might write the previous CGI code using Django:

# models.py (the database tables)

from django.db import models

class Book(models.Model):
    name = models.CharField(maxlength=50)
    pub_date = models.DateField()


# views.py (the business logic)

from django.shortcuts import render_to_response
from models import Book

def latest_books(request):
    book_list = Book.objects.order_by('-pub_date')[:10]
    return render_to_response('latest_books.html', {'book_list': book_list})


# urls.py (the URL configuration)

from django.conf.urls.defaults import *
import views

urlpatterns = patterns('',
    (r'latest/$', views.latest_books),
)


# latest_books.html (the template)

<html><head><title>Books</title></head>
<body>
<h1>Books</h1>
<ul>
{% for book in book_list %}
<li>{{ book.name }}</li>
{% endfor %}
</ul>
</body></html>

Don’t worry about the particulars of how this works just yet — we just want you to get a feel for the overall design. The main thing to note here is the separation of concerns:

The models.py file contains a description of the database table, as a Python class. This is called a model. Using this class, you can create, retrieve, update, and delete records in your database using simple Python code rather than writing repetitive SQL statements.
The views.py file contains the business logic for the page, in the latest_books() function. This function is called a view.
The urls.py file specifies which view is called for a given URL pattern. In this case, the URL /latest/ will be handled by the latest_books() function.
The latest_books.html is an HTML template that describes the design of the page.

Taken together, these pieces loosely follow the Model-View-Controller (MVC) design pattern. Simply put, MVC defines a way of developing software so that the code for defining and accessing data (the model) is separate from request routing logic (the controller), which in turn is separate from the user interface (the view).

A key advantage of such an approach is that components are loosely coupled. That is, each distinct piece of a Django-powered Web application has a single key purpose and can be changed independently without affecting the other pieces. For example, a developer can change the URL for a given part of the application without affecting the underlying implementation. A designer can change a page’s HTML without having to touch the Python code that renders it. A database administrator can rename a database table and specify the change in a single place, rather than having to search and replace through a dozen files.

In this book, each component of this stack gets its own chapter. For example, Chapter 3 covers views, Chapter 4 covers templates, and Chapter 5 covers models. Chapter 5 also discusses Django’s MVC philosophies in depth.

Django’s History

Before we dive into more code, we should take a moment to explain Django’s history. It’s helpful to understand why the framework was created, because a knowledge of the history will put into context why Django works the way it does.

If you’ve been building Web applications for a while, you’re probably familiar with the problems in the CGI example we presented earlier. The classic Web developer’s path goes something like this:

Write a Web application from scratch.
Write another Web application from scratch.
Realize the application from step 1 shares much in common with the application from step 2.
Refactor the code so that application 1 shares code with application 2.
Repeat steps 2-4 several times.
Realize you’ve invented a framework.

This is precisely how Django itself was created!

Django grew organically from real-world applications written by a Web development team in Lawrence, Kansas. It was born in the fall of 2003, when the Web programmers at the Lawrence Journal-World newspaper, Adrian Holovaty and Simon Willison, began using Python to build applications. The World Online team, responsible for the production and maintenance of several local news sites, thrived in a development environment dictated by journalism deadlines. For the sites — including LJWorld.com, Lawrence.com, and KUsports.com — journalists (and management) demanded that features be added and entire applications be built on an intensely fast schedule, often with only days’ or hours’ notice. Thus, Adrian and Simon developed a time-saving Web development framework out of necessity — it was the only way they could build maintainable applications under the extreme deadlines.

In summer 2005, after having developed this framework to a point where it was efficiently powering most of World Online’s sites, the World Online team, which now included Jacob Kaplan-Moss, decided to release the framework as open source software. They released it in July 2005 and named it Django, after the jazz guitarist Django Reinhardt.

Although Django is now an open source project with contributors across the planet, the original World Online developers still provide central guidance for the framework’s growth, and World Online contributes other important aspects such as employee time, marketing materials, and hosting/bandwidth for the framework’s Web site (http://www.djangoproject.com/).

This history is relevant because it helps explain two key matters. The first is Django’s “sweet spot.” Because Django was born in a news environment, it offers several features (particularly its admin interface, covered in Chapter 6) that are particularly well suited for “content” sites — sites like eBay, craigslist.org, and washingtonpost.com that offer dynamic, database-driven information. (Don’t let that turn you off, though — although Django is particularly good for developing those sorts of sites, that doesn’t preclude it from being an effective tool for building any sort of dynamic Web site. There’s a difference between being particularly effective at something and being ineffective at other things.)

The second matter to note is how Django’s origins have shaped the culture of its open source community. Because Django was extracted from real-world code, rather than being an academic exercise or commercial product, it is acutely focused on solving Web development problems that Django’s developers themselves have faced — and continue to face. As a result, Django itself is actively improved on an almost daily basis. The framework’s developers have a keen interest in making sure Django saves developers time, produces applications that are easy to maintain, and performs well under load. If nothing else, the developers are motivated by their own selfish desires to save themselves time and enjoy their jobs. (To put it bluntly, they eat their own dog food.)

How to Read This Book

In writing this book, we tried to strike a balance between readability and reference, with a bias toward readability. Our goal with this book, as stated earlier, is to make you a Django expert, and we believe the best way to teach is through prose and plenty of examples, rather than a providing an exhaustive but bland catalog of Django features. (As someone once said, you can’t expect to teach somebody how to speak merely by teaching them the alphabet.)

With that in mind, we recommend that you read Chapters 1 through 7 in order. They form the foundation of how to use Django; once you’ve read them, you’ll be able to build Django-powered Web sites. The remaining chapters, which focus on specific Django features, can be read in any order.

The appendixes are for reference. They, along with the free documentation at http://www.djangoproject.com/, are probably what you’ll flip back to occasionally to recall syntax or find quick synopses of what certain parts of Django do.

Required Programming Knowledge

Readers of this book should understand the basics of procedural and object-oriented programming: control structures (if, while, and for), data structures (lists, hashes/dictionaries), variables, classes, and objects.

Experience in Web development is, as you may expect, very helpful, but it’s not required to read this book. Throughout the book, we try to promote best practices in Web development for readers who lack this type of experience.

Required Python Knowledge

At its core, Django is simply a collection of libraries written in the Python programming language. To develop a site using Django, you write Python code that uses these libraries. Learning Django, then, is a matter of learning how to program in Python and understanding how the Django libraries work.

If you have experience programming in Python, you should have no trouble diving in. By and large, the Django code doesn’t perform “black magic” (i.e., programming trickery whose implementation is difficult to explain or understand). For you, learning Django will be a matter of learning Django’s conventions and APIs.

If you don’t have experience programming in Python, you’re in for a treat. It’s easy to learn and a joy to use! Although this book doesn’t include a full Python tutorial, it highlights Python features and functionality where appropriate, particularly when code doesn’t immediately make sense. Still, we recommend you read the official Python tutorial, available online at http://docs.python.org/tut/. We also recommend Mark Pilgrim’s free book Dive Into Python, available at http://www.diveintopython.org/ and published in print by Apress.

New Django Features

As we noted earlier, Django is frequently improved, and it will likely have a number of useful — even essential — new features by the time this book is published. Thus, our goal as authors of this book is twofold:

Make sure this book is as “future-proof” as possible, so that whatever you read here will still be relevant in future Django versions
Actively update this book on its Web site, http://www.djangobook.com/, so you can access the latest and greatest documentation as soon as we write it

If you want to implement something with Django that isn’t explained in this book, check the latest version of this book on the aforementioned Web site, and also check the official Django documentation.

Getting Help

One of the greatest benefits of Django is its kind and helpful user community. For help with any aspect of Django — from installation, to application design, to database design, to deployment — feel free to ask questions online.

The django-users mailing list is where thousands of Django users hang out to ask and answer questions. Sign up for free at http://www.djangoproject.com/r/django-users.
The Django IRC channel is where Django users hang out to chat and help each other in real time. Join the fun by logging on to #django on the Freenode IRC network.

What’s Next

In the next chapter, we’ll get started with Django, covering installation and initial setup.

Chapter 2: Getting Started

Chapter 3: The Basics of Dynamic Web Pages

In the previous chapter, we explained how to set up a Django project and run the Django development server. Of course, that site doesn’t actually do anything useful yet—all it does is display the “It worked!” message. Let’s change that. This chapter introduces how to create dynamic Web pages with Django.

Your First View: Dynamic Content

As our first goal, let’s create a Web page that displays the current date and time. This is a good example of a dynamic Web page, because the contents of the page are not static—rather, the contents change according to the result of a computation (in this case, a calculation of the current time). This simple example doesn’t involve a database or any sort of user input—just the output of your server’s internal clock.

To create this page, we’ll write a view function. A view function, or view for short, is simply a Python function that takes a Web request and returns a Web response. This response can be the HTML contents of a Web page, or a redirect, or a 404 error, or an XML document, or an image … or anything, really. The view itself contains whatever arbitrary logic is necessary to return that response. This code can live anywhere you want, as long as it’s on your Python path. There’s no other requirement—no “magic,” so to speak. For the sake of putting the code somewhere, let’s create a file called views.py in the mysite directory, which you created in the previous chapter.

Here’s a view that returns the current date and time, as an HTML document:

from django.http import HttpResponse
import datetime

def current_datetime(request):
    now = datetime.datetime.now()
    html = "<html><body>It is now %s.</body></html>" % now
    return HttpResponse(html)

Let’s step through this code one line at a time:

First, we import the class HttpResponse, which lives in the django.http module. See Appendix H for further details on the HttpRequest and HttpResponse objects.
Then we import the datetime module from Python’s standard library, the set of useful modules that comes with Python. The datetime module contains several functions and classes for dealing with dates and times, including a function that returns the current time.
Next, we define a function called current_datetime. This is the view function. Each view function takes an HttpRequest object as its first parameter, which is typically named request.

Note that the name of the view function doesn’t matter; it doesn’t have to be named in a certain way in order for Django to recognize it. We’re calling it current_datetime here, because that name clearly indicates what it does, but it could just as well be named super_duper_awesome_current_time, or something equally revolting. Django doesn’t care. The next section explains how Django finds this function.
The first line of code within the function calculates the current date/time, as a datetime.datetime object, and stores that as the local variable now.
The second line of code within the function constructs an HTML response using Python’s format-string capability. The %s within the string is a placeholder, and the percent sign after the string means “Replace the %s with the value of the variable now.” (Yes, the HTML is invalid, but we’re trying to keep the example simple and short.)
Finally, the view returns an HttpResponse object that contains the generated response. Each view function is responsible for returning an HttpResponse object. (There are exceptions, but we’ll get to those later.)

Django’s Time Zone

Django includes a TIME_ZONE setting that defaults to America/Chicago. This probably isn’t where you live, so you might want to change it in your settings.py. See Appendix E for details.

Mapping URLs to Views

So, to recap, this view function returns an HTML page that includes the current date and time. But how do we tell Django to use this code? That’s where URLconfs come in.

A URLconf is like a table of contents for your Django-powered Web site. Basically, it’s a mapping between URL patterns and the view functions that should be called for those URL patterns. It’s how you tell Django, “For this URL, call this code, and for that URL, call that code.” Remember that the view functions need to be on the Python path.

Your Python Path

Your Python path is the list of directories on your system where Python looks when you use the Python import statement.

For example, let’s say your Python path is set to ['', '/usr/lib/python2.4/site-packages', '/home/username/djcode/']. If you execute the Python code from foo import bar, Python will first check for a module called foo.py in the current directory. (The first entry in the Python path, an empty string, means “the current directory.”) If that file doesn’t exist, Python will look for the file /usr/lib/python2.4/site-packages/foo.py. If that file doesn’t exist, it will try /home/username/djcode/foo.py. Finally, if that file doesn’t exist, it will raise ImportError.

If you’re interested in seeing the value of your Python path, start the Python interactive interpreter and type import sys, followed by print sys.path.

Generally you don’t have to worry about setting your Python path—Python and Django will take care of things for you automatically behind the scenes. (If you’re curious, setting the Python path is one of the things that the manage.py file does.)

When you executed django-admin.py startproject in the previous chapter, the script created a URLconf for you automatically: the file urls.py. Let’s edit that file. By default, it looks something like this:

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    # Example:
    # (r'^mysite/', include('mysite.apps.foo.urls.foo')),

    # Uncomment this for admin:
#     (r'^admin/', include('django.contrib.admin.urls')),
)

Let’s step through this code one line at a time:

The first line imports all objects from the django.conf.urls.defaults module, including a function called patterns.
The second line calls the function patterns() and saves the result into a variable called urlpatterns. The patterns() function gets passed only a single argument—the empty string. The rest of the lines are commented out. (The string can be used to supply a common prefix for view functions, but we’ll skip this advanced usage for now.)

The main thing to note here is the variable urlpatterns, which Django expects to find in your ROOT_URLCONF module. This variable defines the mapping between URLs and the code that handles those URLs.

By default, everything in the URLconf is commented out—your Django application is a blank slate. (As a side note, that’s how Django knew to show you the “It worked!” page in the last chapter. If your URLconf is empty, Django assumes you just started a new project and, hence, displays that message.)

Let’s edit this file to expose our current_datetime view:

from django.conf.urls.defaults import *
from mysite.views import current_datetime

urlpatterns = patterns('',
    (r'^time/$', current_datetime),
)

We made two changes here. First, we imported the current_datetime view from its module (mysite/views.py, which translates into mysite.views in Python import syntax). Next, we added the line (r'^time/$', current_datetime),. This line is referred to as a URLpattern—it’s a Python tuple in which the first element is a simple regular expression and the second element is the view function to use for that pattern.

In a nutshell, we just told Django that any request to the URL /time/ should be handled by the current_datetime view function.

A few things are worth pointing out:

Note that, in this example, we passed the current_datetime view function as an object without calling the function. This is a key feature of Python (and other dynamic languages): functions are first-class objects, which means you can pass them around just like any other variables. Cool stuff, eh?
The r in r'^time/$' means that '^time/$ is a Python raw string. This allows regular expressions to be written without overly verbose escaping.
You should exclude the expected slash at the beginning of the '^time/$' expression in order to match /time/. Django automatically puts a slash before every expression. At first glance, this may seem odd, but URLconfs can be included in other URLconfs, and leaving off the leading slash simplifies matters. This is further covered in Chapter 8.
The caret character (^) and dollar sign character ($) are important. The caret means “require that the pattern matches the start of the string,” and the dollar sign means “require that the pattern matches the end of the string.”

This concept is best explained by example. If we had instead used the pattern '^time/' (without a dollar sign at the end), then any URL that starts with time/ would match, such as /time/foo and /time/bar, not just /time/. Similarly, if we had left off the initial caret character ('time/$'), Django would match any URL that ends with time/, such as /foo/bar/time/. Thus, we use both the caret and dollar sign to ensure that only the URL /time/ matches. Nothing more, nothing less.

You may be wondering what happens if someone requests /time. This is handled as you’d hope (via a redirect) as long as the APPEND_SLASH setting is True. (See Appendix E for some good bedtime reading on this topic.)

To test our changes to the URLconf, start the Django development server, as you did in Chapter 2, by running the command python manage.py runserver. (If you left it running, that’s fine, too. The development server automatically detects changes to your Python code and reloads as necessary, so you don’t have to restart the server between changes.) The server is running at the address http://127.0.0.1:8000/, so open up a Web browser and go to http://127.0.0.1:8000/time/. You should see the output of your Django view.

Hooray! You’ve made your first Django-powered Web page.

Regular Expressions

Regular expressions (or regexes) are a compact way of specifying patterns in text. While Django URLconfs allow arbitrary regexes for powerful URL-matching capability, you’ll probably use only a few regex patterns in practice. Here’s a small selection of common patterns:

Symbol	Matches
`.` (dot)	Any character
`\d`	Any digit
`[A-Z]`	Any character, A-Z (uppercase)
`[a-z]`	Any character, a-z (lowercase)
`[A-Za-z]`	Any character, a-z (case insensitive)
`+`	One or more of the previous expression (e.g., `\d+` matches one or more digit)
`[^/]+`	All characters except forward slash
`?`	Zero or more of the previous expression (e.g., `\d*` matches zero or more digits)
`{1,3}`	Between one and three (inclusive) of the previous expression

For more on regular expressions, see http://www.djangoproject.com/r/python/re-module/.

How Django Processes a Request

We should point out several things about what just happened. Here’s the nitty-gritty of what goes on when you run the Django development server and make requests to Web pages:

The command python manage.py runserver imports a file called settings.py from the same directory. This file contains all sorts of optional configuration for this particular Django instance, but one of the most important settings is ROOT_URLCONF. The ROOT_URLCONF setting tells Django which Python module should be used as the URLconf for this Web site.

Remember when django-admin.py startproject created the files settings.py and urls.py? Well, the autogenerated settings.py has a ROOT_URLCONF that points to the autogenerated urls.py. Convenient.
When a request comes in—say, a request to the URL /time/—Django loads the URLconf pointed to by the ROOT_URLCONF setting. Then it checks each of the URLpatterns in that URLconf in order, comparing the requested URL with the patterns one at a time, until it finds one that matches. When it finds one that matches, it calls the view function associated with that pattern, passing an HttpRequest object as the first parameter to the function. (More on HttpRequest later.)
The view function is responsible for returning an HttpResponse object.

You now know the basics of how to make Django-powered pages. It’s quite simple, really—just write view functions and map them to URLs via URLconfs. You might think it would be slow to map URLs to functions using a series of regular expressions, but you’d be surprised.

How Django Processes a Request: Complete Details

In addition to the straightforward URL-to-view mapping just described, Django provides quite a bit of flexibility in processing requests.

The typical flow—URLconf resolution to a view function which returns an HttpResponse—can be short-circuited or augmented via middleware. The deep secrets of middleware will be fully covered in Chapter 15, but a quick sketch (see Figure 3-1) should aid you in conceptually fitting the pieces together.

Figure 3-1: The complete flow of a Django request and response.

When an HTTP request comes in from the browser, a server-specific handler constructs the HttpRequest passed to later components and handles the flow of the response processing.

The handler then calls any available Request or View middleware. These types of middleware are useful for augmenting incoming HttpRequest objects as well as providing special handling for specific types of requests. If either returns an HttpResponse, processing bypasses the view.

Bugs slip by even the best programmers, but exception middleware can help squash them. If a view function raises an exception, control passes to the Exception middleware. If this middleware does not return an HttpResponse, the exception is re-raised.

Even then, all is not lost. Django includes default views that create a friendly 404 and 500 response.

Finally, response middleware is good for post-processing an HttpResponse just before it’s sent to the browser or doing cleanup of request-specific resources.

URLconfs and Loose Coupling

Now’s a good time to highlight a key philosophy behind URLconfs and behind Django in general: the principle of loose coupling. Simply put, loose coupling is a software-development approach that values the importance of making pieces interchangeable. If two pieces of code are loosely coupled, then changes made to one of the pieces will have little or no effect on the other.

Django’s URLconfs are a good example of this principle in practice. In a Django Web application, the URL definitions and the view functions they call are loosely coupled; that is, the decision of what the URL should be for a given function, and the implementation of the function itself, reside in two separate places. This lets a developer switch out one piece without affecting the other.

In contrast, other Web development platforms couple the URL to the program. In typical PHP (http://www.php.net/) applications, for example, the URL of your application is designated by where you place the code on your filesystem. In early versions of the CherryPy Python Web framework (http://www.cherrypy.org/), the URL of your application corresponded to the name of the method in which your code lived. This may seem like a convenient shortcut in the short term, but it can get unmanageable in the long run.

For example, consider the view function we wrote earlier, which displays the current date and time. If we wanted to change the URL for the application— say, move it from /time/ to /currenttime/—we could make a quick change to the URLconf, without having to worry about the underlying implementation of the function. Similarly, if we wanted to change the view function—altering its logic somehow—we could do that without affecting the URL to which the function is bound. Furthermore, if we wanted to expose the current-date functionality at several URLs, we could easily take care of that by editing the URLconf, without having to touch the view code.

That’s loose coupling in action. We’ll continue to point out examples of this important philosophy throughout this book.

404 Errors

In our URLconf thus far, we’ve defined only a single URLpattern: the one that handles requests to the URL /time/. What happens when a different URL is requested?

To find out, try running the Django development server and hitting a page such as http://127.0.0.1:8000/hello/ or http://127.0.0.1:8000/does-not-exist/, or even http://127.0.0.1:8000/ (the site “root”). You should see a “Page not found” message (see Figure 3-2). (Pretty, isn’t it? We Django people sure do like our pastel colors.) Django displays this message because you requested a URL that’s not defined in your URLconf.

Figure 3-2. Django’s 404 page

The utility of this page goes beyond the basic 404 error message; it also tells you precisely which URLconf Django used and every pattern in that URLconf. From that information, you should be able to tell why the requested URL threw a 404.

Naturally, this is sensitive information intended only for you, the Web developer. If this were a production site deployed live on the Internet, we wouldn’t want to expose that information to the public. For that reason, this “Page not found” page is only displayed if your Django project is in debug mode. We’ll explain how to deactivate debug mode later. For now, just know that every Django project is in debug mode when you first create it, and if the project is not in debug mode, a different response is given.

Your Second View: Dynamic URLs

In our first view example, the contents of the page—the current date/time— were dynamic, but the URL (/time/) was static. In most dynamic Web applications, though, a URL contains parameters that influence the output of the page.

Let’s create a second view that displays the current date and time offset by a certain number of hours. The goal is to craft a site in such a way that the page /time/plus/1/ displays the date/time one hour into the future, the page /time/plus/2/ displays the date/time two hours into the future, the page /time/plus/3/ displays the date/time three hours into the future, and so on.

A novice might think to code a separate view function for each hour offset, which might result in a URLconf like this:

urlpatterns = patterns('',
    (r'^time/$', current_datetime),
    (r'^time/plus/1/$', one_hour_ahead),
    (r'^time/plus/2/$', two_hours_ahead),
    (r'^time/plus/3/$', three_hours_ahead),
    (r'^time/plus/4//$', four_hours_ahead),
)

Clearly, this line of thought is flawed. Not only would this result in redundant view functions, but also the application is fundamentally limited to supporting only the predefined hour ranges—one, two, three, or four hours. If, all of a sudden, we wanted to create a page that displayed the time five hours into the future, we’d have to create a separate view and URLconf line for that, furthering the duplication and insanity. We need to do some abstraction here.

A Word About Pretty URLs

If you’re experienced in another Web development platform, such as PHP or Java, you may be thinking, “Hey, let’s use a query string parameter!”, something like /time/plus?hours=3, in which the hours would be designated by the hours parameter in the URL’s query string (the part after the ?).

You can do that with Django (and we’ll tell you how later, if you really must know), but one of Django’s core philosophies is that URLs should be beautiful. The URL /time/plus/3/ is far cleaner, simpler, more readable, easier to recite to somebody aloud and … just plain prettier than its query string counterpart. Pretty URLs are a sign of a quality Web application.

Django’s URLconf system encourages pretty URLs by making it easier to use pretty URLs than not to.

Wildcard URLpatterns

Continuing with our hours_ahead example, let’s put a wildcard in the URLpattern. As we mentioned previously, a URLpattern is a regular expression; hence, we can use the regular expression pattern \d+ to match one or more digits:

from django.conf.urls.defaults import *
from mysite.views import current_datetime, hours_ahead

urlpatterns = patterns('',
    (r'^time/$', current_datetime),
    (r'^time/plus/\d+/$', hours_ahead),
)

This URLpattern will match any URL such as /time/plus/2/, /time/plus/25/, or even /time/plus/100000000000/. Come to think of it, let’s limit it so that the maximum allowed offset is 99 hours. That means we want to allow either one- or two-digit numbers—in regular expression syntax, that translates into \d{1,2}:

(r'^time/plus/\d{1,2}/$', hours_ahead),

Note

When building Web applications, it’s always important to consider the most outlandish data input possible, and decide whether or not the application should support that input. We’ve curtailed the outlandishness here by limiting the offset to 99 hours. And, by the way, The Outlandishness Curtailers would be a fantastic, if verbose, band name.

Now that we’ve designated a wildcard for the URL, we need a way of passing that data to the view function, so that we can use a single view function for any arbitrary hour offset. We do this by placing parentheses around the data in the URLpattern that we want to save. In the case of our example, we want to save whatever number was entered in the URL, so let’s put parentheses around the \d{1,2}:

(r'^time/plus/(\d{1,2})/$', hours_ahead),

If you’re familiar with regular expressions, you’ll be right at home here; we’re using parentheses to capture data from the matched text.

The final URLconf, including our previous current_datetime view, looks like this:

from django.conf.urls.defaults import *
from mysite.views import current_datetime, hours_ahead

urlpatterns = patterns('',
    (r'^time/$', current_datetime),
    (r'^time/plus/(\d{1,2})/$', hours_ahead),
)

With that taken care of, let’s write the hours_ahead view.

Coding Order

In this example, we wrote the URLpattern first and the view second, but in the previous example, we wrote the view first, then the URLpattern. Which technique is better? Well, every developer is different.

If you’re a big-picture type of person, it may make the most sense to you to write all of the URLpatterns for your application at the same time, at the start of your project, and then code up the views. This has the advantage of giving you a clear to-do list, and it essentially defines the parameter requirements for the view functions you’ll need to write.

If you’re more of a bottom-up developer, you might prefer to write the views first, and then anchor them to URLs afterward. That’s OK, too.

In the end, it comes down to which technique fits your brain the best. Both approaches are valid.

hours_ahead is very similar to the current_datetime view we wrote earlier, with a key difference: it takes an extra argument, the number of hours of offset. Add this to views.py:

def hours_ahead(request, offset):
    offset = int(offset)
    dt = datetime.datetime.now() + datetime.timedelta(hours=offset)
    html = "<html><body>In %s hour(s), it will be %s.</body></html>" % (offset, dt)
    return HttpResponse(html)

Let’s step through this code one line at a time:

Just as we did for our current_datetime view, we import the class django.http.HttpResponse and the datetime module.
The view function, hours_ahead, takes two parameters: request and offset.
- request is an HttpRequest object, just as in current_datetime. We’ll say it again: each view always takes an HttpRequest object as its first parameter.
- offset is the string captured by the parentheses in the URLpattern. For example, if the requested URL were /time/plus/3/, then offset would be the string '3'. If the requested URL were /time/plus/21/, then offset would be the string '21'. Note that captured strings will always be strings, not integers, even if the string is composed of only digits, such as '21'.
  
  We decided to call the variable offset, but you can call it whatever you’d like, as long as it’s a valid Python identifier. The variable name doesn’t matter; all that matters is that it’s the second argument to the function (after request). It’s also possible to use keyword, rather than positional, arguments in an URLconf. We cover that in Chapter 8.
The first thing we do within the function is call int() on offset. This converts the string value to an integer.

Note that Python will raise a ValueError exception if you call int() on a value that cannot be converted to an integer, such as the string 'foo'. However, in this example we don’t have to worry about catching that exception, because we can be certain offset will be a string containing only digits. We know that because the regular-expression pattern in our URLconf— (\d{1,2})—captures only digits. This illustrates another nicety of URLconfs: they provide a fair level of input validation.
The next line of the function shows why we called int() on offset. On this line, we calculate the current time plus a time offset of offset hours, storing the result in dt. The datetime.timedelta function requires the hours parameter to be an integer.
Next, we construct the HTML output of this view function, just as we did in current_datetime. A small difference in this line from the previous line is that it uses Python’s format-string capability with two values, not just one. Hence, there are two %s symbols in the string and a tuple of values to insert: (offset, dt).
Finally, we return an HttpResponse of the HTML—again, just as we did in current_datetime.

With that view function and URLconf written, start the Django development server (if it’s not already running), and visit http://127.0.0.1:8000/time/plus/3/ to verify it works. Then try http://127.0.0.1:8000/time/plus/5/. Then http://127.0.0.1:8000/time/plus/24/. Finally, visit http://127.0.0.1:8000/time/plus/100/ to verify that the pattern in your URLconf only accepts one- or two-digit numbers; Django should display a “Page not found” error in this case, just as we saw in the “404 Errors” section earlier. The URL http://127.0.0.1:8000/time/plus/ (with no hour designation) should also throw a 404.

If you’re following along while coding at the same time, you’ll notice that the views.py file now contains two views. (We omitted the current_datetime view from the last set of examples for clarity.) Put together, views.py should look like this:

from django.http import HttpResponse
import datetime

def current_datetime(request):
    now = datetime.datetime.now()
    html = "<html><body>It is now %s.</body></html>" % now
    return HttpResponse(html)

def hours_ahead(request, offset):
    offset = int(offset)
    dt = datetime.datetime.now() + datetime.timedelta(hours=offset)
    html = "<html><body>In %s hour(s), it will be %s.</body></html>" % (offset, dt)
    return HttpResponse(html)

Django’s Pretty Error Pages

Take a moment to admire the fine Web application we’ve made so far … now let’s break it! We’ll deliberately introduce a Python error into our views.py file by commenting out the offset = int(offset) line in the hours_ahead view:

def hours_ahead(request, offset):
    #offset = int(offset)
    dt = datetime.datetime.now() + datetime.timedelta(hours=offset)
    html = "<html><body>In %s hour(s), it will be %s.</body></html>" % (offset, dt)
    return HttpResponse(html)

Load up the development server and navigate to /time/plus/3/. You’ll see an error page with a significant amount of information, including a TypeError message displayed at the very top: "unsupported type for timedelta hours component: str".

What happened? Well, the datetime.timedelta function expects the hours parameter to be an integer, and we commented out the bit of code that converted offset to an integer. That caused datetime.timedelta to raise the TypeError. It’s the typical kind of small bug that every programmer runs into at some point.

The point of this example was to demonstrate Django’s error pages. Take some time to explore the error page and get to know the various bits of information it gives you.

Here are some things to notice:

At the top of the page, you get the key information about the exception: the type of exception, any parameters to the exception (the "unsupported type" message in this case), the file in which the exception was raised, and the offending line number.
Under the key exception information, the page displays the full Python traceback for this exception. This is similar to the standard traceback you get in Python’s command-line interpreter, except it’s more interactive. For each frame in the stack, Django displays the name of the file, the function/method name, the line number, and the source code of that line.

Click the line of source code (in dark gray), and you’ll see several lines from before and after the erroneous line, to give you context.

Click “Local vars” under any frame in the stack to view a table of all local variables and their values, in that frame, at the exact point in the code at which the exception was raised. This debugging information is invaluable.
Note the “Switch to copy-and-paste view” text under the “Traceback” header. Click those words, and the traceback will switch to a alternate version that can be easily copied and pasted. Use this when you want to share your exception traceback with others to get technical support— such as the kind folks in the Django IRC chat room or on the Django users mailing list.
Next, the “Request information” section includes a wealth of information about the incoming Web request that spawned the error: GET and POST information, cookie values, and meta information, such as CGI headers. Appendix H has a complete reference of all the information a request object contains.

Below the “Request information” section, the “Settings” section lists all of the settings for this particular Django installation. All the available settings are covered in detail in Appendix E. For now, take a look at the settings to get an idea of the information available.

The Django error page is capable of displaying more information in certain special cases, such as the case of template syntax errors. We’ll get to those later, when we discuss the Django template system. For now, uncomment the offset = int(offset) line to get the view function working properly again.

Are you the type of programmer who likes to debug with the help of carefully placed print statements? You can use the Django error page to do so—just without the print statements. At any point in your view, temporarily insert an assert False to trigger the error page. Then, you can view the local variables and state of the program. (There’s a more advanced way to debug Django views, which we’ll explain later, but this is the quickest and easiest.)

Finally, it’s obvious that much of this information is sensitive—it exposes the innards of your Python code and Django configuration—and it would be foolish to show this information on the public Internet. A malicious person could use it to attempt to reverse-engineer your Web application and do nasty things. For that reason, the Django error page is only displayed when your Django project is in debug mode. We’ll explain how to deactivate debug mode later. For now, just know that every Django project is in debug mode automatically when you start it. (Sound familiar? The “Page not found” errors, described in the “404 Errors” section, work the same way.)

What’s next?

We’ve so far been producing views by hard-coding HTML into the Python code. Unfortunately, this is nearly always a bad idea. Luckily, Django ships with a simple yet powerful template engine that allows you to separate the design of the page from the underlying code. We’ll dive into Django’s template engine in the next chapter.

Chapter 4: The Django Template System

In the previous chapter, you may have noticed something peculiar in how we returned the text in our example views. Namely, the HTML was hard-coded directly in our Python code.

This arrangement leads to several problems:

Any change to the design of the page requires a change to the Python code. The design of a site tends to change far more frequently than the underlying Python code, so it would be convenient if the the design could change without needing to modify the Python code.
Writing Python code and designing HTML are two different disciplines, and most professional Web development environments split these responsibilities between separate people (or even separate departments). Designers and HTML/CSS coders shouldn’t have to edit Python code to get their job done; they should deal with HTML.
Similarly, it’s most efficient if programmers can work on Python code and designers can work on templates at the same time, rather than one person waiting for the other to finish editing a single file that contains both Python and HTML.

For these reasons, it’s much cleaner and more maintainable to separate the design of the page from the Python code itself. We can do this with Django’s template system, which we discuss in this chapter.

Template System Basics

A Django template is a string of text that is intended to separate the presentation of a document from its data. A template defines placeholders and various bits of basic logic (i.e., template tags) that regulate how the document should be displayed. Usually, templates are used for producing HTML, but Django templates are equally capable of generating any text-based format.

Let’s dive in with a simple example template. This template describes an HTML page that thanks a person for placing an order with a company. Think of it as a form letter:

<html>
<head><title>Ordering notice</title></head>

<body>

<p>Dear {{ person_name }},</p>

<p>Thanks for placing an order from {{ company }}. It's scheduled to
ship on {{ ship_date|date:"F j, Y" }}.</p>

<p>Here are the items you've ordered:</p>

<ul>
{% for item in item_list %}
<li>{{ item }}</li>
{% endfor %}
</ul>

{% if ordered_warranty %}
<p>Your warranty information will be included in the packaging.</p>
{% endif %}

<p>Sincerely,<br />{{ company }}</p>

</body>
</html>

This template is basic HTML with some variables and template tags thrown in. Let’s step through it:

Any text surrounded by a pair of braces (e.g., {{ person_name }}) is a variable. This means “insert the value of the variable with the given name.” How do we specify the values of the variables? We’ll get to that in a moment.
Any text that’s surrounded by curly braces and percent signs (e.g., {% if ordered_warranty %}) is a template tag. The definition of a tag is quite broad: a tag just tells the template system to “do something.”

This example template contains two tags: the {% for item in item_list %} tag (a for tag) and the {% if ordered_warranty %} tag (an if tag).

A for tag acts as a simple loop construct, letting you loop over each item in a sequence. An if tag, as you may expect, acts as a logical “if” statement. In this particular case, the tag checks whether the value of the ordered_warranty variable evaluates to True. If it does, the template system will display everything between the {% if ordered_warranty %} and {% endif %}. If not, the template system won’t display it. The template system also supports {% else %} and other various logic statements.
Finally, the second paragraph of this template has an example of a filter, with which you can alter the display of a variable. In this example, {{ ship_date|date:"F j, Y" }}, we’re passing the ship_date variable to the date filter, giving the date filter the argument "F j, Y". The date filter formats dates in a given format, as specified by that argument. Filters are attached using a pipe character (|), as a reference to Unix pipes.

Each Django template has access to several built-in tags and filters, many of which are discussed in the sections that follow. Appendix F contains the full list of tags and filters, and it’s a good idea to familiarize yourself with that list so you know what’s possible. It’s also possible to create your own filters and tags, which we cover in Chapter 10.

Using the Template System

To use the template system in Python code, just follow these two steps:

Create a Template object by providing the raw template code as a string. Django also offers a way to create Template objects by designating the path to a template file on the filesystem; we’ll examine that in a bit.
Call the render() method of the Template object with a given set of variables (i.e., the context). This returns a fully rendered template as a string, with all of the variables and block tags evaluated according to the context.

The following sections describe each step in more detail.

Creating Template Objects

The easiest way to create a Template object is to instantiate it directly. The Template class lives in the django.template module, and the constructor takes one argument, the raw template code. Let’s dip into the Python interactive interpreter to see how this works in code.

Interactive Interpreter Examples

Throughout this book, we feature example Python interactive interpreter sessions. You can recognize these examples by the triple greater-than signs (>>>), which designate the interpreter’s prompt. If you’re copying examples from this book, don’t copy those greater-than signs.

Multiline statements in the interactive interpreter are padded with three dots (...), for example:

>>> print """This is a
... string that spans
... three lines."""
This is a
string that spans
three lines.
>>> def my_function(value):
...     print value
>>> my_function('hello')
hello

Those three dots at the start of the additional lines are inserted by the Python shell—they’re not part of our input. We include them here to be faithful to the actual output of the interpreter. If you copy our examples to follow along, don’t copy those dots.

From within the project directory created by django-admin.py startproject (as covered in Chapter 2), type python manage.py shell to start the interactive interpreter. Here’s a basic walk-through:

>>> from django.template import Template
>>> t = Template("My name is {{ name }}.")
>>> print t

If you’re following along interactively, you’ll see something like this:

<django.template.Template object at 0xb7d5f24c>

That 0xb7d5f24c will be different every time, and it doesn’t really matter; it’s simply the Python “identity” of the Template object.

Django Settings

When using Django, you need to tell Django which settings to use. Interactively, this is typically done using python manage.py shell, but you’ve got a few other options described in Appendix E.

When you create a Template object, the template system compiles the raw template code into an internal, optimized form, ready for rendering. But if your template code includes any syntax errors, the call to Template() will cause a TemplateSyntaxError exception:

>>> from django.template import Template
>>> t = Template('{% notatag %} ')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  ...
  django.template.TemplateSyntaxError: Invalid block tag: 'notatag'

The system raises a TemplateSyntaxError exception for any of the following cases:

Invalid block tags
Invalid arguments to valid block tags
Invalid filters
Invalid arguments to valid filters
Invalid template syntax
Unclosed block tags (for block tags that require closing tags)

Rendering a Template

Once you have a Template object, you can pass it data by giving it a context. A context is simply a set of variables and their associated values. A template uses this to populate its variable tags and evaluate its block tags.

A context is represented in Django by the Context class, which lives in the django.template module. Its constructor takes one optional argument: a dictionary mapping variable names to variable values. Call the Template object’s render() method with the context to “fill” the template:

>>> from django.template import Context, Template
>>> t = Template("My name is {{ name }}.")
>>> c = Context({"name": "Stephane"})
>>> t.render(c)
'My name is Stephane.'

Dictionaries and Contexts

A Python dictionary is a mapping between known keys and variable values. A Context is similar to a dictionary, but a Context provides additional functionality, as covered in Chapter 10.

Variable names must begin with a letter (A-Z or a-z) and may contain digits, underscores, and dots. (Dots are a special case we’ll get to in a moment.) Variable names are case sensitive.

Here’s an example of template compilation and rendering, using the sample template from the beginning of this chapter:

>>> from django.template import Template, Context
>>> raw_template = """<p>Dear {{ person_name }},</p>
...
... <p>Thanks for ordering {{ product }} from {{ company }}. It's scheduled
... to ship on {{ ship_date|date:"F j, Y" }}.</p>
...
... {% if ordered_warranty %}
... <p>Your warranty information will be included in the packaging.</p>
... {% endif %}
...
... <p>Sincerely,<br />{{ company }}</p>"""
>>> t = Template(raw_template)
>>> import datetime
>>> c = Context({'person_name': 'John Smith',
...     'product': 'Super Lawn Mower',
...     'company': 'Outdoor Equipment',
...     'ship_date': datetime.date(2009, 4, 2),
...     'ordered_warranty': True})
>>> t.render(c)
"<p>Dear John Smith,</p>\n\n<p>Thanks for ordering Super Lawn Mower from
Outdoor Equipment. It's scheduled \nto ship on April 2, 2009.</p>\n\n\n
<p>Your warranty information will be included in the packaging.</p>\n\n\n
<p>Sincerely,<br />Outdoor Equipment</p>"

Let’s step through this code one statement at a time:

First, we import the classes Template and Context, which both live in the module django.template.
We save the raw text of our template into the variable raw_template. Note that we use triple quote marks to designate the string, because it wraps over multiple lines; in Python codde, strings designated with single quote marks cannot be wrapped over multiple lines.
Next, we create a template object, t, by passing raw_template to the Template class constructor.
We import the datetime module from Python’s standard library, because we’ll need it in the following statement.
Then, we create a Context object, c. The Context constructor takes a Python dictionary, which maps variable names to values. Here, for example, we specify that the person_name is 'John Smith', product is 'Super Lawn Mower', and so forth.
Finally, we call the render() method on our template object, passing it the context. This returns the rendered template—that is, it replaces template variables with the actual values of the variables, and it executes any block tags.

Note that the warranty paragraph was displayed because the ordered_warranty variable evaluated to True. Also note the date, April 2, 2009, which is displayed according to the format string 'F j, Y'. (We explain format strings for the date filter shortly.)

If you’re new to Python, you may wonder why this output includes newline characters ('\n') rather than displaying the line breaks. That’s happening because of a subtlety in the Python interactive interpreter: the call to t.render(c) returns a string, and by default the interactive interpreter displays the representation of the string, rather than the printed value of the string. If you want to see the string with line breaks displayed as true line breaks rather than '\n' characters, use the print statement: print t.render(c).

Those are the fundamentals of using the Django template system: just write a template, create a Template object, create a Context, and call the render() method.

Multiple Contexts, Same Template

Once you have a Template object, you can render multiple contexts through it, for example:

>>> from django.template import Template, Context
>>> t = Template('Hello, {{ name }}')
>>> print t.render(Context({'name': 'John'}))
Hello, John
>>> print t.render(Context({'name': 'Julie'}))
Hello, Julie
>>> print t.render(Context({'name': 'Pat'}))
Hello, Pat

Whenever you’re using the same template source to render multiple contexts like this, it’s more efficient to create the Template object once, and then call render() on it multiple times:

# Bad
for name in ('John', 'Julie', 'Pat'):
    t = Template('Hello, {{ name }}')
    print t.render(Context({'name': name}))

# Good
t = Template('Hello, {{ name }}')
for name in ('John', 'Julie', 'Pat'):
    print t.render(Context({'name': name}))

Django’s template parsing is quite fast. Behind the scenes, most of the parsing happens via a single call to a short regular expression. This is in stark contrast to XML-based template engines, which incur the overhead of an XML parser and tend to be orders of magnitude slower than Django’s template rendering engine.

Context Variable Lookup

In the examples so far, we’ve passed simple values in the contexts—mostly strings, plus a datetime.date example. However, the template system elegantly handles more complex data structures, such as lists, dictionaries, and custom objects.

The key to traversing complex data structures in Django templates is the dot character (.). Use a dot to access dictionary keys, attributes, indices, or methods of an object.

This is best illustrated with a few examples. For instance, suppose you’re passing a Python dictionary to a template. To access the values of that dictionary by dictionary key, use a dot:

>>> from django.template import Template, Context
>>> person = {'name': 'Sally', 'age': '43'}
>>> t = Template('{{ person.name }} is {{ person.age }} years old.')
>>> c = Context({'person': person})
>>> t.render(c)
'Sally is 43 years old.'

Similarly, dots also allow access of object attributes. For example, a Python datetime.date object has year, month, and day attributes, and you can use a dot to access those attributes in a Django template:

>>> from django.template import Template, Context
>>> import datetime
>>> d = datetime.date(1993, 5, 2)
>>> d.year
1993
>>> d.month
5
>>> d.day
2
>>> t = Template('The month is {{ date.month }} and the year is {{ date.year }}.')
>>> c = Context({'date': d})
>>> t.render(c)
'The month is 5 and the year is 1993.'

This example uses a custom class:

>>> from django.template import Template, Context
>>> class Person(object):
...     def __init__(self, first_name, last_name):
...         self.first_name, self.last_name = first_name, last_name
>>> t = Template('Hello, {{ person.first_name }} {{ person.last_name }}.')
>>> c = Context({'person': Person('John', 'Smith')})
>>> t.render(c)
'Hello, John Smith.'

Dots are also used to call methods on objects. For example, each Python string has the methods upper() and isdigit(), and you can call those in Django templates using the same dot syntax:

>>> from django.template import Template, Context
>>> t = Template('{{ var }} -- {{ var.upper }} -- {{ var.isdigit }}')
>>> t.render(Context({'var': 'hello'}))
'hello -- HELLO -- False'
>>> t.render(Context({'var': '123'}))
'123 -- 123 -- True'

Note that you don’t include parentheses in the method calls. Also, it’s not possible to pass arguments to the methods; you can only call methods that have no required arguments. (We explain this philosophy later in this chapter.)

Finally, dots are also used to access list indices, for example:

>>> from django.template import Template, Context
>>> t = Template('Item 2 is {{ items.2 }}.')
>>> c = Context({'items': ['apples', 'bananas', 'carrots']})
>>> t.render(c)
'Item 2 is carrots.'

Negative list indices are not allowed. For example, the template variable {{ items.-1 }} would cause a TemplateSyntaxError.

Python Lists

Python lists have 0-based indices so that the first item is at index 0, the second is at index 1, and so on.

The dot lookups can be summarized like this: when the template system encounters a dot in a variable name, it tries the following lookups, in this order:

Dictionary lookup (e.e., foo["bar"])
Attribute lookup (e.g., foo.bar)
Method call (e.g., foo.bar())
List-index lookup (e.g., foo[bar])

The system uses the first lookup type that works. It’s short-circuit logic.

Dot lookups can be nested multiple levels deep. For instance, the following example uses {{ person.name.upper }}, which translates into a dictionary lookup (person['name']) and then a method call (upper()):

>>> from django.template import Template, Context
>>> person = {'name': 'Sally', 'age': '43'}
>>> t = Template('{{ person.name.upper }} is {{ person.age }} years old.')
>>> c = Context({'person': person})
>>> t.render(c)
'SALLY is 43 years old.'

Method Call Behavior

Method calls are slightly more complex than the other lookup types. Here are some things to keep in mind:

If, during the method lookup, a method raises an exception, the exception will be propagated, unless the exception has an attribute silent_variable_failure whose value is True. If the exception does have a silent_variable_failure attribute, the variable will render as an empty string, for example:

>>> t = Template("My name is {{ person.first_name }}.")
>>> class PersonClass3:
...     def first_name(self):
...         raise AssertionError, "foo"
>>> p = PersonClass3()
>>> t.render(Context({"person": p}))
Traceback (most recent call last):
...
AssertionError: foo

>>> class SilentAssertionError(AssertionError):
...     silent_variable_failure = True
>>> class PersonClass4:
...     def first_name(self):
...         raise SilentAssertionError
>>> p = PersonClass4()
>>> t.render(Context({"person": p}))
"My name is ."

A method call will only work if the method has no required arguments. Otherwise, the system will move to the next lookup type (list-index lookup).
Obviously, some methods have side effects, and it would be foolish at best, and possibly even a security hole, to allow the template system to access them.

Say, for instance, you have a BankAccount object that has a delete() method. A template shouldn’t be allowed to include something like {{ account.delete }}.

To prevent this, set the function attribute alters_data on the method:
```
def delete(self):
    # Delete the account
delete.alters_data = True
```
The template system won’t execute any method marked in this way. In other words, if a template includes {{ account.delete }}, that tag will not execute the delete() method. It will fail silently.

How Invalid Variables Are Handled

By default, if a variable doesn’t exist, the template system renders it as an empty string, failing silently, for example:

>>> from django.template import Template, Context
>>> t = Template('Your name is {{ name }}.')
>>> t.render(Context())
'Your name is .'
>>> t.render(Context({'var': 'hello'}))
'Your name is .'
>>> t.render(Context({'NAME': 'hello'}))
'Your name is .'
>>> t.render(Context({'Name': 'hello'}))
'Your name is .'

The system fails silently rather than raising an exception because it’s intended to be resilient to human error. In this case, all of the lookups failed because variable names have the wrong case or name. In the real world, it’s unacceptable for a Web site to become inaccessible due to a small template syntax error.

Note that it’s possible to change Django’s default behavior in this regard, by tweaking a setting in your Django configuration. We discuss this further in Chapter 10.

Playing with Context Objects

Most of the time, you’ll instantiate Context objects by passing in a fully populated dictionary to Context(). But you can add and delete items from a Context object once it’s been instantiated, too, using standard Python dictionary syntax:

>>> from django.template import Context
>>> c = Context({"foo": "bar"})
>>> c['foo']
'bar'
>>> del c['foo']
>>> c['foo']
''
>>> c['newvariable'] = 'hello'
>>> c['newvariable']
'hello'

Basic Template Tags and Filters

As we’ve mentioned already, the template system ships with built-in tags and filters. The sections that follow provide a rundown of the most common tags and filters.

Filters

As explained earlier in this chapter, template filters are simple ways of altering the value of variables before they’re displayed. Filters look like this:

{{ name|lower }}

This displays the value of the {{ name }} variable after being filtered through the lower filter, which converts text to lowercase. Use a pipe (|) to apply a filter.

Filters can be chained—that is, the output of one filter is applied to the next. Here’s a common idiom for escaping text contents, and then converting line breaks to <p> tags:

{{ my_text|escape|linebreaks }}

Some filters take arguments. A filter argument looks like this:

{{ bio|truncatewords:"30" }}

This displays the first 30 words of the bio variable. Filter arguments are always in double quotes.

The following are a few of the most important filters; Appendix F covers the rest.

addslashes: Adds a backslash before any backslash, single quote, or double quote. This is useful if the produced text is included in a JavaScript string.
date: Formats a date or datetime object according to a format string given in the parameter, for example:
```
{{ pub_date|date:"F j, Y" }}
```
Format strings are defined in Appendix F.
escape: Escapes ampersands, quotes, and angle brackets in the given string. This is useful for sanitizing user-submitted data and for ensuring data is valid XML or XHTML. Specifically, escape makes these conversions:
- Converts & to &
- Converts < to <
- Converts > to >
- Converts " (double quote) to "
- Converts ' (single quote) to '
length: Returns the length of the value. You can use this on a list or a string, or any Python object that knows how to determine its length (i.e., any object that has a __len__() method).

Philosophies and Limitations

Now that you’ve gotten a feel for the Django template language, we should point out some of its intentional limitations, along with some philosophies behind why it works the way it works.

More than any other component of Web applications, programmer opinions on template systems vary wildly. The fact that Python alone has dozens, if not hundreds, of open source template-language implementations supports this point. Each was likely created because its developer deemed all existing template languages inadequate. (In fact, it is said to be a rite of passage for a Python developer to write his or her own template language! If you haven’t done this yet, consider it. It’s a fun exercise.)

With that in mind, you might be interested to know that Django doesn’t require that you use its template language. Because Django is intended to be a full-stack Web framework that provides all the pieces necessary for Web developers to be productive, many times it’s more convenient to use Django’s template system than other Python template libraries, but it’s not a strict requirement in any sense. As you’ll see in the upcoming section “Using Templates in Views”, it’s very easy to use another template language with Django.

Still, it’s clear we have a strong preference for the way Django’s template language works. The template system has roots in how Web development is done at World Online and the combined experience of Django’s creators. Here are a few of those philosophies:

Business logic should be separated from presentation logic. We see a template system as a tool that controls presentation and presentation-related logic—and that’s it. The template system shouldn’t support functionality that goes beyond this basic goal.

For that reason, it’s impossible to call Python code directly within Django templates. All “programming” is fundamentally limited to the scope of what template tags can do. It is possible to write custom template tags that do arbitrary things, but the out-of-the-box Django template tags intentionally do not allow for arbitrary Python code execution.
Syntax should be decoupled from HTML/XML. Although Django’s template system is used primarily to produce HTML, it’s intended to be just as usable for non-HTML formats, such as plain text. Some other template languages are XML based, placing all template logic within XML tags or attributes, but Django deliberately avoids this limitation. Requiring valid XML to write templates introduces a world of human mistakes and hard-to-understand error messages, and using an XML engine to parse templates incurs an unacceptable level of overhead in template processing.
Designers are assumed to be comfortable with HTML code. The template system isn’t designed so that templates necessarily are displayed nicely in WYSIWYG editors such as Dreamweaver. That is too severe a limitation and wouldn’t allow the syntax to be as nice as it is. Django expects template authors to be comfortable editing HTML directly.
Designers are assumed not to be Python programmers. The template system authors recognize that Web page templates are most often written by designers, not programmers, and therefore should not assume Python knowledge.

However, the system also intends to accommodate small teams in which the templates are created by Python programmers. It offers a way to extend the system’s syntax by writing raw Python code. (More on this in Chapter 10.)
The goal is not to invent a programming language. The goal is to offer just enough programming-esque functionality, such as branching and looping, that is essential for making presentation-related decisions.

As a result of these design philosophies, the Django template language has the following limitations:

A template cannot set a variable or change the value of a variable. It’s possible to write custom template tags that accomplish these goals (see Chapter 10), but the stock Django template tags do not allow it.
A template cannot call raw Python code. There’s no way to “drop into Python mode” or use raw Python constructs. Again, it’s possible to write custom template tags to do this, but the stock Django template tags don’t allow it.

Using Templates in Views

You’ve learned the basics of using the template system; now let’s use this knowledge to create a view. Recall the current_datetime view in mysite.views, which we started in the previous chapter. Here’s what it looks like:

from django.http import HttpResponse
import datetime

def current_datetime(request):
    now = datetime.datetime.now()
    html = "<html><body>It is now %s.</body></html>" % now
    return HttpResponse(html)

Let’s change this view to use Django’s template system. At first, you might think to do something like this:

from django.template import Template, Context
from django.http import HttpResponse
import datetime

def current_datetime(request):
    now = datetime.datetime.now()
    t = Template("<html><body>It is now {{ current_date }}.</body></html>")
    html = t.render(Context({'current_date': now}))
    return HttpResponse(html)

Sure, that uses the template system, but it doesn’t solve the problems we pointed out in the introduction of this chapter. Namely, the template is still embedded in the Python code. Let’s fix that by putting the template in a separate file, which this view will load.

You might first consider saving your template somewhere on your filesystem and using Python’s built-in file-opening functionality to read the contents of the template. Here’s what that might look like, assuming the template was saved as the file /home/djangouser/templates/mytemplate.html:

from django.template import Template, Context
from django.http import HttpResponse
import datetime

def current_datetime(request):
    now = datetime.datetime.now()
    # Simple way of using templates from the filesystem.
    # This doesn't account for missing files!
    fp = open('/home/djangouser/templates/mytemplate.html')
    t = Template(fp.read())
    fp.close()
    html = t.render(Context({'current_date': now}))
    return HttpResponse(html)

This approach, however, is inelegant for these reasons:

It doesn’t handle the case of a missing file. If the file mytemplate.html doesn’t exist or isn’t readable, the open() call will raise an IOError exception.
It hard-codes your template location. If you were to use this technique for every view function, you’d be duplicating the template locations. Not to mention it involves a lot of typing!
It includes a lot of boring boilerplate code. You’ve got better things to do than to write calls to open(), fp.read(), and fp.close() each time you load a template.

To solve these issues, we’ll use template loading and template directories, both of which are described in the sections that follow.

Template Loading

Django provides a convenient and powerful API for loading templates from disk, with the goal of removing redundancy both in your template-loading calls and in your templates themselves.

In order to use this template-loading API, first you’ll need to tell the framework where you store your templates. The place to do this is in your settings file.

A Django settings file is the place to put configuration for your Django instance (aka your Django project). It’s a simple Python module with module-level variables, one for each setting.

When you ran django-admin.py startproject mysite in Chapter 2, the script created a default settings file for you, aptly named settings.py. Have a look at the file’s contents. It contains variables that look like this (though not necessarily in this order):

DEBUG = True
TIME_ZONE = 'America/Chicago'
USE_I18N = True
ROOT_URLCONF = 'mysite.urls'

This is pretty self-explanatory; the settings and their respective values are simple Python variables. And because the settings file is just a plain Python module, you can do dynamic things such as checking the value of one variable before setting another. (This also means that you should avoid Python syntax errors in your settings file.)

We’ll cover settings files in depth in Appendix E, but for now, have a look at the TEMPLATE_DIRS setting. This setting tells Django’s template-loading mechanism where to look for templates. By default, it’s an empty tuple. Pick a directory where you’d like to store your templates and add it to TEMPLATE_DIRS, like so:

TEMPLATE_DIRS = (
    '/home/django/mysite/templates',
)

There are a few things to note:

You can specify any directory you want, as long as the directory and templates within that directory are readable by the user account under which your Web server runs. If you can’t think of an appropriate place to put your templates, we recommend creating a templates directory within your Django project (i.e., within the mysite directory you created in Chapter 2, if you’ve been following along with this book’s examples).
Don’t forget the comma at the end of the template directory string! Python requires commas within single-element tuples to disambiguate the tuple from a parenthetical expression. This is a common newbie gotcha.

If you want to avoid this error, you can make TEMPLATE_DIRS a list instead of a tuple, because single-element lists don’t require a trailing comma:
```
TEMPLATE_DIRS = [
    '/home/django/mysite/templates'
]
```
A tuple is slightly more semantically correct than a list (tuples cannot be changed after being created, and nothing should be changing settings once they’ve been read), so we recommend using a tuple for your TEMPLATE_DIRS setting.
If you’re on Windows, include your drive letter and use Unix-style forward slashes rather than backslashes, as follows:
```
TEMPLATE_DIRS = (
    'C:/www/django/templates',
)
```
It’s simplest to use absolute paths (i.e., directory paths that start at the root of the filesystem). If you want to be a bit more flexible and decoupled, though, you can take advantage of the fact that Django settings files are just Python code by constructing the contents of TEMPLATE_DIRS dynamically, for example:
```
import os.path

TEMPLATE_DIRS = (
    os.path.join(os.path.dirname(__file__), 'templates').replace('\\','/'),
)
```
This example uses the “magic” Python variable __file__, which is automatically set to the file name of the Python module in which the code lives.

With TEMPLATE_DIRS set, the next step is to change the view code to use Django’s template-loading functionality rather than hard-coding the template paths. Returning to our current_datetime view, let’s change it like so:

from django.template.loader import get_template
from django.template import Context
from django.http import HttpResponse
import datetime

def current_datetime(request):
    now = datetime.datetime.now()
    t = get_template('current_datetime.html')
    html = t.render(Context({'current_date': now}))
    return HttpResponse(html)

In this example, we’re using the function django.template.loader.get_template() rather than loading the template from the filesystem manually. The get_template() function takes a template name as its argument, figures out where the template lives on the filesystem, opens that file, and returns a compiled Template object.

If get_template() cannot find the template with the given name, it raises a TemplateDoesNotExist exception. To see what that looks like, fire up the Django development server again, as in Chapter 3, by running python manage.py runserver within your Django project’s directory. Then, point your browser at the page that activates the current_datetime view (e.g., http://127.0.0.1:8000/time/). Assuming your DEBUG setting is set to True and you haven’t yet created a current_datetime.html template, you should see a Django error page highlighting the TemplateDoesNotExist error.

Screenshot of a “TemplateDoesNotExist” error.

Figure 4-1: The error page shown when a template cannot be found.

This error page is similar to the one we explained in Chapter 3, with one additional piece of debugging information: a “Template-loader postmortem” section. This section tells you which templates Django tried to load, along with the reason each attempt failed (e.g., “File does not exist”). This information is invaluable when you’re trying to debug template-loading errors.

As you can probably tell from the error messages found in the Figure 4-1, Django attempted to find the template by combining the directory in the TEMPLATE_DIRS setting with the template name passed to get_template(). So if your TEMPLATE_DIRS contains '/home/django/templates', Django looks for the file '/home/django/templates/current_datetime.html'. If TEMPLATE_DIRS contains more than one directory, each is checked until the template is found or they’ve all been checked.

Moving along, create the current_datetime.html file within your template directory using the following template code:

<html><body>It is now {{ current_date }}.</body></html>

Refresh the page in your Web browser, and you should see the fully rendered page.

render_to_response()

Because it’s such a common idiom to load a template, fill a Context, and return an HttpResponse object with the result of the rendered template, Django provides a shortcut that lets you do those things in one line of code. This shortcut is a function called render_to_response(), which lives in the module django.shortcuts. Most of the time, you’ll be using render_to_response() rather than loading templates and creating Context and HttpResponse objects manually.

Here’s the ongoing current_datetime example rewritten to use render_to_response():

from django.shortcuts import render_to_response
import datetime

def current_datetime(request):
    now = datetime.datetime.now()
    return render_to_response('current_datetime.html', {'current_date': now})

What a difference! Let’s step through the code changes:

We no longer have to import get_template, Template, Context, or HttpResponse. Instead, we import django.shortcuts.render_to_response. The import datetime remains.
Within the current_datetime function, we still calculate now, but the template loading, context creation, template rendering, and HttpResponse creation is all taken care of by the render_to_response() call. Because render_to_response() returns an HttpResponse object, we can simply return that value in the view.

The first argument to render_to_response() should be the name of the template to use. The second argument, if given, should be a dictionary to use in creating a Context for that template. If you don’t provide a second argument, render_to_response() will use an empty dictionary.

The locals() Trick

Consider our latest incarnation of current_datetime:

def current_datetime(request):
    now = datetime.datetime.now()
    return render_to_response('current_datetime.html', {'current_date': now})

Many times, as in this example, you’ll find yourself calculating some values, storing them in variables (e.g., now in the preceding code), and sending those variables to the template. Particularly lazy programmers should note that it’s slightly redundant to have to give names for temporary variables and give names for the template variables. Not only is it redundant, but also it’s extra typing.

So if you’re one of those lazy programmers and you like keeping code particularly concise, you can take advantage of a built-in Python function called locals(). It returns a dictionary mapping all local variable names to their values. Thus, the preceding view could be rewritten like so:

def current_datetime(request):
    current_date = datetime.datetime.now()
    return render_to_response('current_datetime.html', locals())

Here, instead of manually specifying the context dictionary as before, we pass the value of locals(), which will include all variables defined at that point in the function’s execution. As a consequence, we’ve renamed the now variable to current_date, because that’s the variable name that the template expects. In this example, locals() doesn’t offer a huge improvement, but this technique can save you some typing if you have several template variables to define—or if you’re lazy.

One thing to watch out for when using locals() is that it includes every local variable, which may comprise more variables than you actually want your template to have access to. In the previous example, locals() will also include request. Whether this matters to you depends on your application.

A final thing to consider is that locals() incurs a small bit of overhead, because when you call it, Python has to create the dictionary dynamically. If you specify the context dictionary manually, you avoid this overhead.

Subdirectories in get_template()

It can get unwieldy to store all of your templates in a single directory. You might like to store templates in subdirectories of your template directory, and that’s fine. In fact, we recommend doing so; some more advanced Django features (such as the generic views system, which we cover in Chapter 9) expect this template layout as a default convention.

Storing templates in subdirectories of your template directory is easy. In your calls to get_template(), just include the subdirectory name and a slash before the template name, like so:

t = get_template('dateapp/current_datetime.html')

Because render_to_response() is a small wrapper around get_template(), you can do the same thing with the first argument to render_to_response().

There’s no limit to the depth of your subdirectory tree. Feel free to use as many as you like.

Note

Windows users, be sure to use forward slashes rather than backslashes. get_template() assumes a Unix-style file name designation.

The `include` Template Tag

Now that we’ve covered the template-loading mechanism, we can introduce a built-in template tag that takes advantage of it: {% include %}. This tag allows you to include the contents of another template. The argument to the tag should be the name of the template to include, and the template name can be either a variable or a hard-coded (quoted) string, in either single or double quotes. Anytime you have the same code in multiple templates, consider using an {% include %} to remove the duplication.

These two examples include the contents of the template nav.html. The examples are equivalent and illustrate that either single or double quotes are allowed:

{% include 'nav.html' %}
{% include "nav.html" %}

This example includes the contents of the template includes/nav.html:

{% include 'includes/nav.html' %}

This example includes the contents of the template whose name is contained in the variable template_name:

{% include template_name %}

As in get_template(), the file name of the template is determined by adding the template directory from TEMPLATE_DIRS to the requested template name.

Included templates are evaluated with the context of the template that’s including them.

If a template with the given name isn’t found, Django will do one of two things:

If DEBUG is set to True, you’ll see the TemplateDoesNotExist exception on a Django error page.
If DEBUG is set to False, the tag will fail silently, displaying nothing in the place of the tag.

Template Inheritance

Our template examples so far have been tiny HTML snippets, but in the real world, you’ll be using Django’s template system to create entire HTML pages. This leads to a common Web development problem: across a Web site, how does one reduce the duplication and redundancy of common page areas, such as sitewide navigation?

A classic way of solving this problem is to use server-side includes, directives you can embed within your HTML pages to “include” one Web page inside another. Indeed, Django supports that approach, with the {% include %} template tag just described. But the preferred way of solving this problem with Django is to use a more elegant strategy called template inheritance.

In essence, template inheritance lets you build a base “skeleton” template that contains all the common parts of your site and defines “blocks” that child templates can override.

Let’s see an example of this by creating a more complete template for our current_datetime view, by editing the current_datetime.html file:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>
    <title>The current time</title>
</head>
<body>
    <h1>My helpful timestamp site</h1>
    <p>It is now {{ current_date }}.</p>

    <hr>
    <p>Thanks for visiting my site.</p>
</body>
</html>

That looks just fine, but what happens when we want to create a template for another view—say, the hours_ahead view from Chapter 3? If we want again to make a nice, valid, full HTML template, we’d create something like:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>
    <title>Future time</title>
</head>
<body>
    <h1>My helpful timestamp site</h1>
    <p>In {{ hour_offset }} hour(s), it will be {{ next_time }}.</p>

    <hr>
    <p>Thanks for visiting my site.</p>
</body>
</html>

Clearly, we’ve just duplicated a lot of HTML. Imagine if we had a more typical site, including a navigation bar, a few style sheets, perhaps some JavaScript—we’d end up putting all sorts of redundant HTML into each template.

The server-side include solution to this problem is to factor out the common bits in both templates and save them in separate template snippets, which are then included in each template. Perhaps you’d store the top bit of the template in a file called header.html:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>

And perhaps you’d store the bottom bit in a file called footer.html:

    <hr>
    <p>Thanks for visiting my site.</p>
</body>
</html>

With an include-based strategy, headers and footers are easy. It’s the middle ground that’s messy. In this example, both pages feature a title— <h1>My helpful timestamp site</h1>—but that title can’t fit into header.html because the <title> on both pages is different. If we included the <h1> in the header, we’d have to include the <title>, which wouldn’t allow us to customize it per page. See where this is going?

Django’s template inheritance system solves these problems. You can think of it as an “inside-out” version of server-side includes. Instead of defining the snippets that are common, you define the snippets that are different.

The first step is to define a base template—a skeleton of your page that child templates will later fill in. Here’s a base template for our ongoing example:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>
    <title>{% block title %}{% endblock %}</title>
</head>
<body>
    <h1>My helpful timestamp site</h1>
    {% block content %}{% endblock %}
    {% block footer %}
    <hr>
    <p>Thanks for visiting my site.</p>
    {% endblock %}
</body>
</html>

This template, which we’ll call base.html, defines a simple HTML skeleton document that we’ll use for all the pages on the site. It’s the job of child templates to override, or add to, or leave alone the contents of the blocks. (If you’re following along at home, save this file to your template directory.)

We’re using a template tag here that you haven’t seen before: the {% block %} tag. All the {% block %} tags do is tell the template engine that a child template may override those portions of the template.

Now that we have this base template, we can modify our existing current_datetime.html template to use it:

{% extends "base.html" %}

{% block title %}The current time{% endblock %}

{% block content %}
<p>It is now {{ current_date }}.</p>
{% endblock %}

While we’re at it, let’s create a template for the hours_ahead view from Chapter 3. (If you’re following along with code, we’ll leave it up to you to change hours_ahead to use the template system.) Here’s what that would look like:

{% extends "base.html" %}

{% block title %}Future time{% endblock %}

{% block content %}
<p>In {{ hour_offset }} hour(s), it will be {{ next_time }}.</p>
{% endblock %}

Isn’t this beautiful? Each template contains only the code that’s unique to that template. No redundancy needed. If you need to make a site-wide design change, just make the change to base.html, and all of the other templates will immediately reflect the change.

Here’s how it works. When you load the template current_datetime.html, the template engine sees the {% extends %} tag, noting that this template is a child template. The engine immediately loads the parent template—in this case, base.html.

At that point, the template engine notices the three {% block %} tags in base.html and replaces those blocks with the contents of the child template. So, the title we’ve defined in {% block title %} will be used, as will the {% block content %}.

Note that since the child template doesn’t define the footer block, the template system uses the value from the parent template instead. Content within a {% block %} tag in a parent template is always used as a fallback.

Inheritance doesn’t affect the way the context works, and you can use as many levels of inheritance as needed. One common way of using inheritance is the following three-level approach:

Create a base.html template that holds the main look and feel of your site. This is the stuff that rarely, if ever, changes.
Create a base_SECTION.html template for each “section” of your site (e.g., base_photos.html and base_forum.html). These templates extend base.html and include section-specific styles/design.
Create individual templates for each type of page, such as a forum page or a photo gallery. These templates extend the appropriate section template.

This approach maximizes code reuse and makes it easy to add items to shared areas, such as section-wide navigation.

Here are some tips for working with template inheritance:

If you use {% extends %} in a template, it must be the first template tag in that template. Otherwise, template inheritance won’t work.
Generally, the more {% block %} tags in your base templates, the better. Remember, child templates don’t have to define all parent blocks, so you can fill in reasonable defaults in a number of blocks, and then define only the ones you need in the child templates. It’s better to have more hooks than fewer hooks.
If you find yourself duplicating code in a number of templates, it probably means you should move that code to a {% block %} in a parent template.
If you need to get the content of the block from the parent template, the {{ block.super }} variable will do the trick. This is useful if you want to add to the contents of a parent block instead of completely overriding it.
You may not define multiple {% block %} tags with the same name in the same template. This limitation exists because a block tag works in “both” directions. That is, a block tag doesn’t just provide a hole to fill, it also defines the content that fills the hole in the parent. If there were two similarly named {% block %} tags in a template, that template’s parent wouldn’t know which one of the blocks’ content to use.
The template name you pass to {% extends %} is loaded using the same method that get_template() uses. That is, the template name is appended to your TEMPLATE_DIRS setting.
In most cases, the argument to {% extends %} will be a string, but it can also be a variable, if you don’t know the name of the parent template until runtime. This lets you do some cool, dynamic stuff.

What’s next?

Most modern Web sites are database-driven: the content of the Web site is stored in a relational database. This allows a clean separate of data and logic (in the same way views and templates allow the separation of logic and display.)

The next chapter covers the tools Django gives you to interact with a database.

Chapter 5: Interacting with a Database: Models

In Chapter 3, we covered the fundamentals of building dynamic Web sites with Django: setting up views and URLconfs. As we explained, a view is responsible for doing some arbitrary logic, and then returning a response. In the example, our arbitrary logic was to calculate the current date and time.

In modern Web applications, the arbitrary logic often involves interacting with a database. Behind the scenes, a database-driven Web site connects to a database server, retrieves some data out of it, and displays that data, nicely formatted, on a Web page. Or, similarly, the site could provide functionality that lets site visitors populate the database on their own.

Many complex Web sites provide some combination of the two. Amazon.com, for instance, is a great example of a database-driven site. Each product page is essentially a query into Amazon’s product database formatted as HTML, and when you post a customer review, it gets inserted into the database of reviews.

Django is well suited for making database-driven Web sites, as it comes with easy yet powerful ways of performing database queries using Python. This chapter explains that functionality: Django’s database layer.

(Note: While it’s not strictly necessary to know basic database theory and SQL in order to use Django’s database layer, it’s highly recommended. An introduction to those concepts is beyond the scope of this book, but keep reading even if you’re a database newbie. You’ll probably be able to follow along and grasp concepts based on the context.)

The “Dumb” Way to Do Database Queries in Views

Just as Chapter 3 detailed a “dumb” way to produce output within a view (by hard-coding the text directly within the view), there’s a “dumb” way to retrieve data from a database in a view. It’s simple: just use any existing Python library to execute an SQL query and do something with the results.

In this example view, we use the MySQLdb library (available at http://www.djangoproject.com/r/python-mysql/) to connect to a MySQL database, retrieve some records, and feed them to a template for display as a Web page:

from django.shortcuts import render_to_response
import MySQLdb

def book_list(request):
    db = MySQLdb.connect(user='me', db='mydb', passwd='secret', host='localhost')
    cursor = db.cursor()
    cursor.execute('SELECT name FROM books ORDER BY name')
    names = [row[0] for row in cursor.fetchall()]
    db.close()
    return render_to_response('book_list.html', {'names': names})

This approach works, but some problems should jump out at you immediately:

We’re hard-coding the database connection parameters. Ideally, these parameters would be stored in the Django configuration.
We’re having to write a fair bit of boilerplate code: creating a connection, creating a cursor, executing a statement, and closing the connection. Ideally, all we’d have to do is specify which results we wanted.
It ties us to MySQL. If, down the road, we switch from MySQL to PostgreSQL, we’ll have to use a different database adapter (e.g., psycopg rather than MySQLdb), alter the connection parameters, and — depending on the nature of the SQL statement — possibly rewrite the SQL. Ideally, the database server we’re using would be abstracted, so that a database server change could be made in a single place.

As you might expect, Django’s database layer aims to solve these problems. Here’s a sneak preview of how the previous view can be rewritten using Django’s database API:

from django.shortcuts import render_to_response
from mysite.books.models import Book

def book_list(request):
    books = Book.objects.order_by('name')
    return render_to_response('book_list.html', {'books': books})

We’ll explain this code a little later in the chapter. For now, just get a feel for how it looks.

The MTV Development Pattern

Before we delve into any more code, let’s take a moment to consider the overall design of a database-driven Django Web application.

As we mentioned in previous chapters, Django is designed to encourage loose coupling and strict separation between pieces of an application. If you follow this philosophy, it’s easy to make changes to one particular piece of the application without affecting the other pieces. In view functions, for instance, we discussed the importance of separating the business logic from the presentation logic by using a template system. With the database layer, we’re applying that same philosophy to data access logic.

Those three pieces together — data access logic, business logic, and presentation logic — comprise a concept that’s sometimes called the Model-View-Controller (MVC) pattern of software architecture. In this pattern, “Model” refers to the data access layer, “View” refers to the part of the system that selects what to display and how to display it, and “Controller” refers to the part of the system that decides which view to use, depending on user input, accessing the model as needed.

Why the Acronym?

The goal of explicitly defining patterns such as MVC is mostly to streamline communication among developers. Instead of having to tell your coworkers, “Let’s make an abstraction of the data access, then let’s have a separate layer that handles data display, and let’s put a layer in the middle that regulates this,” you can take advantage of a shared vocabulary and say, “Let’s use the MVC pattern here.”

Django follows this MVC pattern closely enough that it can be called an MVC framework. Here’s roughly how the M, V, and C break down in Django:

M, the data-access portion, is handled by Django’s database layer, which is described in this chapter.
V, the portion that selects which data to display and how to display it, is handled by views and templates.
C, the portion that delegates to a view depending on user input, is handled by the framework itself by following your URLconf and calling the appropriate Python function for the given URL.

Because the “C” is handled by the framework itself and most of the excitement in Django happens in models, templates, and views, Django has been referred to as an MTV framework. In the MTV development pattern,

M stands for “Model,” the data access layer. This layer contains anything and everything about the data: how to access it, how to validate it, which behaviors it has, and the relationships between the data.
T stands for “Template,” the presentation layer. This layer contains presentation-related decisions: how something should be displayed on a Web page or other type of document.
V stands for “View,” the business logic layer. This layer contains the logic that access the model and defers to the appropriate template(s). You can think of it as the bridge between models and templates.

If you’re familiar with other MVC Web-development frameworks, such as Ruby on Rails, you may consider Django views to be the “controllers” and Django templates to be the “views.” This is an unfortunate confusion brought about by differing interpretations of MVC. In Django’s interpretation of MVC, the “view” describes the data that gets presented to the user; it’s not necessarily just how the data looks, but which data is presented. In contrast, Ruby on Rails and similar frameworks suggest that the controller’s job includes deciding which data gets presented to the user, whereas the view is strictly how the data looks, not which data is presented.

Neither interpretation is more “correct” than the other. The important thing is to understand the underlying concepts.

Configuring the Database

With all of that philosophy in mind, let’s start exploring Django’s database layer. First, we need to take care of some initial configuration: we need to tell Django which database server to use and how to connect to it.

We’ll assume you’ve set up a database server, activated it, and created a database within it (e.g., using a CREATE DATABASE statement). SQLite is a special case; in that case, there’s no database to create, because SQLite uses standalone files on the filesystem to store its data.

As with TEMPLATE_DIRS in the previous chapter, database configuration lives in the Django settings file, called settings.py by default. Edit that file and look for the database settings:

DATABASE_ENGINE = ''
DATABASE_NAME = ''
DATABASE_USER = ''
DATABASE_PASSWORD = ''
DATABASE_HOST = ''
DATABASE_PORT = ''

Here’s a rundown of each setting.

DATABASE_ENGINE tells Django which database engine to use. If you’re using a database with Django, DATABASE_ENGINE must be set to one of the strings shown in Table 5-1.

Table 5-1. Database Engine Settings
Setting	Database	Required Adapter
`postgresql`	PostgreSQL	`psycopg` version 1.x, http://www.djangoproject.com/r/python-pgsql/1/.
`postgresql_psycopg2`	PostgreSQL	`psycopg` version 2.x, http://www.djangoproject.com/r/python-pgsql/.
`mysql`	MySQL	`MySQLdb`, http://www.djangoproject.com/r/python-mysql/.
`sqlite3`	SQLite	No adapter needed if using Python 2.5+. Otherwise, `pysqlite`, http://www.djangoproject.com/r/python-sqlite/.
`ado_mssql`	Microsoft SQL Server	`adodbapi` version 2.0.1+, http://www.djangoproject.com/r/python-ado/.
`oracle`	Oracle	`cx_Oracle`, http://www.djangoproject.com/r/python-oracle/.

Note that for whichever database back-end you use, you’ll need to download and install the appropriate database adapter. Each one is available for free on the Web; just follow the links in the “Required Adapter” column in Table 5-1.

DATABASE_NAME tells Django the name of your database. If you’re using SQLite, specify the full filesystem path to the database file on your filesystem (e.g., '/home/django/mydata.db').
DATABASE_USER tells Django which username to use when connecting to your database. If you’re using SQLite, leave this blank.
DATABASE_PASSWORD tells Django which password to use when connecting to your database. If you’re using SQLite or have an empty password, leave this blank.
DATABASE_HOST tells Django which host to use when connecting to your database. If your database is on the same computer as your Django installation (i.e., localhost), leave this blank. If you’re using SQLite, leave this blank.

MySQL is a special case here. If this value starts with a forward slash ('/') and you’re using MySQL, MySQL will connect via a Unix socket to the specified socket, for example:
```
DATABASE_HOST = '/var/run/mysql'
```
If you’re using MySQL and this value doesn’t start with a forward slash, then this value is assumed to be the host.
DATABASE_PORT tells Django which port to use when connecting to your database. If you’re using SQLite, leave this blank. Otherwise, if you leave this blank, the underlying database adapter will use whichever port is default for your given database server. In most cases, the default port is fine, so you can leave this blank.

Once you’ve entered those settings, test your configuration. First, from within the mysite project directory you created in Chapter 2, run the command python manage.py shell.

You’ll notice this starts a Python interactive interpreter. Looks can be deceiving, though! There’s an important difference between running the command python manage.py shell within your Django project directory and the more generic python. The latter is the basic Python shell, but the former tells Django which settings file to use before it starts the shell. This is a key requirement for doing database queries: Django needs to know which settings file to use in order to get your database connection information.

Behind the scenes, python manage.py shell simply assumes that your settings file is in the same directory as manage.py. There are other ways to tell Django which settings module to use, but these subtleties will be covered later. For now, use python manage.py shell whenever you need to drop into the Python interpreter to do Django-specific tinkering.

Once you’ve entered the shell, type these commands to test your database configuration:

>>> from django.db import connection
>>> cursor = connection.cursor()

If nothing happens, then your database is configured properly. Otherwise, check the error message for clues about what’s wrong. Table 5-2 shows some common errors.

Table 5-2. Database Configuration Error Messages
Error Message	Solution
You haven’t set the DATABASE_ENGINE setting yet.	Set the `DATABASE_ENGINE` setting to something other than an empty string.
Environment variable DJANGO_SETTINGS_MODULE is undefined.	Run the command `python manage.py shell` rather than `python`.
Error loading _____ module: No module named _____.	You haven’t installed the appropriate database-specific adapter (e.g., `psycopg` or `MySQLdb`).
_____ isn’t an available database backend.	Set your `DATABASE_ENGINE` setting to one of the valid engine settings described previously. Perhaps you made a typo?
database _____ does not exist	Change the `DATABASE_NAME` setting to point to a database that exists, or execute the appropriate `CREATE DATABASE` statement in order to create it.
role _____ does not exist	Change the `DATABASE_USER` setting to point to a user that exists, or create the user in your database.
could not connect to server	Make sure `DATABASE_HOST` and `DATABASE_PORT` are set correctly, and make sure the server is running.

Your First App

Now that you’ve verified the connection is working, it’s time to create a Django app — a bundle of Django code, including models and views, that lives together in a single Python package and represents a full Django application.

It’s worth explaining the terminology here, because this tends to trip up beginners. We’d already created a project, in Chapter 2, so what’s the difference between a project and an app? The difference is that of configuration vs. code:

A project is an instance of a certain set of Django apps, plus the configuration for those apps.

Technically, the only requirement of a project is that it supplies a settings file, which defines the database connection information, the list of installed apps, the TEMPLATE_DIRS, and so forth.
An app is a portable set of Django functionality, usually including models and views, that lives together in a single Python package.

For example, Django comes with a number of apps, such as a commenting system and an automatic admin interface. A key thing to note about these apps is that they’re portable and reusable across multiple projects.

There are very few hard-and-fast rules about how you fit your Django code into this scheme; it’s flexible. If you’re building a simple Web site, you may use only a single app. If you’re building a complex Web site with several unrelated pieces such as an e-commerce system and a message board, you’ll probably want to split those into separate apps so that you’ll be able to reuse them individually in the future.

Indeed, you don’t necessarily need to create apps at all, as evidenced by the example view functions we’ve created so far in this book. In those cases, we simply created a file called views.py, filled it with view functions, and pointed our URLconf at those functions. No “apps” were needed.

However, there’s one requirement regarding the app convention: if you’re using Django’s database layer (models), you must create a Django app. Models must live within apps. Thus, in order to start writing our models, we’ll need to create a new app.

Within the mysite project directory you created in Chapter 2, type this command to create a new app named books:

python manage.py startapp books

This command does not produce any output, but it does create a books directory within the mysite directory. Let’s look at the contents of that directory:

books/
    __init__.py
    models.py
    views.py

These files will contain the models and views for this app.

Have a look at models.py and views.py in your favorite text editor. Both files are empty, except for an import in models.py. This is the blank slate for your Django app.

Defining Models in Python

As we discussed earlier in this chapter, the “M” in “MTV” stands for “Model.” A Django model is a description of the data in your database, represented as Python code. It’s your data layout — the equivalent of your SQL CREATE TABLE statements — except it’s in Python instead of SQL, and it includes more than just database column definitions. Django uses a model to execute SQL code behind the scenes and return convenient Python data structures representing the rows in your database tables. Django also uses models to represent higher-level concepts that SQL can’t necessarily handle.

If you’re familiar with databases, your immediate thought might be, “Isn’t it redundant to define data models in Python and in SQL?” Django works the way it does for several reasons:

Introspection requires overhead and is imperfect. In order to provide convenient data-access APIs, Django needs to know the database layout somehow, and there are two ways of accomplishing this. The first way would be to explicitly describe the data in Python, and the second way would be to introspect the database at runtime to determine the data models.

This second way seems cleaner, because the metadata about your tables lives in only one place, but it introduces a few problems. First, introspecting a database at runtime obviously requires overhead. If the framework had to introspect the database each time it processed a request, or even when the Web server was initialized, this would incur an unacceptable level of overhead. (While some believe that level of overhead is acceptable, Django’s developers aim to trim as much framework overhead as possible, and this approach has succeeded in making Django faster than its high-level framework competitors in benchmarks.) Second, some databases, notably older versions of MySQL, do not store sufficient metadata for accurate and complete introspection.
Writing Python is fun, and keeping everything in Python limits the number of times your brain has to do a “context switch.” It helps productivity if you keep yourself in a single programming environment/mentality for as long as possible. Having to write SQL, then Python, and then SQL again is disruptive.
Having data models stored as code rather than in your database makes it easier to keep your models under version control. This way, you can easily keep track of changes to your data layouts.
SQL allows for only a certain level of metadata about a data layout. Most database systems, for example, do not provide a specialized data type for representing email addresses or URLs. Django models do. The advantage of higher-level data types is higher productivity and more reusable code.
SQL is inconsistent across database platforms. If you’re distributing a Web application, for example, it’s much more pragmatic to distribute a Python module that describes your data layout than separate sets of CREATE TABLE statements for MySQL, PostgreSQL, and SQLite.

A drawback of this approach, however, is that it’s possible for the Python code to get out of sync with what’s actually in the database. If you make changes to a Django model, you’ll need to make the same changes inside your database to keep your database consistent with the model. We’ll detail some strategies for handling this problem later in this chapter.

Finally, we should note that Django includes a utility that can generate models by introspecting an existing database. This is useful for quickly getting up and running with legacy data.

Your First Model

As an ongoing example in this chapter and the next chapter, we’ll focus on a basic book/author/publisher data layout. We use this as our example because the conceptual relationships between books, authors, and publishers are well known, and this is a common data layout used in introductory SQL textbooks. You’re also reading a book that was written by authors and produced by a publisher!

We’ll suppose the following concepts, fields, and relationships:

An author has a salutation (e.g., Mr. or Mrs.), a first name, a last name, an email address, and a headshot photo.
A publisher has a name, a street address, a city, a state/province, a country, and a Web site.
A book has a title and a publication date. It also has one or more authors (a many-to-many relationship with authors) and a single publisher (a one-to-many relationship — aka foreign key — to publishers).

The first step in using this database layout with Django is to express it as Python code. In the models.py file that was created by the startapp command, enter the following:

from django.db import models

class Publisher(models.Model):
    name = models.CharField(maxlength=30)
    address = models.CharField(maxlength=50)
    city = models.CharField(maxlength=60)
    state_province = models.CharField(maxlength=30)
    country = models.CharField(maxlength=50)
    website = models.URLField()

class Author(models.Model):
    salutation = models.CharField(maxlength=10)
    first_name = models.CharField(maxlength=30)
    last_name = models.CharField(maxlength=40)
    email = models.EmailField()
    headshot = models.ImageField(upload_to='/tmp')

class Book(models.Model):
    title = models.CharField(maxlength=100)
    authors = models.ManyToManyField(Author)
    publisher = models.ForeignKey(Publisher)
    publication_date = models.DateField()

Let’s quickly examine this code to cover the basics. The first thing to notice is that each model is represented by a Python class that is a subclass of django.db.models.Model. The parent class, Model, contains all the machinery necessary to make these objects capable of interacting with a database — and that leaves our models responsible solely for defining their fields, in a nice and compact syntax. Believe it or not, this is all the code we need to write to have basic data access with Django.

Each model generally corresponds to a single database table, and each attribute on a model generally corresponds to a column in that database table. The attribute name corresponds to the column’s name, and the type of field (e.g., CharField) corresponds to the database column type (e.g., varchar). For example, the Publisher model is equivalent to the following table (assuming PostgreSQL CREATE TABLE syntax):

CREATE TABLE "books_publisher" (
    "id" serial NOT NULL PRIMARY KEY,
    "name" varchar(30) NOT NULL,
    "address" varchar(50) NOT NULL,
    "city" varchar(60) NOT NULL,
    "state_province" varchar(30) NOT NULL,
    "country" varchar(50) NOT NULL,
    "website" varchar(200) NOT NULL
);

Indeed, Django can generate that CREATE TABLE statement automatically, as we’ll show in a moment.

The exception to the one-class-per-database-table rule is the case of many-to-many relationships. In our example models, Book has a ManyToManyField called authors. This designates that a book has one or many authors, but the Book database table doesn’t get an authors column. Rather, Django creates an additional table — a many-to-many “join table” — that handles the mapping of books to authors.

For a full list of field types and model syntax options, see Appendix B.

Finally, note we haven’t explicitly defined a primary key in any of these models. Unless you instruct it otherwise, Django automatically gives every model an integer primary key field called id. Each Django model is required to have a single-column primary key.

Installing the Model

We’ve written the code; now let’s create the tables in our database. In order to do that, the first step is to activate these models in our Django project. We do that by adding the books app to the list of installed apps in the settings file.

Edit the settings.py file again, and look for the INSTALLED_APPS setting. INSTALLED_APPS tells Django which apps are activated for a given project. By default, it looks something like this:

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
)

Temporarily comment out all four of those strings by putting a hash character (#) in front of them. (They’re included by default as a common-case convenience, but we’ll activate and discuss them later.) While you’re at it, modify the default MIDDLEWARE_CLASSES and TEMPLATE_CONTEXT_PROCESSORS settings. These depend on some of the apps we just commented out. Then, add 'mysite.books' to the INSTALLED_APPS list, so the setting ends up looking like this:

MIDDLEWARE_CLASSES = (
#    'django.middleware.common.CommonMiddleware',
#    'django.contrib.sessions.middleware.SessionMiddleware',
#    'django.contrib.auth.middleware.AuthenticationMiddleware',
#    'django.middleware.doc.XViewMiddleware',
)

TEMPLATE_CONTEXT_PROCESSORS = ()
#...

INSTALLED_APPS = (
    #'django.contrib.auth',
    #'django.contrib.contenttypes',
    #'django.contrib.sessions',
    #'django.contrib.sites',
    'mysite.books',
)

(As we’re dealing with a single-element tuple here, don’t forget the trailing comma. By the way, this book’s authors prefer to put a comma after every element of a tuple, regardless of whether the tuple has only a single element. This avoids the issue of forgetting commas, and there’s no penalty for using that extra comma.)

'mysite.books' refers to the books app we’re working on. Each app in INSTALLED_APPS is represented by its full Python path — that is, the path of packages, separated by dots, leading to the app package.

Now that the Django app has been activated in the settings file, we can create the database tables in our database. First, let’s validate the models by running this command:

python manage.py validate

The validate command checks whether your models’ syntax and logic are correct. If all is well, you’ll see the message 0 errors found. If you don’t, make sure you typed in the model code correctly. The error output should give you helpful information about what was wrong with the code.

Any time you think you have problems with your models, run python manage.py validate. It tends to catch all the common model problems.

If your models are valid, run the following command for Django to generate CREATE TABLE statements for your models in the books app (with colorful syntax highlighting available if you’re using Unix):

python manage.py sqlall books

In this command, books is the name of the app. It’s what you specified when you ran the command manage.py startapp. When you run the command, you should see something like this:

BEGIN;
CREATE TABLE "books_publisher" (
    "id" serial NOT NULL PRIMARY KEY,
    "name" varchar(30) NOT NULL,
    "address" varchar(50) NOT NULL,
    "city" varchar(60) NOT NULL,
    "state_province" varchar(30) NOT NULL,
    "country" varchar(50) NOT NULL,
    "website" varchar(200) NOT NULL
);
CREATE TABLE "books_book" (
    "id" serial NOT NULL PRIMARY KEY,
    "title" varchar(100) NOT NULL,
    "publisher_id" integer NOT NULL REFERENCES "books_publisher" ("id"),
    "publication_date" date NOT NULL
);
CREATE TABLE "books_author" (
    "id" serial NOT NULL PRIMARY KEY,
    "salutation" varchar(10) NOT NULL,
    "first_name" varchar(30) NOT NULL,
    "last_name" varchar(40) NOT NULL,
    "email" varchar(75) NOT NULL,
    "headshot" varchar(100) NOT NULL
);
CREATE TABLE "books_book_authors" (
    "id" serial NOT NULL PRIMARY KEY,
    "book_id" integer NOT NULL REFERENCES "books_book" ("id"),
    "author_id" integer NOT NULL REFERENCES "books_author" ("id"),
    UNIQUE ("book_id", "author_id")
);
CREATE INDEX books_book_publisher_id ON "books_book" ("publisher_id");
COMMIT;

Note the following:

Table names are automatically generated by combining the name of the app (books) and the lowercase name of the model (publisher, book, and author). You can override this behavior, as detailed in Appendix B.
As we mentioned earlier, Django adds a primary key for each table automatically — the id fields. You can override this, too.
By convention, Django appends "_id" to the foreign key field name. As you might have guessed, you can override this behavior, too.
The foreign key relationship is made explicit by a REFERENCES statement.
These CREATE TABLE statements are tailored to the database you’re using, so database-specific field types such as auto_increment (MySQL), serial (PostgreSQL), or integer primary key (SQLite) are handled for you automatically. The same goes for quoting of column names (e.g., using double quotes or single quotes). This example output is in PostgreSQL syntax.

The sqlall command doesn’t actually create the tables or otherwise touch your database — it just prints output to the screen so you can see what SQL Django would execute if you asked it. If you wanted to, you could copy and paste this SQL into your database client, or use Unix pipes to pass it directly. However, Django provides an easier way of committing the SQL to the database. Run the syncdb command, like so:

python manage.py syncdb

You’ll see something like this:

Creating table books_publisher
Creating table books_book
Creating table books_author
Installing index for books.Book model

The syncdb command is a simple “sync” of your models to your database. It looks at all of the models in each app in your INSTALLED_APPS setting, checks the database to see whether the appropriate tables exist yet, and creates the tables if they don’t yet exist. Note that syncdb does not sync changes in models or deletions of models; if you make a change to a model or delete a model, and you want to update the database, syncdb will not handle that. (More on this later.)

If you run python manage.py syncdb again, nothing happens, because you haven’t added any models to the books app or added any apps to INSTALLED_APPS. Ergo, it’s always safe to run python manage.py syncdb — it won’t clobber things.

If you’re interested, take a moment to dive into your database server’s command-line client and see the database tables Django created. You can manually run the command-line client (e.g., psql for PostgreSQL) or you can run the command python manage.py dbshell, which will figure out which command-line client to run, depending on your DATABASE_SERVER setting. The latter is almost always more convenient.

Basic Data Access

Once you’ve created a model, Django automatically provides a high-level Python API for working with those models. Try it out by running python manage.py shell and typing the following:

>>> from books.models import Publisher
>>> p1 = Publisher(name='Addison-Wesley', address='75 Arlington Street',
...     city='Boston', state_province='MA', country='U.S.A.',
...     website='http://www.apress.com/')
>>> p1.save()
>>> p2 = Publisher(name="O'Reilly", address='10 Fawcett St.',
...     city='Cambridge', state_province='MA', country='U.S.A.',
...     website='http://www.oreilly.com/')
>>> p2.save()
>>> publisher_list = Publisher.objects.all()
>>> publisher_list
[<Publisher: Publisher object>, <Publisher: Publisher object>]

These few lines of code accomplish quite a bit. Here are the highlights:

To create an object, just import the appropriate model class and instantiate it by passing in values for each field.
To save the object to the database, call the save() method on the object. Behind the scenes, Django executes an SQL INSERT statement here.
To retrieve objects from the database, use the attribute Publisher.objects. Fetch a list of all Publisher objects in the database with the statement Publisher.objects.all(). Behind the scenes, Django executes an SQL SELECT statement here.

Naturally, you can do quite a lot with the Django database API — but first, let’s take care of a small annoyance.

Adding Model String Representations

When we printed out the list of publishers, all we got was this unhelpful display that makes it difficult to tell the Publisher objects apart:

[<Publisher: Publisher object>, <Publisher: Publisher object>]

We can fix this easily by adding a method called __str__() to our Publisher object. A __str__() method tells Python how to display the “string” representation of an object. You can see this in action by adding a __str__() method to the three models:

from django.db import models

class Publisher(models.Model):
    name = models.CharField(maxlength=30)
    address = models.CharField(maxlength=50)
    city = models.CharField(maxlength=60)
    state_province = models.CharField(maxlength=30)
    country = models.CharField(maxlength=50)
    website = models.URLField()

    def __str__(self):
        return self.name

class Author(models.Model):
    salutation = models.CharField(maxlength=10)
    first_name = models.CharField(maxlength=30)
    last_name = models.CharField(maxlength=40)
    email = models.EmailField()
    headshot = models.ImageField(upload_to='/tmp')

    def __str__(self):
        return '%s %s' % (self.first_name, self.last_name)

class Book(models.Model):
    title = models.CharField(maxlength=100)
    authors = models.ManyToManyField(Author)
    publisher = models.ForeignKey(Publisher)
    publication_date = models.DateField()

    def __str__(self):
        return self.title

As you can see, a __str__() method can do whatever it needs to do in order to return a string representation. Here, the __str__() methods for Publisher and Book simply return the object’s name and title, respectively, but the __str__() for Author is slightly more complex — it pieces together the first_name and last_name fields. The only requirement for __str__() is that it return a string. If __str__() doesn’t return a string — if it returns, say, an integer — then Python will raise a TypeError with a message like "__str__ returned non-string".

For the changes to take effect, exit out of the Python shell and enter it again with python manage.py shell. (This is the simplest way to make code changes take effect.) Now the list of Publisher objects is much easier to understand:

>>> from books.models import Publisher
>>> publisher_list = Publisher.objects.all()
>>> publisher_list
[<Publisher: Addison-Wesley>, <Publisher: O'Reilly>]

Make sure any model you define has a __str__() method — not only for your own convenience when using the interactive interpreter, but also because Django uses the output of __str__() in several places when it needs to display objects.

Finally, note that __str__() is a good example of adding behavior to models. A Django model describes more than the database table layout for an object; it also describes any functionality that object knows how to do. __str__() is one example of such functionality — a model knows how to display itself.

Inserting and Updating Data

You’ve already seen this done: to insert a row into your database, first create an instance of your model using keyword arguments, like so:

>>> p = Publisher(name='Apress',
...         address='2855 Telegraph Ave.',
...         city='Berkeley',
...         state_province='CA',
...         country='U.S.A.',
...         website='http://www.apress.com/')

This act of instantiating a model class does not touch the database.

To save the record into the database (i.e., to perform the SQL INSERT statement), call the object’s save() method:

>>> p.save()

In SQL, this can roughly be translated into the following:

INSERT INTO book_publisher
    (name, address, city, state_province, country, website)
VALUES
    ('Apress', '2855 Telegraph Ave.', 'Berkeley', 'CA',
     'U.S.A.', 'http://www.apress.com/');

Because the Publisher model uses an autoincrementing primary key id, the initial call to save() does one more thing: it calculates the primary key value for the record and sets it to the id attribute on the instance:

>>> p.id
52    # this will differ based on your own data

Subsequent calls to save() will save the record in place, without creating a new record (i.e., performing an SQL UPDATE statement instead of an INSERT):

>>> p.name = 'Apress Publishing'
>>> p.save()

The preceding save() statement will result in roughly the following SQL:

UPDATE book_publisher SET
    name = 'Apress Publishing',
    address = '2855 Telegraph Ave.',
    city = 'Berkeley',
    state_province = 'CA',
    country = 'U.S.A.',
    website = 'http://www.apress.com'
WHERE id = 52;

Selecting Objects

Creating and updating data sure is fun, but it is also useless without a way to sift through that data. We’ve already seen a way to look up all the data for a certain model:

>>> Publisher.objects.all()
[<Publisher: Addison-Wesley>, <Publisher: O'Reilly>, <Publisher: Apress Publishing>]

This roughly translates to this SQL:

SELECT
    id, name, address, city, state_province, country, website
FROM book_publisher;

Note

Notice that Django doesn’t use SELECT * when looking up data and instead lists all fields explicitly. This is by design: in certain circumstances SELECT * can be slower, and (more important) listing fields more closely follows one tenet of the Zen of Python: “Explicit is better than implicit.”

For more on the Zen of Python, try typing import this at a Python prompt.

Let’s take a close look at each part of this Publisher.objects.all() line:

First, we have the model we defined, Publisher. No surprise here: when you want to look up data, you use the model for that data.
Next, we have this objects business. Technically, this is a manager. Managers are discussed in detail in Appendix B. For now, all you need to know is that managers take care of all “table-level” operations on data including, most important, data lookup.

All models automatically get a objects manager; you’ll use it any time you want to look up model instances.
Finally, we have all(). This is a method on the objects manager that returns all the rows in the database. Though this object looks like a list, it’s actually a QuerySet — an object that represents some set of rows from the database. Appendix C deals with QuerySets in detail. For the rest of this chapter, we’ll just treat them like the lists they emulate.

Any database lookup is going to follow this general pattern — we’ll call methods on the manager attached to the model we want to query against.

Filtering Data

While fetching all objects certainly has its uses, most of the time we’re going to want to deal with a subset of the data. We’ll do this with the filter() method:

>>> Publisher.objects.filter(name="Apress Publishing")
[<Publisher: Apress Publishing>]

filter() takes keyword arguments that get translated into the appropriate SQL WHERE clauses. The preceding example would get translated into something like this:

SELECT
    id, name, address, city, state_province, country, website
FROM book_publisher
WHERE name = 'Apress Publishing';

You can pass multiple arguments into filter() to narrow down things further:

>>> Publisher.objects.filter(country="U.S.A.", state_province="CA")
[<Publisher: Apress Publishing>]

Those multiple arguments get translated into SQL AND clauses. Thus, the example in the code snippet translates into the following:

SELECT
    id, name, address, city, state_province, country, website
FROM book_publisher
WHERE country = 'U.S.A.' AND state_province = 'CA';

Notice that by default the lookups use the SQL = operator to do exact match lookups. Other lookup types are available:

>>> Publisher.objects.filter(name__contains="press")
[<Publisher: Apress Publishing>]

That’s a double underscore there between name and contains. Like Python itself, Django uses the double underscore to signal that something “magic” is happening — here, the __contains part gets translated by Django into a SQL LIKE statement:

SELECT
    id, name, address, city, state_province, country, website
FROM book_publisher
WHERE name LIKE '%press%';

Many other types of lookups are available, including icontains (case-insensitive LIKE), startswith and endswith, and range (SQL BETWEEN queries). Appendix C describes all of these lookup types in detail.

Retrieving Single Objects

Sometimes you want to fetch only a single object. That’s what the get() method is for:

>>> Publisher.objects.get(name="Apress Publishing")
<Publisher: Apress Publishing>

Instead of a list (rather, QuerySet), only a single object is returned. Because of that, a query resulting in multiple objects will cause an exception:

>>> Publisher.objects.get(country="U.S.A.")
Traceback (most recent call last):
    ...
AssertionError: get() returned more than one Publisher -- it returned 2!

A query that returns no objects also causes an exception:

>>> Publisher.objects.get(name="Penguin")
Traceback (most recent call last):
    ...
DoesNotExist: Publisher matching query does not exist.

Ordering Data

As you play around with the previous examples, you might discover that the objects are being returned in a seemingly random order. You aren’t imagining things; so far we haven’t told the database how to order its results, so we’re simply getting back data in some arbitrary order chosen by the database.

That’s obviously a bit silly; we wouldn’t want a Web page listing publishers to be ordered randomly. So, in practice, we’ll probably want to use order_by() to reorder our data into a useful list:

>>> Publisher.objects.order_by("name")
[<Publisher: Apress Publishing>, <Publisher: Addison-Wesley>, <Publisher: O'Reilly>]

This doesn’t look much different from the earlier all() example, but the SQL now includes a specific ordering:

SELECT
    id, name, address, city, state_province, country, website
FROM book_publisher
ORDER BY name;

We can order by any field we like:

>>> Publisher.objects.order_by("address")
[<Publisher: O'Reilly>, <Publisher: Apress Publishing>, <Publisher: Addison-Wesley>]

>>> Publisher.objects.order_by("state_province")
[<Publisher: Apress Publishing>, <Publisher: Addison-Wesley>, <Publisher: O'Reilly>]

and by multiple fields:

>>> Publisher.objects.order_by("state_provice", "address")
 [<Publisher: Apress Publishing>, <Publisher: O'Reilly>, <Publisher: Addison-Wesley>]

We can also specify reverse ordering by prefixing the field name with a - (that’s a minus character):

>>> Publisher.objects.order_by("-name")
[<Publisher: O'Reilly>, <Publisher: Apress Publishing>, <Publisher: Addison-Wesley>]

While this flexibility is useful, using order_by() all the time can be quite repetitive. Most of the time you’ll have a particular field you usually want to order by. In these cases, Django lets you attach a default ordering to the model:

class Publisher(models.Model):
    name = models.CharField(maxlength=30)
    address = models.CharField(maxlength=50)
    city = models.CharField(maxlength=60)
    state_province = models.CharField(maxlength=30)
    country = models.CharField(maxlength=50)
    website = models.URLField()

    def __str__(self):
        return self.name

    class Meta:
        ordering = ["name"]

This ordering = ["name"] bit tells Django that unless an ordering is given explicitly with order_by(), all publishers should be ordered by name.

What’s This Meta Thing?

Django uses this internal class Meta as a place to specify additional metadata about a model. It’s completely optional, but it can do some very useful things. See Appendix B for the options you can put under Meta.

Chaining Lookups

You’ve seen how you can filter data, and you’ve seen how you can order it. At times, of course, you’re going to want to do both. In these cases, you simply “chain” the lookups together:

>>> Publisher.objects.filter(country="U.S.A.").order_by("-name")
[<Publisher: O'Reilly>, <Publisher: Apress Publishing>, <Publisher: Addison-Wesley>]

As you might expect, this translates to a SQL query with both a WHERE and an ORDER BY:

SELECT
    id, name, address, city, state_province, country, website
FROM book_publisher
WHERE country = 'U.S.A'
ORDER BY name DESC;

You can keep chaining queries as long as you like. There’s no limit.

Slicing Data

Another common need is to look up only a fixed number of rows. Imagine you have thousands of publishers in your database, but you want to display only the first one. You can do this using Python’s standard list slicing syntax:

>>> Publisher.objects.all()[0]
<Publisher: Addison-Wesley>

This translates roughly to:

SELECT
    id, name, address, city, state_province, country, website
FROM book_publisher
ORDER BY name
LIMIT 1;

And More…

We’ve only just scratched the surface of dealing with models, but you should now know enough to understand all the examples in the rest of the book. When you’re ready to learn the complete details behind object lookups, turn to Appendix C.

Deleting Objects

To delete objects, simply call the delete() method on your object:

>>> p = Publisher.objects.get(name="Addison-Wesley")
>>> p.delete()
>>> Publisher.objects.all()
[<Publisher: Apress Publishing>, <Publisher: O'Reilly>]

You can also delete objects in bulk by calling delete() on the result of some lookup:

>>> publishers = Publisher.objects.all()
>>> publishers.delete()
>>> Publisher.objects.all()
[]

Note

Deletions are permanent, so be careful! In fact, it’s usually a good idea to avoid deleting objects unless you absolutely have to — relational databases don’t do “undo” so well, and restoring from backups is painful.

It’s often a good idea to add “active” flags to your data models. You can look up only “active” objects, and simply set the active field to False instead of deleting the object. Then, if you realize you’ve made a mistake, you can simply flip the flag back.

Making Changes to a Database Schema

When we introduced the syncdb command earlier in this chapter, we noted that syncdb merely creates tables that don’t yet exist in your database — it does not sync changes in models or perform deletions of models. If you add or change a model’s field, or if you delete a model, you’ll need to make the change in your database manually. This section explains how to do that.

When dealing with schema changes, it’s important to keep a few things in mind about how Django’s database layer works:

Django will complain loudly if a model contains a field that has not yet been created in the database table. This will cause an error the first time you use the Django database API to query the given table (i.e., it will happen at code execution time, not at compilation time).
Django does not care if a database table contains columns that are not defined in the model.
Django does not care if a database contains a table that is not represented by a model.

Making schema changes is a matter of changing the various pieces — the Python code and the database itself — in the right order.

Adding Fields

When adding a field to a table/model in a production setting, the trick is to take advantage of the fact that Django doesn’t care if a table contains columns that aren’t defined in the model. The strategy is to add the column in the database, and then update the Django model to include the new field.

However, there’s a bit of a chicken-and-egg problem here, because in order to know how the new database column should be expressed in SQL, you need to look at the output of Django’s manage.py sqlall command, which requires that the field exist in the model. (Note that you’re not required to create your column with exactly the same SQL that Django would, but it’s a good idea to do so, just to be sure everything’s in sync.)

The solution to the chicken-and-egg problem is to use a development environment instead of making the changes on a production server. (You are using a testing/development environment, right?) Here are the detailed steps to take.

First, take these steps in the development environment (i.e., not on the production server):

Add the field to your model.
Run manage.py sqlall [yourapp] to see the new CREATE TABLE statement for the model. Note the column definition for the new field.
Start your database’s interactive shell (e.g., psql or mysql, or you can use manage.py dbshell). Execute an ALTER TABLE statement that adds your new column.
(Optional.) Launch the Python interactive shell with manage.py shell and verify that the new field was added properly by importing the model and selecting from the table (e.g., MyModel.objects.all()[:5]).

Then on the production server perform these steps:

Start your database’s interactive shell.
Execute the ALTER TABLE statement you used in step 3 of the development environment steps.
Add the field to your model. If you’re using source-code revision control and you checked in your change in development environment step 1, now is the time to update the code (e.g., svn update, with Subversion) on the production server.
Restart the Web server for the code changes to take effect.

For example, let’s walk through what we’d do if we added a num_pages field to the Book model described earlier in this chapter. First, we’d alter the model in our development environment to look like this:

class Book(models.Model):
    title = models.CharField(maxlength=100)
    authors = models.ManyToManyField(Author)
    publisher = models.ForeignKey(Publisher)
    publication_date = models.DateField()
    num_pages = models.IntegerField(blank=True, null=True)

    def __str__(self):
        return self.title

(Note: Read the “Adding NOT NULL Columns” sidebar for important details on why we included blank=True and null=True.)

Then we’d run the command manage.py sqlall books to see the CREATE TABLE statement. It would look something like this:

CREATE TABLE "books_book" (
    "id" serial NOT NULL PRIMARY KEY,
    "title" varchar(100) NOT NULL,
    "publisher_id" integer NOT NULL REFERENCES "books_publisher" ("id"),
    "publication_date" date NOT NULL,
    "num_pages" integer NULL
);

The new column is represented like this:

"num_pages" integer NULL

Next, we’d start the database’s interactive shell for our development database by typing psql (for PostgreSQL), and we’d execute the following statements:

ALTER TABLE books_book ADD COLUMN num_pages integer;

Adding NOT NULL Columns

There’s a subtlety here that deserves mention. When we added the num_pages field to our model, we included the blank=True and null=True options. We did this because a database column will contain NULL values when you first create it.

However, it’s also possible to add columns that cannot contain NULL values. To do this, you have to create the column as NULL, then populate the column’s values using some default(s), and then alter the column to set the NOT NULL modifier. For example:

BEGIN;
ALTER TABLE books_book ADD COLUMN num_pages integer;
UPDATE books_book SET num_pages=0;
ALTER TABLE books_book ALTER COLUMN num_pages SET NOT NULL;
COMMIT;

If you go down this path, remember that you should leave off blank=True and null=True in your model.

After the ALTER TABLE statement, we’d verify that the change worked properly by starting the Python shell and running this code:

>>> from mysite.books.models import Book
>>> Book.objects.all()[:5]

If that code didn’t cause errors, we’d switch to our production server and execute the ALTER TABLE statement on the production database. Then, we’d update the model in the production environment and restart the Web server.

Removing Fields

Removing a field from a model is a lot easier than adding one. To remove a field, just follow these steps:

Remove the field from your model and restart the Web server.
Remove the column from your database, using a command like this:
```
ALTER TABLE books_book DROP COLUMN num_pages;
```

Removing Many-to-Many Fields

Because many-to-many fields are different than normal fields, the removal process is different:

Remove the ManyToManyField from your model and restart the Web server.
Remove the many-to-many table from your database, using a command like this:
```
DROP TABLE books_books_publishers;
```

Removing Models

Removing a model entirely is as easy as removing a field. To remove a model, just follow these steps:

Remove the model from your models.py file and restart the Web server.
Remove the table from your database, using a command like this:
```
DROP TABLE books_book;
```

What’s Next?

Once you’ve defined your models, the next step is to populate your database with data. You might have legacy data, in which case Chapter 16 will give you advice about integrating with legacy databases. You might rely on site users to supply your data, in which case Chapter 7 will teach you how to process user-submitted form data.

But in some cases, you or your team might need to enter data manually, in which case it would be helpful to have a Web-based interface for entering and managing data. The next chapter covers Django’s admin interface, which exists precisely for that reason.

Chapter 6: The Django Administration Site

For a certain class of Web sites, an admin interface is an essential part of the infrastructure. This is a Web-based interface, limited to trusted site administrators, that enables the adding, editing and deletion of site content. The interface you use to post to your blog, the backend site managers use to moderate reader-generated comments, the tool your clients use to update the press releases on the Web site you built for them — these are all examples of admin interfaces.

There’s a problem with admin interfaces, though: it’s boring to build them. Web development is fun when you’re developing public-facing functionality, but building admin interfaces is always the same. You have to authenticate users, display and handle forms, validate input, and so on. It’s boring, and it’s repetitive.

So what’s Django’s approach to these boring, repetitive tasks? It does it all for you—in just a couple of lines of code, no less. With Django, building an admin interface is a solved problem.

This chapter is about Django’s automatic admin interface. This feature works by reading metadata in your model to provide a powerful and production-ready interface that site administrators can start using immediately. Here, we discuss how to activate, use, and customize this feature.

Activating the Admin Interface

We think the admin interface is the coolest part of Django—and most Djangonauts agree—but since not everyone actually needs it, it’s an optional piece. That means there are three steps you’ll need to follow to activate it:

Add admin metadata to your models.

Not all models can (or should) be editable by admin users, so you need to “mark” models that should have an admin interface. You do that by adding an inner Admin class to your model (alongside the Meta class, if you have one). So, to add an admin interface to our Book model from the previous chapter, we use this:
```
class Book(models.Model):
    title = models.CharField(maxlength=100)
    authors = models.ManyToManyField(Author)
    publisher = models.ForeignKey(Publisher)
    publication_date = models.DateField()
    num_pages = models.IntegerField(blank=True, null=True)

    def __str__(self):
        return self.title

    class Admin:
        pass
```
The Admin declaration flags the class as having an admin interface. There are a number of options that you can put beneath Admin, but for now we’re sticking with all the defaults, so we put pass in there to signify to Python that the Admin class is empty.

If you’re following this example with your own code, it’s probably a good idea to add Admin declarations to the Publisher and Author classes at this point.
Install the admin application. Do this by adding "django.contrib.admin" to your INSTALLED_APPS setting.
If you’ve been following along, make sure that "django.contrib.sessions", "django.contrib.auth", and "django.contrib.contenttypes" are uncommented, since the admin application depends on them. Also uncomment all the lines in the MIDDLEWARE_CLASSES setting tuple and delete the TEMPLATE_CONTEXT_PROCESSOR setting to allow it to take the default values again.
Run python manage.py syncdb. This step will install the extra database tables the admin interface uses.

Note

When you first run syncdb with "django.contrib.auth" in INSTALLED_APPS, you’ll be asked about creating a superuser. If you didn’t do so at that time, you’ll need to run django/contrib/auth/bin/create_superuser.py to create an admin user. Otherwise, you won’t be able to log in to the admin interface.
Add the URL pattern to your urls.py. If you’re still using the one created by startproject, the admin URL pattern should be already there, but commented out. Either way, your URL patterns should look like the following:
```
from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^admin/', include('django.contrib.admin.urls')),
)
```

That’s it. Now run python manage.py runserver to start the development server. You’ll see something like this:

Validating models...
0 errors found.

Django version 0.96, using settings 'mysite.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Now you can visit the URL given to you by Django (http://127.0.0.1:8000/admin/ in the preceding example), log in, and play around.

Using the Admin Interface

The admin interface is designed to be used by nontechnical users, and as such it should be pretty self-explanatory. Nevertheless, a few notes about the features of the admin interface are in order.

The first thing you’ll see is a login screen, as shown in Figure 6-1.

Figure 6-1. Django’s login screen

You’ll use the username and password you set up when you added your superuser. Once you’re logged in, you’ll see that you can manage users, groups, and permissions (more on that shortly).

Each object given an Admin declaration shows up on the main index page, as shown in Figure 6-2.

Screenshot of the main Django admin index.

Figure 6-2. The main Django admin index

Links to add and change objects lead to two pages we refer to as object change lists and edit forms. Change lists are essentially index pages of objects in the system, as shown in Figure 6-3.

Screenshot of a typical change list view.

Figure 6-3. A typical change list view

A number of options control which fields appear on these lists and the appearance of extra features like date drill-downs, search fields, and filter interfaces. We discuss these features in more detail shortly.

Edit forms are used to modify existing objects and create new ones (see Figure 6-4). Each field defined in your model appears here, and you’ll notice that fields of different types get different widgets (e.g., date/time fields have calendar controls, foreign keys use a select box, etc.).

Figure 6-4. A typical edit form

You’ll notice that the admin interface also handles input validation for you. Try leaving a required field blank or putting an invalid time into a time field, and you’ll see those errors when you try to save, as shown in Figure 6-5.

Screenshot of an edit form displaying errors.

Figure 6-5. An edit form displaying errors

When you edit an existing object, you’ll notice a History button in the upper-right corner of the window. Every change made through the admin interface is logged, and you can examine this log by clicking the History button (see Figure 6-6).

Screenshot of Django’s object history page.

Figure 6-6. Django’s object history page

When you delete an existing object, the admin interface asks you to confirm the delete action to avoid costly mistakes. Deletions also cascade; the deletion confirmation page shows you all the related objects that will be deleted as well (see Figure 6-7).

Screenshot of Django’s delete confirmation page.

Figure 6-7. Django’s delete confirmation page

Users, Groups, and Permissions

Since you’re logged in as a superuser, you have access to create, edit, and delete any object. However, the admin interface has a user permissions system that you can use to give other users access only to the portions of the interface that they need.

You edit these users and permissions through the admin interface just like any other object. The link to the User and Group models is there on the admin index along with all the objects you’ve defined yourself.

User objects have the standard username, password, e-mail, and real name fields you might expect, along with a set of fields that define what the user is allowed to do in the admin interface. First, there’s a set of three flags:

The “is active” flag controls whether the user is active at all. If this flag is off, the user has no access to any URLs that require login.
The “is staff” flag controls whether the user is allowed to log in to the admin interface (i.e., whether that user is considered a “staff member” in your organization). Since this same user system can be used to control access to public (i.e., non-admin) sites (see Chapter 12), this flag differentiates between public users and administrators.
The “is superuser” flag gives the user full, unfettered access to every item in the admin interface; regular permissions are ignored.

“Normal” admin users—that is, active, non-superuser staff members—are granted access that depends on a set of assigned permissions. Each object editable through the admin interface has three permissions: a create permission, an edit permission, and a delete permission. Assigning permissions to a user grants the user access to do what is described by those permissions.

Note

Access to edit users and permissions is also controlled by this permission system. If you give someone permission to edit users, she will be able to edit her own permissions, which might not be what you want!

You can also assign users to groups. A group is simply a set of permissions to apply to all members of that group. Groups are useful for granting identical permissions to large number of users.

Customizing the Admin Interface

You can customize the way the admin interface looks and behaves in a number of ways. We cover just a few of them in this section as they relate to our Book model; Chapter 17 covers customizing the admin interface in detail.

As it stands now, the change list for our books shows only the string representation of the model we added to its __str__. This works fine for just a few books, but if we had hundreds or thousands of books, it would be very hard to locate a single needle in the haystack. However, we can easily add some display, searching, and filtering functions to this interface. Change the Admin declaration as follows:

class Book(models.Model):
    title = models.CharField(maxlength=100)
    authors = models.ManyToManyField(Author)
    publisher = models.ForeignKey(Publisher)
    publication_date = models.DateField()

    class Admin:
        list_display = ('title', 'publisher', 'publication_date')
        list_filter = ('publisher', 'publication_date')
        ordering = ('-publication_date',)
        search_fields = ('title',)

These four lines of code dramatically change our list interface, as shown in Figure 6-8.

Screenshot of the modified change list page.

Figure 6-8. Modified change list page

Each of those lines instructed the admin interface to construct a different piece of this interface:

The list_display option controls which columns appear in the change list table. By default, the change list displays only a single column that contains the object’s string representation. Here, we’ve changed that to show the title, publisher, and publication date.
The list_filter option creates the filtering bar on the right side of the list. We’ve allowed filtering by date (which allows you to see only books published in the last week, month, etc.) and by publisher.

You can instruct the admin interface to filter by any field, but foreign keys, dates, Booleans, and fields with a choices attribute work best. The filters show up as long as there are at least 2 values to choose from.
The ordering option controls the order in which the objects are presented in the admin interface. It’s simply a list of fields by which to order the results; prefixing a field with a minus sign reverses the given order. In this example, we’re ordering by publication date, with the most recent first.
Finally, the search_fields option creates a field that allows text searches. It allows searches by the title field (so you could type Django to show all books with “Django” in the title).

Using these options (and the others described in Chapter 12) you can, with only a few lines of code, make a very powerful, production-ready interface for data editing.

Customizing the Admin Interface’s Look and Feel

Clearly, having the phrase “Django administration” at the top of each admin page is ridiculous. It’s just placeholder text.

It’s easy to change, though, using Django’s template system. The Django admin site is powered by Django itself, and its interfaces use Django’s own template system. (Django’s template system was covered in Chapter 4.)

As we explained in Chapter 4, the TEMPLATE_DIRS setting specifies a list of directories to check when loading Django templates. To customize Django’s admin templates, simply copy the relevant stock admin template from the Django distribution into your one of the directories pointed-to by TEMPLATE_DIRS.

The admin site finds the “Django administration” header by looking for the template admin/base_site.html. By default, this template lives in the Django admin template directory, django/contrib/admin/templates, which you can find by looking in your Python site-packages directory, or wherever Django was installed. To customize this base_site.html template, copy that template into an admin subdirectory of whichever directory you’re using in TEMPLATE_DIRS. For example, if your TEMPLATE_DIRS includes "/home/mytemplates", then copy django/contrib/admin/templates/admin/base_site.html to /home/mytemplates/admin/base_site.html. Don’t forget that admin subdirectory.

Then, just edit the new admin/base_site.html file to replace the generic Django text with your own site’s name as you see fit.

Note that any of Django’s default admin templates can be overridden. To override a template, just do the same thing you did with base_site.html: copy it from the default directory into your custom directory and make changes to the copy.

You might wonder how, if TEMPLATE_DIRS was empty by default, Django found the default admin templates. The answer is that, by default, Django automatically looks for templates within a templates/ subdirectory in each application package as a fallback. See the “Writing Custom Template Loaders” in Chapter 10 for more information about how this works.

Customizing the Admin Index Page

On a similar note, you might want to customize the look and feel of the Django admin index page. By default, it displays all available applications, according to your INSTALLED_APPS setting, sorted by the name of the application. You might, however, want to change this order to make it easier to find the applications you’re looking for. After all, the index is probably the most important page of the admin interface, so it should be easy to use.

The template to customize is admin/index.html. (Remember to copy admin/index.html to your custom template directory as in the previous example.) Edit the file, and you’ll see it uses a template tag called {% get_admin_app_list as app_list %}. This tag retrieves every installed Django application. Instead of using the tag, you can hard-code links to object-specific admin pages in whatever way you think is best. If hard-coding links doesn’t appeal to you, see Chapter 10 for details on implementing your own template tags.

Django offers another shortcut in this department. Run the command python manage.py adminindex <app> to get a chunk of template code for inclusion in the admin index template. It’s a useful starting point.

For full details on customizing the look and feel of the Django admin site in general, see Chapter 17.

When and Why to Use the Admin Interface

We think Django’s admin interface is pretty spectacular. In fact, we’d call it one of Django’s “killer features.” However, we often get asked about “use cases” for the admin interface—when do we use it, and why? Over the years, we’ve discovered a number of patterns for using the admin interface that we think might be helpful.

Obviously, the admin interface is extremely useful for editing data (fancy that). If you have any sort of data entry tasks, the admin interface simply can’t be beat. We suspect that the vast majority of readers of this book will have a whole host of data entry tasks.

Django’s admin interface especially shines when nontechnical users need to be able to enter data; that’s the purpose behind the feature, after all. At the newspaper where Django was first developed, development of a typical online feature—a special report on water quality in the municipal supply, say—goes something like this:

The reporter responsible for the story meets with one of the developers and goes over the available data.
The developer designs a model around this data and then opens up the admin interface to the reporter.
While the reporter enters data into Django, the programmer can focus on developing the publicly accessible interface (the fun part!).

In other words, the raison d’être of Django’s admin interface is facilitating the simultaneous work of content producers and programmers.

However, beyond the obvious data entry tasks, we find the admin interface useful in a few other cases:

Inspecting data models: The first thing we do when we’ve defined a new model is to call it up in the admin interface and enter some dummy data. This is usually when we find any data modeling mistakes; having a graphical interface to a model quickly reveals problems.
Managing acquired data: There’s little actual data entry associated with a site like http://chicagocrime.org, since most of the data comes from an automated source. However, when problems with the automatically acquired data crop up, it’s useful to be able to go in and edit that data easily.

What’s Next?

So far we’ve created a few models and configured a top-notch interface for editing data. In the next chapter, we’ll move on to the real “meat and potatoes” of Web development: form creation and processing.

Chapter 7: Form Processing

Guest author: Simon Willison

After following along with the last chapter, you should now have a fully functioning if somewhat simple site. In this chapter, we’ll deal with the next piece of the puzzle: building views that take input from readers.

We’ll start by making a simple search form “by hand” and looking at how to handle data submitted from the browser. From there, we’ll move on to using Django’s forms framework.

Search

The Web is all about search. Two of the Net’s biggest success stories, Google and Yahoo, built their multi-billion-dollar businesses around search. Nearly every site sees a large percentage of traffic coming to and from its search pages. Often the difference between the success or failure of a site is the quality of its search. So it looks like we’d better add some searching to our fledgling books site, no?

We’ll start by adding the search view to our URLconf (mysite.urls). Recall that this means adding something like (r'^search/$', 'mysite.books.views.search') to the set of URL patterns.

Next, we’ll write this search view into our view module (mysite.books.views):

from django.db.models import Q
from django.shortcuts import render_to_response
from models import Book

def search(request):
    query = request.GET.get('q', '')
    if query:
        qset = (
            Q(title__icontains=query) |
            Q(authors__first_name__icontains=query) |
            Q(authors__last_name__icontains=query)
        )
        results = Book.objects.filter(qset).distinct()
    else:
        results = []
    return render_to_response("books/search.html", {
        "results": results,
        "query": query
    })

There are a couple of things going on here that you haven’t yet seen. First, there’s request.GET. This is how you access GET data from Django; POST data is accessed through a similar request.POST object. These objects behave exactly like standard Python dictionaries with some extra features covered in Appendix H.

What’s GET and POST Data?

GET and POST are the two methods that browsers use to send data to a server. Most of the time, you’ll see them in HTML form tags:

<form action="/books/search/" method="get">

This instructs the browser to submit the form data to the URL /books/search/ using the GET method.

There are important differences between the semantics of GET and POST that we won’t get into right now, but see http://www.w3.org/2001/tag/doc/whenToUseGet.html if you want to learn more.

So the line:

query = request.GET.get('q', '')

looks for a GET parameter named q and returns an empty string if that parameter wasn’t submitted.

Note that we’re using the get() method on request.GET, which is potentially confusing. The get() method here is the one that every Python dictionary has. We’re using it here to be careful: it is not safe to assume that request.GET contains a 'q' key, so we use get('q', '') to provide a default fallback value of '' (the empty string). If we merely accessed the variable using request.GET['q'], that code would raise a KeyError if q wasn’t available in the GET data.

Second, what about this Q business? Q objects are used to build up complex queries — in this case, we’re searching for any books where either the title or the name of one of the authors matches the search query. Technically, these Q objects comprise a QuerySet, and you can read more about them in Appendix C.

In these queries, icontains is a case-insensitive search that uses the SQL LIKE operator in the underlying database.

Since we’re searching against a many-to-many field, it’s possible for the same book to be returned more than once by the query (e.g., a book with two authors who both match the search query). Adding .distinct() to the filter lookup eliminates any duplicate results.

There’s still no template for this search view, however. This should do the trick:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>
    <title>Search{% if query %} Results{% endif %}</title>
</head>
<body>
  <h1>Search</h1>
  <form action="." method="GET">
    <label for="q">Search: </label>
    <input type="text" name="q" value="{{ query|escape }}">
    <input type="submit" value="Search">
  </form>

  {% if query %}
    <h2>Results for "{{ query|escape }}":</h2>

    {% if results %}
      <ul>
      {% for book in results %}
        <li>{{ book|escape }}</l1>
      {% endfor %}
      </ul>
    {% else %}
      <p>No books found</p>
    {% endif %}
  {% endif %}
</body>
</html>

Hopefully by now what this does is fairly obvious. However, there are a few subtleties worth pointing out:

The form’s action is ., which means “the current URL.” This is a standard best practice: don’t use separate views for the form page and the results page; use a single one that serves the form and search results.
We reinsert the value of the query back into the <input>. This lets readers easily refine their searches without having to retype what they searched for.
Everywhere query and book is used, we pass it through the escape filter to make sure that any potentially malicious search text is filtered out before being inserted into the page.

It’s vital that you do this with any user-submitted content! Otherwise you open your site up to cross-site scripting (XSS) attacks. Chapter 19 discusses XSS and security in more detail.
However, we don’t need to worry about harmful content in your database lookups — we can simply pass the query into the lookup as is. This is because Django’s database layer handles this aspect of security for you.

Now we have a working search. A further improvement would be putting a search form on every page (i.e., in the base template); we’ll let you handle that one yourself.

Next, we’ll look at a more complex example. But before we do, let’s discuss a more abstract topic: the “perfect form.”

The “Perfect Form”

Forms can often be a major cause of frustration for the users of your site. Let’s consider the behavior of a hypothetical perfect form:

It should ask the user for some information, obviously. Accessibility and usability matter here, so smart use of the HTML <label> element and useful contextual help are important.
The submitted data should be subjected to extensive validation. The golden rule of Web application security is “never trust incoming data,” so validation is essential.
If the user has made any mistakes, the form should be redisplayed with detailed, informative error messages. The original data should be prefilled, to save the user from having to reenter everything.
The form should continue to redisplay until all of the fields have been correctly filled.

Constructing the perfect form seems like a lot of work! Thankfully, Django’s forms framework is designed to do most of the work for you. You provide a description of the form’s fields, validation rules, and a simple template, and Django does the rest. The result is a “perfect form” with very little effort.

Creating a Feedback Form

The best way to build a site that people love is to listen to their feedback. Many sites appear to have forgotten this; they hide their contact details behind layers of FAQs, and they seem to make it as difficult as possible to get in touch with an actual human being.

When your site has millions of users, this may be a reasonable strategy. When you’re trying to build up an audience, though, you should actively encourage feedback at every opportunity. Let’s build a simple feedback form and use it to illustrate Django’s forms framework in action.

We’ll start by adding adding (r'^contact/$', 'mysite.books.views.contact') to the URLconf, then defining our form. Forms in Django are created in a similar way to models: declaratively, using a Python class. Here’s the class for our simple form. By convention, we’ll insert it into a new forms.py file within our application directory:

from django import newforms as forms

TOPIC_CHOICES = (
    ('general', 'General enquiry'),
    ('bug', 'Bug report'),
    ('suggestion', 'Suggestion'),
)

class ContactForm(forms.Form):
    topic = forms.ChoiceField(choices=TOPIC_CHOICES)
    message = forms.CharField()
    sender = forms.EmailField(required=False)

“New” Forms? What?

When Django was first released to the public, it had a complicated, confusing forms system. It made producing forms far too difficult, so it was completely rewritten and is now called “newforms.” However, there’s still a fair amount of code that depends on the “old” form system, so for the time being Django ships with two form packages.

As we write this book, Django’s old form system is still available as django.forms and the new form package as django.newforms. At some point that will change and django.forms will point to the new form package. However, to make sure the examples in this book work as widely as possible, all the examples will refer to django.newforms.

A Django form is a subclass of django.newforms.Form, just as a Django model is a subclass of django.db.models.Model. The django.newforms module also contains a number of Field classes; a full list is available in Django’s documentation at http://www.djangoproject.com/documentation/0.96/newforms/.

Our ContactForm consists of three fields: a topic, which is a choice among three options; a message, which is a character field; and a sender, which is an email field and is optional (because even anonymous feedback can be useful). There are a number of other field types available, and you can write your own if they don’t cover your needs.

The form object itself knows how to do a number of useful things. It can validate a collection of data, it can generate its own HTML “widgets,” it can construct a set of useful error messages and, if we’re feeling lazy, it can even draw the entire form for us. Let’s hook it into a view and see it in action. In views.py:

from django.db.models import Q
from django.shortcuts import render_to_response
from models import Book
from forms import ContactForm

def search(request):
    query = request.GET.get('q', '')
    if query:
        qset = (
            Q(title__icontains=query) |
            Q(authors__first_name__icontains=query) |
            Q(authors__last_name__icontains=query)
        )
        results = Book.objects.filter(qset).distinct()
    else:
        results = []
    return render_to_response("books/search.html", {
        "results": results,
        "query": query
    })

def contact(request):
    form = ContactForm()
    return render_to_response('contact.html', {'form': form})

and in contact.html:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>
    <title>Contact us</title>
</head>
<body>
    <h1>Contact us</h1>
    <form action="." method="POST">
        <table>
            {{ form.as_table }}
        </table>
        <p><input type="submit" value="Submit"></p>
    </form>
</body>
</html>

The most interesting line here is {{ form.as_table }}. form is our ContactForm instance, as passed to render_to_response. as_table is a method on that object that renders the form as a sequence of table rows (as_ul and as_p can also be used). The generated HTML looks like this:

<tr>
    <th><label for="id_topic">Topic:</label></th>
    <td>
        <select name="topic" id="id_topic">
            <option value="general">General enquiry</option>
            <option value="bug">Bug report</option>
            <option value="suggestion">Suggestion</option>
        </select>
    </td>
</tr>
<tr>
    <th><label for="id_message">Message:</label></th>
    <td><input type="text" name="message" id="id_message" /></td>
</tr>
<tr>
    <th><label for="id_sender">Sender:</label></th>
    <td><input type="text" name="sender" id="id_sender" /></td>
</tr>

Note that the <table> and <form> tags are not included; you need to define those yourself in the template, which gives you control over how the form behaves when it is submitted. Label elements are included, making forms accessible out of the box.

Our form is currently using a <input type="text"> widget for the message field. We don’t want to restrict our users to a single line of text, so we’ll swap in a <textarea> widget instead:

class ContactForm(forms.Form):
    topic = forms.ChoiceField(choices=TOPIC_CHOICES)
    message = forms.CharField(widget=forms.Textarea())
    sender = forms.EmailField(required=False)

The forms framework separates out the presentation logic for each field into a set of widgets. Each field type has a default widget, but you can easily override the default, or provide a custom widget of your own.

At the moment, submitting the form doesn’t actually do anything. Let’s hook in our validation rules:

def contact(request):
    if request.method == 'POST':
        form = ContactForm(request.POST)
    else:
        form = ContactForm()
    return render_to_response('contact.html', {'form': form})

A form instance can be in one of two states: bound or unbound. A bound instance is constructed with a dictionary (or dictionary-like object) and knows how to validate and redisplay the data from it. An unbound form has no data associated with it and simply knows how to display itself.

Try clicking Submit on the blank form. The page should redisplay, showing a validation error that informs us that our message field is required.

Try entering an invalid email address as well. The EmailField knows how to validate email addresses, at least to a reasonable level of doubt.

Setting Initial Data

Passing data directly to the form constructor binds that data and indicates that validation should be performed. Often, though, we need to display an initial form with some of the fields prefilled — for example, an “edit” form. We can do this with the initial keyword argument:

form = CommentForm(initial={'sender': 'user@example.com'})

If our form will always use the same default values, we can configure them in the form definition itself:

message = forms.CharField(widget=forms.Textarea(),
                          initial="Replace with your feedback")

Processing the Submission

Once the user has filled the form to the point that it passes our validation rules, we need to do something useful with the data. In this case, we want to construct and send an email containing the user’s feedback. We’ll use Django’s email package to do this.

First, though, we need to tell if the data is indeed valid, and if it is, we need access to the validated data. The forms framework does more than just validate the data, it also converts it into Python types. Our contact form only deals with strings, but if we were to use an IntegerField or DateTimeField, the forms framework would ensure that we got back a Python integer or datetime object, respectively.

To tell whether a form is bound to valid data, call the is_valid() method:

form = ContactForm(request.POST)
if form.is_valid():
    # Process form data

Now we need access to the data. We could pull it straight out of request.POST, but if we did, we’d miss out on the type conversions performed by the forms framework. Instead, we use form.clean_data:

if form.is_valid():
    topic = form.clean_data['topic']
    message = form.clean_data['message']
    sender = form.clean_data.get('sender', 'noreply@example.com')
    # ...

Note that since sender is not required, we provide a default when it’s missing. Finally, we need to record the user’s feedback. The easiest way to do this is to email it to a site administrator. We can do that using the send_mail function:

from django.core.mail import send_mail

# ...

send_mail(
    'Feedback from your site, topic: %s' % topic,
    message, sender,
    ['administrator@example.com']
)

The send_mail function has four required arguments: the email subject, the email body, the “from” address, and a list of recipient addresses. send_mail is a convenient wrapper around Django’s EmailMessage class, which provides advanced features such as attachments, multipart emails, and full control over email headers.

Having sent the feedback email, we’ll redirect our user to a static confirmation page. The finished view function looks like this:

from django.http import HttpResponseRedirect
from django.shortcuts import render_to_response
from django.core.mail import send_mail
from forms import ContactForm

def contact(request):
    if request.method == 'POST':
        form = ContactForm(request.POST)
        if form.is_valid():
            topic = form.clean_data['topic']
            message = form.clean_data['message']
            sender = form.clean_data.get('sender', 'noreply@example.com')
            send_mail(
                'Feedback from your site, topic: %s' % topic,
                message, sender,
                ['administrator@example.com']
            )
            return HttpResponseRedirect('/contact/thanks/')
    else:
        form = ContactForm()
    return render_to_response('contact.html', {'form': form})

Redirect After POST

If a user selects Refresh on a page that was displayed by a POST request, that request will be repeated. This can often lead to undesired behavior, such as a duplicate record being added to the database. Redirect after POST is a useful pattern that can help avoid this scenario: after a successful POST has been processed, redirect the user to another page rather than returning HTML directly.

Custom Validation Rules

Imagine we’ve launched our feedback form, and the emails have started tumbling in. There’s just one problem: some of the emails are just one or two words, hardly enough for a detailed missive. We decide to adopt a new validation policy: four words or more, please.

There are a number of ways to hook custom validation into a Django form. If our rule is something we will reuse again and again, we can create a custom field type. Most custom validations are one-off affairs, though, and can be tied directly to the form class.

We want additional validation on the message field, so we need to add a clean_message method to our form:

class ContactForm(forms.Form):
    topic = forms.ChoiceField(choices=TOPIC_CHOICES)
    message = forms.CharField(widget=forms.Textarea())
    sender = forms.EmailField(required=False)

    def clean_message(self):
        message = self.clean_data.get('message', '')
        num_words = len(message.split())
        if num_words < 4:
            raise forms.ValidationError("Not enough words!")
        return message

This new method will be called after the default field validator (in this case, the validator for a required CharField). Because the field data has already been partially processed, we need to pull it out of the form’s clean_data dictionary.

We naively use a combination of len() and split() to count the number of words. If the user has entered too few words, we raise a ValidationError. The string attached to this exception will be displayed to the user as an item in the error list.

It is important that we explicitly return the value for the field at the end of the method. This allows us to modify the value (or convert it to a different Python type) within our custom validation method. If we forget the return statement, then None will be returned, and the original value will be lost.

A Custom Look and Feel

The quickest way to customize the form’s presentation is with CSS. The list of errors in particular could do with some visual enhancement, and the <ul> has a class attribute of errorlist for that exact purpose. The following CSS really makes our errors stand out:

<style type="text/css">
    ul.errorlist {
        margin: 0;
        padding: 0;
    }
    .errorlist li {
        background-color: red;
        color: white;
        display: block;
        font-size: 10px;
        margin: 0 0 3px;
        padding: 4px 5px;
    }
</style>

While it’s convenient to have our form’s HTML generated for us, in many cases the default rendering won’t be right for our application. {{ form.as_table }} and friends are useful shortcuts while we develop our application, but everything about the way a form is displayed can be overridden, mostly within the template itself.

Each field widget (<input type="text">, <select>, <textarea>, or similar) can be rendered individually by accessing {{ form.fieldname }}. Any errors associated with a field are available as {{ form.fieldname.errors }}. We can use these form variables to construct a custom template for our contact form:

<form action="." method="POST">
    <div class="fieldWrapper">
        {{ form.topic.errors }}
        <label for="id_topic">Kind of feedback:</label>
        {{ form.topic }}
    </div>
    <div class="fieldWrapper">
        {{ form.message.errors }}
        <label for="id_message">Your message:</label>
        {{ form.message }}
    </div>
    <div class="fieldWrapper">
        {{ form.sender.errors }}
        <label for="id_sender">Your email (optional):</label>
        {{ form.sender }}
    </div>
    <p><input type="submit" value="Submit"></p>
</form>

{{ form.message.errors }} will display as a <ul class="errorlist"> if errors are present and a blank string if the field is valid (or the form is unbound). We can also treat form.message.errors as a Boolean or even iterate over it as a list, for example:

<div class="fieldWrapper{% if form.message.errors %} errors{% endif %}">
    {% if form.message.errors %}
        <ol>
        {% for error in form.message.errors %}
            <li><strong>{{ error|escape }}</strong></li>
        {% endfor %}
        </ol>
    {% endif %}
    {{ form.message }}
</div>

In the case of validation errors, this will add an “errors” class to the containing <div> and display the list of errors in an ordered list.

Creating Forms from Models

Let’s build something a little more interesting: a form that submits a new publisher to our book application from Chapter 5.

An important rule of thumb in software development that Django tries to adhere to is Don’t Repeat Yourself (DRY). Andy Hunt and Dave Thomas in The Pragmatic Programmer define this as follows:

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

Our Publisher model class says that a publisher has a name, address, city, state_province, country, and website. Duplicating this information in a form definition would break the DRY rule. Instead, we can use a useful shortcut: form_for_model():

from models import Publisher
from django.newforms import form_for_model

PublisherForm = form_for_model(Publisher)

PublisherForm is a Form subclass, just like the ContactForm class we created manually earlier on. We can use it in much the same way:

from forms import PublisherForm

def add_publisher(request):
    if request.method == 'POST':
        form = PublisherForm(request.POST)
        if form.is_valid():
            form.save()
            return HttpResponseRedirect('/add_publisher/thanks/')
    else:
        form = PublisherForm()
    return render_to_response('books/add_publisher.html', {'form': form})

The add_publisher.html file is almost identical to our original contact.html template, so it has been omitted. Also remember to add a new pattern to the URLconf: (r'^add_publisher/$', 'mysite.books.views.add_publisher').

There’s one more shortcut being demonstrated here. Since forms derived from models are often used to save new instances of the model to the database, the form class created by form_for_model includes a convenient save() method. This deals with the common case; you’re welcome to ignore it if you want to do something a bit more involved with the submitted data.

form_for_instance() is a related method that can create a preinitialized form from an instance of a model class. This is useful for creating “edit” forms.

What’s Next?

This chapter concludes the introductory material in this book. The next 13 chapters deal with various advanced topics, including generating content other than HTML (Chapter 11), security (Chapter 19), and deployment (Chapter 20).

After these first seven chapters, you should know enough to start writing your own Django projects. The rest of the material in this book will help fill in the missing pieces as you need them.

We’ll start in Chapter 8 by doubling back and taking a closer look at views and URLconfs (introduced first in Chapter 3).

Chapter 8: Advanced Views and URLconfs

In Chapter 3, we explained the basics of Django view functions and URLconfs. This chapter goes into more detail about advanced functionality in those two pieces of the framework.

URLconf Tricks

There’s nothing “special” about URLconfs — like anything else in Django, they’re just Python code. You can take advantage of this in several ways, as described in the sections that follow.

Streamlining Function Imports

Consider this URLconf, which builds on the example in Chapter 3:

from django.conf.urls.defaults import *
from mysite.views import current_datetime, hours_ahead, hours_behind, now_in_chicago, now_in_london

urlpatterns = patterns('',
    (r'^now/$', current_datetime),
    (r'^now/plus(\d{1,2})hours/$', hours_ahead),
    (r'^now/minus(\d{1,2})hours/$', hours_behind),
    (r'^now/in_chicago/$', now_in_chicago),
    (r'^now/in_london/$', now_in_london),
)

As explained in Chapter 3, each entry in the URLconf includes its associated view function, passed directly as a function object. This means it’s necessary to import the view functions at the top of the module.

But as a Django application grows in complexity, its URLconf grows, too, and keeping those imports can be tedious to manage. (For each new view function, you have to remember to import it, and the import statement tends to get overly long if you use this approach.) It’s possible to avoid that tedium by importing the views module itself. This example URLconf is equivalent to the previous one:

from django.conf.urls.defaults import *
from mysite import views

urlpatterns = patterns('',
    (r'^now/$', views.current_datetime),
    (r'^now/plus(\d{1,2})hours/$', views.hours_ahead),
    (r'^now/minus(\d{1,2})hours/$', views.hours_behind),
    (r'^now/in_chicago/$', views.now_in_chicago),
    (r'^now/in_london/$', views.now_in_london),
)

Django offers another way of specifying the view function for a particular pattern in the URLconf: you can pass a string containing the module name and function name rather than the function object itself. Continuing the ongoing example:

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^now/$', 'mysite.views.current_datetime'),
    (r'^now/plus(\d{1,2})hours/$', 'mysite.views.hours_ahead'),
    (r'^now/minus(\d{1,2})hours/$', 'mysite.views.hours_behind'),
    (r'^now/in_chicago/$', 'mysite.views.now_in_chicago'),
    (r'^now/in_london/$', 'mysite.views.now_in_london'),
)

(Note the quotes around the view names. We’re using 'mysite.views.current_datetime' — with quotes — instead of mysite.views.current_datetime.)

Using this technique, it’s no longer necessary to import the view functions; Django automatically imports the appropriate view function the first time it’s needed, according to the string describing the name and path of the view function.

A further shortcut you can take when using the string technique is to factor out a common “view prefix.” In our URLconf example, each of the view strings starts with 'mysite.views', which is redundant to type. We can factor out that common prefix and pass it as the first argument to patterns(), like this:

from django.conf.urls.defaults import *

urlpatterns = patterns('mysite.views',
    (r'^now/$', 'current_datetime'),
    (r'^now/plus(\d{1,2})hours/$', 'hours_ahead'),
    (r'^now/minus(\d{1,2})hours/$', 'hours_behind'),
    (r'^now/in_chicago/$', 'now_in_chicago'),
    (r'^now/in_london/$', 'now_in_london'),
)

Note that you don’t put a trailing dot (".") in the prefix, nor do you put a leading dot in the view strings. Django puts those in automatically.

With these two approaches in mind, which is better? It really depends on your personal coding style and needs.

Advantages of the string approach are as follows:

It’s more compact, because it doesn’t require you to import the view functions.
It results in more readable and manageable URLconfs if your view functions are spread across several different Python modules.

Advantages of the function object approach are as follows:

It allows for easy “wrapping” of view functions. See the section “Wrapping View Functions” later in this chapter.
It’s more “Pythonic” — that is, it’s more in line with Python traditions, such as passing functions as objects.

Both approaches are valid, and you can even mix them within the same URLconf. The choice is yours.

Using Multiple View Prefixes

In practice, if you use the string technique, you’ll probably end up mixing views to the point where the views in your URLconf won’t have a common prefix. However, you can still take advantage of the view prefix shortcut to remove duplication. Just add multiple patterns() objects together, like this:

Old:

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^/?$', 'mysite.views.archive_index'),
    (r'^(\d{4})/([a-z]{3})/$', 'mysite.views.archive_month'),
    (r'^tag/(\w+)/$', 'weblog.views.tag'),
)

New:

from django.conf.urls.defaults import *

urlpatterns = patterns('mysite.views',
    (r'^/?$', 'archive_index'),
    (r'^(\d{4})/([a-z]{3})/$', 'archive_month'),
)

urlpatterns += patterns('weblog.views',
    (r'^tag/(\w+)/$', 'tag'),
)

All the framework cares about is that there’s a module-level variable called urlpatterns. This variable can be constructed dynamically, as we do in this example.

Special-Casing URLs in Debug Mode

Speaking of constructing urlpatterns dynamically, you might want to take advantage of this technique to alter your URLconf’s behavior while in Django’s debug mode. To do this, just check the value of the DEBUG setting at runtime, like so:

from django.conf.urls.defaults import*
from django.conf import settings

urlpatterns = patterns('',
    (r'^$', 'mysite.views.homepage'),
    (r'^(\d{4})/([a-z]{3})/$', 'mysite.views.archive_month'),
)

if settings.DEBUG:
    urlpatterns += patterns('',
        (r'^debuginfo$', 'mysite.views.debug'),
    )

In this example, the URL /debuginfo/ will only be available if your DEBUG setting is set to True.

Using Named Groups

In all of our URLconf examples so far, we’ve used simple, non-named regular expression groups — that is, we put parentheses around parts of the URL we wanted to capture, and Django passes that captured text to the view function as a positional argument. In more advanced usage, it’s possible to use named regular expression groups to capture URL bits and pass them as keyword arguments to a view.

Keyword Arguments vs. Positional Arguments

A Python function can be called using keyword arguments or positional arguments — and, in some cases, both at the same time. In a keyword argument call, you specify the names of the arguments along with the values you’re passing. In a positional argument call, you simply pass the arguments without explicitly specifying which argument matches which value; the association is implicit in the arguments’ order.

For example, consider this simple function:

def sell(item, price, quantity):
    print "Selling %s unit(s) of %s at %s" % (quantity, item, price)

To call it with positional arguments, you specify the arguments in the order in which they’re listed in the function definition:

sell('Socks', '$2.50', 6)

To call it with keyword arguments, you specify the names of the arguments along with the values. The following statements are equivalent:

sell(item='Socks', price='$2.50', quantity=6)
sell(item='Socks', quantity=6, price='$2.50')
sell(price='$2.50', item='Socks', quantity=6)
sell(price='$2.50', quantity=6, item='Socks')
sell(quantity=6, item='Socks', price='$2.50')
sell(quantity=6, price='$2.50', item='Socks')

Finally, you can mix keyword and positional arguments, as long as all positional arguments are listed before keyword arguments. The following statements are equivalent to the previous examples:

sell('Socks', '$2.50', quantity=6)
sell('Socks', price='$2.50', quantity=6)
sell('Socks', quantity=6, price='$2.50')

In Python regular expressions, the syntax for named regular expression groups is (?P<name>pattern), where name is the name of the group and pattern is some pattern to match.

Here’s an example URLconf that uses non-named groups:

from django.conf.urls.defaults import *
from mysite import views

urlpatterns = patterns('',
    (r'^articles/(\d{4})/$', views.year_archive),
    (r'^articles/(\d{4})/(\d{2})/$', views.month_archive),
)

Here’s the same URLconf, rewritten to use named groups:

from django.conf.urls.defaults import *
from mysite import views

urlpatterns = patterns('',
    (r'^articles/(?P<year>\d{4})/$', views.year_archive),
    (r'^articles/(?P<year>\d{4})/(?P<month>\d{2})/$', views.month_archive),
)

This accomplishes exactly the same thing as the previous example, with one subtle difference: the captured values are passed to view functions as keyword arguments rather than positional arguments.

For example, with non-named groups, a request to /articles/2006/03/ would result in a function call equivalent to this:

month_archive(request, '2006', '03')

With named groups, though, the same request would result in this function call:

month_archive(request, year='2006', month='03')

In practice, using named groups makes your URLconfs slightly more explicit and less prone to argument-order bugs — and you can reorder the arguments in your views’ function definitions. Following the preceding example, if we wanted to change the URLs to include the month before the year, and we were using non-named groups, we’d have to remember to change the order of arguments in the month_archive view. If we were using named groups, changing the order of the captured parameters in the URL would have no effect on the view.

Of course, the benefits of named groups come at the cost of brevity; some developers find the named-group syntax ugly and too verbose. Still, another advantage of named groups is readability, especially by those who aren’t intimately familiar with regular expressions or your particular Django application. It’s easier to see what’s happening, at a glance, in a URLconf that uses named groups.

Understanding the Matching/Grouping Algorithm

A caveat with using named groups in a URLconf is that a single URLconf pattern cannot contain both named and non-named groups. If you do this, Django won’t throw any errors, but you’ll probably find that your URLs aren’t matching as you expect. Specifically, here’s the algorithm the URLconf parser follows, with respect to named groups vs. non-named groups in a regular expression:

If there are any named arguments, it will use those, ignoring non-named arguments.
Otherwise, it will pass all non-named arguments as positional arguments.
In both cases, it will pass any extra options as keyword arguments. See the next section for more information.

Passing Extra Options to View Functions

Sometimes you’ll find yourself writing view functions that are quite similar, with only a few small differences. For example, say you have two views whose contents are identical except for the template they use:

# urls.py

from django.conf.urls.defaults import *
from mysite import views

urlpatterns = patterns('',
    (r'^foo/$', views.foo_view),
    (r'^bar/$', views.bar_view),
)

# views.py

from django.shortcuts import render_to_response
from mysite.models import MyModel

def foo_view(request):
    m_list = MyModel.objects.filter(is_new=True)
    return render_to_response('template1.html', {'m_list': m_list})

def bar_view(request):
    m_list = MyModel.objects.filter(is_new=True)
    return render_to_response('template2.html', {'m_list': m_list})

We’re repeating ourselves in this code, and that’s inelegant. At first, you may think to remove the redundancy by using the same view for both URLs, putting parentheses around the URL to capture it, and checking the URL within the view to determine the template, like so:

# urls.py

from django.conf.urls.defaults import *
from mysite import views

urlpatterns = patterns('',
    (r'^(foo)/$', views.foobar_view),
    (r'^(bar)/$', views.foobar_view),
)

# views.py

from django.shortcuts import render_to_response
from mysite.models import MyModel

def foobar_view(request, url):
    m_list = MyModel.objects.filter(is_new=True)
    if url == 'foo':
        template_name = 'template1.html'
    elif url == 'bar':
        template_name = 'template2.html'
    return render_to_response(template_name, {'m_list': m_list})

The problem with that solution, though, is that it couples your URLs to your code. If you decide to rename /foo/ to /fooey/, you’ll have to remember to change the view code.

The elegant solution involves an optional URLconf parameter. Each pattern in a URLconf may include a third item: a dictionary of keyword arguments to pass to the view function.

With this in mind, we can rewrite our ongoing example like this:

# urls.py

from django.conf.urls.defaults import *
from mysite import views

urlpatterns = patterns('',
    (r'^foo/$', views.foobar_view, {'template_name': 'template1.html'}),
    (r'^bar/$', views.foobar_view, {'template_name': 'template2.html'}),
)

# views.py

from django.shortcuts import render_to_response
from mysite.models import MyModel

def foobar_view(request, template_name):
    m_list = MyModel.objects.filter(is_new=True)
    return render_to_response(template_name, {'m_list': m_list})

As you can see, the URLconf in this example specifies template_name in the URLconf. The view function treats it as just another parameter.

This extra URLconf options technique is a nice way of sending additional information to your view functions with minimal fuss. As such, it’s used by a couple of Django’s bundled applications, most notably its generic views system, which we cover in Chapter 9.

The following sections contain a couple of ideas on how you can use the extra URLconf options technique in your own projects.

Faking Captured URLconf Values

Say you have a set of views that match a pattern, along with another URL that doesn’t fit the pattern but whose view logic is the same. In this case, you can “fake” the capturing of URL values by using extra URLconf options to handle that extra URL with the same view.

For example, you might have an application that displays some data for a particular day, with URLs such as these:

/mydata/jan/01/
/mydata/jan/02/
/mydata/jan/03/
# ...
/mydata/dec/30/
/mydata/dec/31/

This is simple enough to deal with — you can capture those in a URLconf like this (using named group syntax):

urlpatterns = patterns('',
    (r'^mydata/(?P<month>\w{3})/(?P<day>\d\d)/$', views.my_view),
)

And the view function signature would look like this:

def my_view(request, month, day):
    # ....

This approach is straightforward — it’s nothing you haven’t seen before. The trick comes in when you want to add another URL that uses my_view but whose URL doesn’t include a month and/or day.

For example, you might want to add another URL, /mydata/birthday/, which would be equivalent to /mydata/jan/06/. You can take advantage of extra URLconf options like so:

urlpatterns = patterns('',
    (r'^mydata/birthday/$', views.my_view, {'month': 'jan', 'day': '06'}),
    (r'^mydata/(?P<month>\w{3})/(?P<day>\d\d)/$', views.my_view),
)

The cool thing here is that you don’t have to change your view function at all. The view function only cares that it gets month and day parameters — it doesn’t matter whether they come from the URL capturing itself or extra parameters.

Making a View Generic

It’s good programming practice to “factor out” commonalities in code. For example, with these two Python functions:

def say_hello(person_name):
    print 'Hello, %s' % person_name

def say_goodbye(person_name):
    print 'Goodbye, %s' % person_name

we can factor out the greeting to make it a parameter:

def greet(person_name, greeting):
    print '%s, %s' % (greeting, person_name)

You can apply this same philosophy to your Django views by using extra URLconf parameters.

With this in mind, you can start making higher-level abstractions of your views. Instead of thinking to yourself, “This view displays a list of Event objects,” and “That view displays a list of BlogEntry objects,” realize they’re both specific cases of “A view that displays a list of objects, where the type of object is variable.”

Take this code, for example:

# urls.py

from django.conf.urls.defaults import *
from mysite import views

urlpatterns = patterns('',
    (r'^events/$', views.event_list),
    (r'^blog/entries/$', views.entry_list),
)

# views.py

from django.shortcuts import render_to_response
from mysite.models import Event, BlogEntry

def event_list(request):
    obj_list = Event.objects.all()
    return render_to_response('mysite/event_list.html', {'event_list': obj_list})

def entry_list(request):
    obj_list = BlogEntry.objects.all()
    return render_to_response('mysite/blogentry_list.html', {'entry_list': obj_list})

The two views do essentially the same thing: they display a list of objects. So let’s factor out the type of object they’re displaying:

# urls.py

from django.conf.urls.defaults import *
from mysite import models, views

urlpatterns = patterns('',
    (r'^events/$', views.object_list, {'model': models.Event}),
    (r'^blog/entries/$', views.object_list, {'model': models.BlogEntry}),
)

# views.py

from django.shortcuts import render_to_response

def object_list(request, model):
    obj_list = model.objects.all()
    template_name = 'mysite/%s_list.html' % model.__name__.lower()
    return render_to_response(template_name, {'object_list': obj_list})

With those small changes, we suddenly have a reusable, model-agnostic view! From now on, anytime we need a view that lists a set of objects, we can simply reuse this object_list view rather than writing view code. Here are a couple of notes about what we did:

We’re passing the model classes directly, as the model parameter. The dictionary of extra URLconf options can pass any type of Python object — not just strings.
The model.objects.all() line is an example of duck typing: “If it walks like a duck and talks like a duck, we can treat it like a duck.” Note the code doesn’t know what type of object model is; the only requirement is that model have an objects attribute, which in turn has an all() method.
We’re using model.__name__.lower() in determining the template name. Every Python class has a __name__ attribute that returns the class name. This feature is useful at times like this, when we don’t know the type of class until runtime. For example, the BlogEntry class’s __name__ is the string 'BlogEntry'.
In a slight difference between this example and the previous example, we’re passing the generic variable name object_list to the template. We could easily change this variable name to be blogentry_list or event_list, but we’ve left that as an exercise for the reader.

Because database-driven Web sites have several common patterns, Django comes with a set of “generic views” that use this exact technique to save you time. We cover Django’s built-in generic views in the next chapter.

Giving a View Configuration Options

If you’re distributing a Django application, chances are that your users will want some degree of configuration. In this case, it’s a good idea to add hooks to your views for any configuration options you think people may want to change. You can use extra URLconf parameters for this purpose.

A common bit of an application to make configurable is the template name:

def my_view(request, template_name):
    var = do_something()
    return render_to_response(template_name, {'var': var})

Understanding Precedence of Captured Values vs. Extra Options

When there’s a conflict, extra URLconf parameters get precedence over captured parameters. In other words, if your URLconf captures a named-group variable and an extra URLconf parameter includes a variable with the same name, the extra URLconf parameter value will be used.

For example, consider this URLconf:

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^mydata/(?P<id>\d+)/$', views.my_view, {'id': 3}),
)

Here, both the regular expression and the extra dictionary include an id. The hard-coded id gets precedence. That means any request (e.g., /mydata/2/ or /mydata/432432/) will be treated as if id is set to 3, regardless of the value captured in the URL.

Astute readers will note that in this case, it’s a waste of time and typing to capture the id in the regular expression, because its value will always be overridden by the dictionary’s value. That’s correct; we bring this up only to help you avoid making the mistake.

Using Default View Arguments

Another convenient trick is to specify default parameters for a view’s arguments. This tells the view which value to use for a parameter by default if none is specified.

Here’s an example:

# urls.py

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^blog/$', views.page),
    (r'^blog/page(?P<num>\d+)/$', views.page),
)

# views.py

def page(request, num="1"):
    # Output the appropriate page of blog entries, according to num.
    # ...

Here, both URL patterns point to the same view — views.page — but the first pattern doesn’t capture anything from the URL. If the first pattern matches, the page() function will use its default argument for num, "1". If the second pattern matches, page() will use whatever num value was captured by the regular expression.

It’s common to use this technique in conjunction with configuration options, as explained earlier. This example makes a slight improvement to the example in the “Giving a View Configuration Options” section by providing a default value for template_name:

def my_view(request, template_name='mysite/my_view.html'):
    var = do_something()
    return render_to_response(template_name, {'var': var})

Special-Casing Views

Sometimes you’ll have a pattern in your URLconf that handles a large set of URLs, but you’ll need to special-case one of them. In this case, take advantage of the linear way a URLconf is processed and put the special case first.

For example, the “add an object” pages in Django’s admin site are represented by this URLconf line:

urlpatterns = patterns('',
    # ...
    ('^([^/]+)/([^/]+)/add/$', 'django.contrib.admin.views.main.add_stage'),
    # ...
)

This matches URLs such as /myblog/entries/add/ and /auth/groups/add/. However, the “add” page for a user object (/auth/user/add/) is a special case — it doesn’t display all of the form fields, it displays two password fields, and so forth. We could solve this problem by special-casing in the view, like so:

def add_stage(request, app_label, model_name):
    if app_label == 'auth' and model_name == 'user':
        # do special-case code
    else:
        # do normal code

but that’s inelegant for a reason we’ve touched on multiple times in this chapter: it puts URL logic in the view. As a more elegant solution, we can take advantage of the fact that URLconfs are processed in order from top to bottom:

urlpatterns = patterns('',
    # ...
    ('^auth/user/add/$', 'django.contrib.admin.views.auth.user_add_stage'),
    ('^([^/]+)/([^/]+)/add/$', 'django.contrib.admin.views.main.add_stage'),
    # ...
)

With this in place, a request to /auth/user/add/ will be handled by the user_add_stage view. Although that URL matches the second pattern, it matches the top one first. (This is short-circuit logic.)

Capturing Text in URLs

Each captured argument is sent to the view as a plain Python string, regardless of what sort of match the regular expression makes. For example, in this URLconf line:

(r'^articles/(?P<year>\d{4})/$', views.year_archive),

the year argument to views.year_archive() will be a string, not an integer, even though \d{4} will only match integer strings.

This is important to keep in mind when you’re writing view code. Many built-in Python functions are fussy (and rightfully so) about accepting only objects of a certain type. A common error is to attempt to create a datetime.date object with string values instead of integer values:

>>> import datetime
>>> datetime.date('1993', '7', '9')
Traceback (most recent call last):
    ...
TypeError: an integer is required
>>> datetime.date(1993, 7, 9)
datetime.date(1993, 7, 9)

Translated to a URLconf and view, the error looks like this:

# urls.py

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^articles/(\d{4})/(\d{2})/(\d{2})/$', views.day_archive),
)

# views.py

import datetime

def day_archive(request, year, month, day)
    # The following statement raises a TypeError!
    date = datetime.date(year, month, day)

Instead, day_archive() can be written correctly like this:

def day_archive(request, year, month, day)
    date = datetime.date(int(year), int(month), int(day))

Note that int() itself raises a ValueError when you pass it a string that is not composed solely of digits, but we’re avoiding that error in this case because the regular expression in our URLconf has ensured that only strings containing digits are passed to the view function.

Determining What the URLconf Searches Against

When a request comes in, Django tries to match the URLconf patterns against the requested URL, as a normal Python string (not as a Unicode string). This does not include GET or POST parameters, or the domain name. It also does not include the leading slash, because every URL has a leading slash.

For example, in a request to http://www.example.com/myapp/, Django will try to match myapp/. In a request to http://www.example.com/myapp/?page=3, Django will try to match myapp/.

The request method (e.g., POST, GET, HEAD) is not taken into account when traversing the URLconf. In other words, all request methods will be routed to the same function for the same URL. It’s the responsibility of a view function to perform branching based on request method.

Including Other URLconfs

If you intend your code to be used on multiple Django-based sites, you should consider arranging your URLconfs in such a way that allows for “including.”

At any point, your URLconf can “include” other URLconf modules. This essentially “roots” a set of URLs below other ones. For example, this URLconf includes other URLconfs:

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^weblog/', include('mysite.blog.urls')),
    (r'^photos/', include('mysite.photos.urls')),
    (r'^about/$', 'mysite.views.about'),
)

There’s an important gotcha here: the regular expressions in this example that point to an include() do not have a $ (end-of-string match character) but do include a trailing slash. Whenever Django encounters include(), it chops off whatever part of the URL matched up to that point and sends the remaining string to the included URLconf for further processing.

Continuing this example, here’s the URLconf mysite.blog.urls:

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^(\d\d\d\d)/$', 'mysite.blog.views.year_detail'),
    (r'^(\d\d\d\d)/(\d\d)/$', 'mysite.blog.views.month_detail'),
)

With these two URLconfs, here’s how a few sample requests would be handled:

/weblog/2007/: In the first URLconf, the pattern r'^weblog/' matches. Because it is an include(), Django strips all the matching text, which is 'weblog/' in this case. The remaining part of the URL is 2007/, which matches the first line in the mysite.blog.urls URLconf.
/weblog//2007/: In the first URLconf, the pattern r'^weblog/' matches. Because it is an include(), Django strips all the matching text, which is 'weblog/' in this case. The remaining part of the URL is /2007/ (with a leading slash), which does not match any of the lines in the mysite.blog.urls URLconf.
/about/: This matches the view mysite.views.about in the first URLconf, demonstrating that you can mix include() patterns with non-include() patterns.

How Captured Parameters Work with include()

An included URLconf receives any captured parameters from parent URLconfs, for example:

# root urls.py

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^(?P<username>\w+)/blog/', include('foo.urls.blog')),
)

# foo/urls/blog.py

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^$', 'foo.views.blog_index'),
    (r'^archive/$', 'foo.views.blog_archive'),
)

In this example, the captured username variable is passed to the included URLconf and, hence, to every view function within that URLconf.

Note that the captured parameters will always be passed to every line in the included URLconf, regardless of whether the line’s view actually accepts those parameters as valid. For this reason, this technique is useful only if you’re certain that every view in the included URLconf accepts the parameters you’re passing.

How Extra URLconf Options Work with include()

Similarly, you can pass extra URLconf options to include(), just as you can pass extra URLconf options to a normal view — as a dictionary. When you do this, each line in the included URLconf will be passed the extra options.

For example, the following two URLconf sets are functionally identical.

Set one:

# urls.py

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^blog/', include('inner'), {'blogid': 3}),
)

# inner.py

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^archive/$', 'mysite.views.archive'),
    (r'^about/$', 'mysite.views.about'),
    (r'^rss/$', 'mysite.views.rss'),
)

Set two:

# urls.py

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^blog/', include('inner')),
)

# inner.py

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    (r'^archive/$', 'mysite.views.archive', {'blogid': 3}),
    (r'^about/$', 'mysite.views.about', {'blogid': 3}),
    (r'^rss/$', 'mysite.views.rss', {'blogid': 3}),
)

As is the case with captured parameters (explained in the previous section), extra options will always be passed to every line in the included URLconf, regardless of whether the line’s view actually accepts those options as valid. For this reason, this technique is useful only if you’re certain that every view in the included URLconf accepts the extra options you’re passing.

What’s Next?

One of Django’s main goals is to reduce the amount of code developers need to write, and in this chapter we suggested how to cut down the code of your views and URLconfs.

The next logical step in code elimination is removing the need to write views entirely. That’s the topic of the next chapter.

Chapter 9: Generic Views

Here again is a recurring theme of this book: at its worst, Web development is boring and monotonous. So far, we’ve covered how Django tries to take away some of that monotony at the model and template layers, but Web developers also experience this boredom at the view level.

Django’s generic views were developed to ease that pain. They take certain common idioms and patterns found in view development and abstract them so that you can quickly write common views of data without having to write too much code. In fact, nearly every view example in the preceding chapters could be rewritten with the help of generic views.

Chapter 8 touched briefly on how you’d go about making a view “generic.” To review, we can recognize certain common tasks, like displaying a list of objects, and write code that displays a list of any object. Then the model in question can be passed as an extra argument to the URLconf.

Django ships with generic views to do the following:

Perform common “simple” tasks: redirect to a different page and render a given template.
Display list and detail pages for a single object. The event_list and entry_list views from Chapter 8 are examples of list views. A single event page is an example of what we call a “detail” view.
Present date-based objects in year/month/day archive pages, associated detail, and “latest” pages. The Django Weblog’s (http://www.djangoproject.com/weblog/) year, month, and day archives are built with these, as would be a typical newspaper’s archives.
Allow users to create, update, and delete objects — with or without authorization.

Taken together, these views provide easy interfaces to perform the most common tasks developers encounter.

Using Generic Views

All of these views are used by creating configuration dictionaries in your URLconf files and passing those dictionaries as the third member of the URLconf tuple for a given pattern.

For example, here’s a simple URLconf you could use to present a static “about” page:

from django.conf.urls.defaults import *
from django.views.generic.simple import direct_to_template

urlpatterns = patterns('',
    ('^about/$', direct_to_template, {
        'template': 'about.html'
    })
)

Though this might seem a bit “magical” at first glance — look, a view with no code! —, it’s actually exactly the same as the examples in Chapter 8: the direct_to_template view simply grabs information from the extra-parameters dictionary and uses that information when rendering the view.

Because this generic view — and all the others — is a regular view functions like any other, we can reuse it inside our own views. As an example, let’s extend our “about” example to map URLs of the form /about/<whatever>/ to statically rendered about/<whatever>.html. We’ll do this by first modifying the URLconf to point to a view function:

from django.conf.urls.defaults import *
from django.views.generic.simple import direct_to_template
from mysite.books.views import about_pages

urlpatterns = patterns('',
    ('^about/$', direct_to_template, {
        'template': 'about.html'
    }),
    ('^about/(w+)/$', about_pages),
)

Next, we’ll write the about_pages view:

from django.http import Http404
from django.template import TemplateDoesNotExist
from django.views.generic.simple import direct_to_template

def about_pages(request, page):
    try:
        return direct_to_template(request, template="about/%s.html" % page)
    except TemplateDoesNotExist:
        raise Http404()

Here we’re treating direct_to_template like any other function. Since it returns an HttpResponse, we can simply return it as-is. The only slightly tricky business here is dealing with missing templates. We don’t want a nonexistent template to cause a server error, so we catch TemplateDoesNotExist exceptions and return 404 errors instead.

Is There a Security Vulnerability Here?

Sharp-eyed readers may have noticed a possible security hole: we’re constructing the template name using interpolated content from the browser (template="about/%s.html" % page). At first glance, this looks like a classic directory traversal vulnerability (discussed in detail in Chapter 19). But is it really?

Not exactly. Yes, a maliciously crafted value of page could cause directory traversal, but although page is taken from the request URL, not every value will be accepted. They key is in the URLconf: we’re using the regular expression \w+ to match the page part of the URL, and \w only accepts letters and numbers. Thus, any malicious characters (dots and slashes, here) will be rejected by the URL resolver before they reach the view itself.

Generic Views of Objects

The direct_to_template certainly is useful, but Django’s generic views really shine when it comes to presenting views on your database content. Because it’s such a common task, Django comes with a handful of built-in generic views that make generating list and detail views of objects incredibly easy.

Let’s take a look at one of these generic views: the “object list” view. We’ll be using this Publisher object from Chapter 5:

class Publisher(models.Model):
    name = models.CharField(maxlength=30)
    address = models.CharField(maxlength=50)
    city = models.CharField(maxlength=60)
    state_province = models.CharField(maxlength=30)
    country = models.CharField(maxlength=50)
    website = models.URLField()

    def __str__(self):
        return self.name

    class Meta:
        ordering = ["-name"]

    class Admin:
        pass

To build a list page of all books, we’d use a URLconf along these lines:

from django.conf.urls.defaults import *
from django.views.generic import list_detail
from mysite.books.models import Publisher

publisher_info = {
    "queryset" : Publisher.objects.all(),
}

urlpatterns = patterns('',
    (r'^publishers/$', list_detail.object_list, publisher_info)
)

That’s all the Python code we need to write. We still need to write a template, however. We could explicitly tell the object_list view which template to use by including a template_name key in the extra arguments dictionary, but in the absence of an explicit template Django will infer one from the object’s name. In this case, the inferred template will be "books/publisher_list.html" — the “books” part comes from the name of the app that defines the model, while the “publisher” bit is just the lowercased version of the model’s name.

This template will be rendered against a context containing a variable called object_list that contains all the book objects. A very simple template might look like the following:

{% extends "base.html" %}

{% block content %}
    <h2>Publishers</h2>
    <ul>
        {% for publisher in object_list %}
            <li>{{ publisher.name }}</li>
        {% endfor %}
    </ul>
{% endblock %}

That’s really all there is to it. All the cool features of generic views come from changing the “info” dictionary passed to the generic view. Appendix D documents all the generic views and all their options in detail; the rest of this chapter will consider some of the common ways you might customize and extend generic views.

Extending Generic Views

There’s no question that using generic views can speed up development substantially. In most projects, however, there comes a moment when the generic views no longer suffice. Indeed, the most common question asked by new Django developers is how to make generic views handle a wider array of situations.

Luckily, in nearly every one of these cases, there are ways to simply extend generic views to handle a larger array of use cases. These situations usually fall into a handful of patterns dealt with in the sections that follow.

Making “Friendly” Template Contexts

You might have noticed that sample publisher list template stores all the books in a variable named object_list. While this works just fine, it isn’t all that “friendly” to template authors: they have to “just know” that they’re dealing with books here. A better name for that variable would be publisher_list; that variable’s content is pretty obvious.

We can change the name of that variable easily with the template_object_name argument:

publisher_info = {
    "queryset" : Publisher.objects.all(),
    "template_object_name" : "publisher",
}

urlpatterns = patterns('',
    (r'^publishers/$', list_detail.object_list, publisher_info)
)

Providing a useful template_object_name is always a good idea. Your coworkers who design templates will thank you.

Adding Extra Context

Often you simply need to present some extra information beyond that provided by the generic view. For example, think of showing a list of all the other publishers on each publisher detail page. The object_detail generic view provides the publisher to the context, but it seems there’s no way to get a list of all publishers in that template.

But there is: all generic views take an extra optional parameter, extra_context. This is a dictionary of extra objects that will be added to the template’s context. So, to provide the list of all publishers on the detail detail view, we’d use an info dict like this:

publisher_info = {
    "queryset" : Publisher.objects.all(),
    "template_object_name" : "publisher",
    "extra_context" : {"book_list" : Book.objects.all()}
}

This would populate a {{ book_list }} variable in the template context. This pattern can be used to pass any information down into the template for the generic view. It’s very handy.

However, there’s actually a subtle bug here — can you spot it?

The problem has to do with when the queries in extra_context are evaluated. Because this example puts Publisher.objects.all() in the URLconf, it will be evaluated only once (when the URLconf is first loaded). Once you add or remove publishers, you’ll notice that the generic view doesn’t reflect those changes until you reload the Web server (see “Caching and QuerySets” in Appendix C for more information about when QuerySets are cached and evaluated).

Note

This problem doesn’t apply to the queryset generic view argument. Since Django knows that particular QuerySet should never be cached, the generic view takes care of clearing the cache when each view is rendered.

The solution is to use a callback in extra_context instead of a value. Any callable (i.e., a function) that’s passed to extra_context will be evaluated when the view is rendered (instead of only once). You could do this with an explicitly defined function:

def get_books():
    return Book.objects.all()

publisher_info = {
    "queryset" : Publisher.objects.all(),
    "template_object_name" : "publisher",
    "extra_context" : {"book_list" : get_books}
}

or you could use a less obvious but shorter version that relies on the fact that Publisher.objects.all is itself a callable:

publisher_info = {
    "queryset" : Publisher.objects.all(),
    "template_object_name" : "publisher",
    "extra_context" : {"book_list" : Book.objects.all}
}

Notice the lack of parentheses after Book.objects.all; this references the function without actually calling it (which the generic view will do later).

Viewing Subsets of Objects

Now let’s take a closer look at this queryset key we’ve been using all along. Most generic views take one of these queryset arguments — it’s how the view knows which set of objects to display (see “Selecting Objects” in Chapter 5 for an introduction to QuerySets, and see Appendix C for the complete details).

To pick a simple example, we might want to order a list of books by publication date, with the most recent first:

book_info = {
    "queryset" : Book.objects.all().order_by("-publication_date"),
}

urlpatterns = patterns('',
    (r'^publishers/$', list_detail.object_list, publisher_info),
    (r'^books/$', list_detail.object_list, book_info),
)

That’s a pretty simple example, but it illustrates the idea nicely. Of course, you’ll usually want to do more than just reorder objects. If you want to present a list of books by a particular publisher, you can use the same technique:

apress_books = {
    "queryset": Book.objects.filter(publisher__name="Apress Publishing"),
    "template_name" : "books/apress_list.html"
}

urlpatterns = patterns('',
    (r'^publishers/$', list_detail.object_list, publisher_info),
    (r'^books/apress/$', list_detail.object_list, apress_books),
)

Notice that along with a filtered queryset, we’re also using a custom template name. If we didn’t, the generic view would use the same template as the “vanilla” object list, which might not be what we want.

Also notice that this isn’t a very elegant way of doing publisher-specific books. If we want to add another publisher page, we’d need another handful of lines in the URLconf, and more than a few publishers would get unreasonable. We’ll deal with this problem in the next section.

Note

If you get a 404 when requesting /books/apress/, check to ensure you actually have a Publisher with the name ‘Apress Publishing’. Generic views have an allow_empty parameter for this case. See Appendix D for more details.

Complex Filtering with Wrapper Functions

Another common need is to filter down the objects given in a list page by some key in the URL. Earlier we hard-coded the publisher’s name in the URLconf, but what if we wanted to write a view that displayed all the books by some arbitrary publisher? We can “wrap” the object_list generic view to avoid writing a lot of code by hand. As usual, we’ll start by writing a URLconf:

urlpatterns = patterns('',
    (r'^publishers/$', list_detail.object_list, publisher_info),
    (r'^books/(w+)/$', books_by_publisher),
)

Next, we’ll write the books_by_publisher view itself:

from django.http import Http404
from django.views.generic import list_detail
from mysite.books.models import Book, Publisher

def books_by_publisher(request, name):

    # Look up the publisher (and raise a 404 if it can't be found).
    try:
        publisher = Publisher.objects.get(name__iexact=name)
    except Publisher.DoesNotExist:
        raise Http404

    # Use the object_list view for the heavy lifting.
    return list_detail.object_list(
        request,
        queryset = Book.objects.filter(publisher=publisher),
        template_name = "books/books_by_publisher.html",
        template_object_name = "books",
        extra_context = {"publisher" : publisher}
    )

This works because there’s really nothing special about generic views — they’re just Python functions. Like any view function, generic views expect a certain set of arguments and return HttpResponse objects. Thus, it’s incredibly easy to wrap a small function around a generic view that does additional work before (or after; see the next section) handing things off to the generic view.

Note

Notice that in the preceding example we passed the current publisher being displayed in the extra_context. This is usually a good idea in wrappers of this nature; it lets the template know which “parent” object is currently being browsed.

Performing Extra Work

The last common pattern we’ll look at involves doing some extra work before or after calling the generic view.

Imagine we had a last_accessed field on our Author object that we were using to keep track of the last time anybody looked at that author. The generic object_detail view, of course, wouldn’t know anything about this field, but once again we could easily write a custom view to keep that field updated.

First, we’d need to add an author detail bit in the URLconf to point to a custom view:

from mysite.books.views import author_detail

urlpatterns = patterns('',
    #...
    (r'^authors/(?P<author_id>d+)/$', author_detail),
)

Then we’d write our wrapper function:

import datetime
from mysite.books.models import Author
from django.views.generic import list_detail
from django.shortcuts import get_object_or_404

def author_detail(request, author_id):
    # Look up the Author (and raise a 404 if she's not found)
    author = get_object_or_404(Author, pk=author_id)

    # Record the last accessed date
    author.last_accessed = datetime.datetime.now()
    author.save()

    # Show the detail page
    return list_detail.object_detail(
        request,
        queryset = Author.objects.all(),
        object_id = author_id,
    )

Note

This code won’t actually work unless you add a last_accessed field to your Author model and create a books/author_detail.html template.

We can use a similar idiom to alter the response returned by the generic view. If we wanted to provide a downloadable plain-text version of the list of authors, we could use a view like this:

def author_list_plaintext(request):
    response = list_detail.object_list(
        request,
        queryset = Author.objects.all(),
        mimetype = "text/plain",
        template_name = "books/author_list.txt"
    )
    response["Content-Disposition"] = "attachment; filename=authors.txt"
    return response

This works because the generic views return simple HttpResponse objects that can be treated like dictionaries to set HTTP headers. This Content-Disposition business, by the way, instructs the browser to download and save the page instead of displaying it in the browser.

What’s Next?

In this chapter we looked at only a couple of the generic views Django ships with, but the general ideas presented here should apply pretty closely to any generic view. Appendix D covers all the available views in detail, and it’s recommended reading if you want to get the most out of this powerful feature.

In the next chapter we delve deep into the inner workings of Django’s templates, showing all the cool ways they can be extended. Until now, we’ve treated the template engine as a mostly static tool you can use to render your content.

Chapter 10: Extending the Template Engine

Although most of your interactions with Django’s template language will be in the role of template author, you may want to customize and extend the template engine — either to make it do something it doesn’t already do, or to make your job easier in some other way.

This chapter delves deep into the guts of Django’s template system. It covers what you need to know if you plan to extend the system or if you’re just curious about how it works.

If you’re looking to use the Django template system as part of another application (i.e., without the rest of the framework), make sure to read the “Configuring the Template System in Standalone Mode” section later in the chapter.

Template Language Review

First, let’s quickly review a number of terms introduced in Chapter 4:

A template is a text document, or a normal Python string, that is marked up using the Django template language. A template can contain block tags and variables.
A block tag is a symbol within a template that does something. This definition is deliberately vague. For example, a block tag can produce content, serve as a control structure (an if statement or for loop), grab content from a database, or enable access to other template tags.

Block tags are surrounded by {% and %}:
```
{% if is_logged_in %}
  Thanks for logging in!
{% else %}
  Please log in.
{% endif %}
```
A variable is a symbol within a template that outputs a value.

Variable tags are surrounded by {{ and }}:
```
My first name is {{ first_name }}. My last name is {{ last_name }}.
```
A context is a name -> value mapping (similar to a Python dictionary) that is passed to a template.
A template renders a context by replacing the variable “holes” with values from the context and executing all block tags.

For more details about the basics of these terms, refer back to Chapter 4.

The rest of this chapter discusses ways of extending the template engine. First, though, let’s take a quick look at a few internals left out of Chapter 4 for simplicity.

RequestContext and Context Processors

When rendering a template, you need a context. Usually this is an instance of django.template.Context, but Django also comes with a special subclass, django.template.RequestContext, that acts slightly differently. RequestContext adds a bunch of variables to your template context by default — things like the HttpRequest object or information about the currently logged-in user.

Use RequestContext when you don’t want to have to specify the same set of variables in a series of templates. For example, consider these four views:

from django.template import loader, Context

def view_1(request):
    # ...
    t = loader.get_template('template1.html')
    c = Context({
        'app': 'My app',
        'user': request.user,
        'ip_address': request.META['REMOTE_ADDR'],
        'message': 'I am view 1.'
    })
    return t.render(c)

def view_2(request):
    # ...
    t = loader.get_template('template2.html')
    c = Context({
        'app': 'My app',
        'user': request.user,
        'ip_address': request.META['REMOTE_ADDR'],
        'message': 'I am the second view.'
    })
    return t.render(c)

def view_3(request):
# ...
    t = loader.get_template('template3.html')
    c = Context({
        'app': 'My app',
        'user': request.user,
        'ip_address': request.META['REMOTE_ADDR'],
        'message': 'I am the third view.'
    })
    return t.render(c)

def view_4(request):
    # ...
    t = loader.get_template('template4.html')
    c = Context({
        'app': 'My app',
        'user': request.user,
        'ip_address': request.META['REMOTE_ADDR'],
        'message': 'I am the fourth view.'
    })
    return t.render(c)

(Note that we’re deliberately not using the render_to_response() shortcut in these examples — we’re manually loading the templates, constructing the context objects and rendering the templates. We’re “spelling out” all of the steps for the purpose of clarity.)

Each view passes the same three variables — app, user and ip_address — to its template. Wouldn’t it be nice if we could remove that redundancy?

RequestContext and context processors were created to solve this problem. Context processors let you specify a number of variables that get set in each context automatically — without you having to specify the variables in each render_to_response() call. The catch is that you have to use RequestContext instead of Context when you render a template.

The most low-level way of using context processors is to create some processors and pass them to RequestContext. Here’s how the above example could be written with context processors:

from django.template import loader, RequestContext

def custom_proc(request):
    "A context processor that provides 'app', 'user' and 'ip_address'."
    return {
        'app': 'My app',
        'user': request.user,
        'ip_address': request.META['REMOTE_ADDR']
    }

def view_1(request):
    # ...
    t = loader.get_template('template1.html')
    c = RequestContext(request, {'message': 'I am view 1.'},
            processors=[custom_proc])
    return t.render(c)

def view_2(request):
    # ...
    t = loader.get_template('template2.html')
    c = RequestContext(request, {'message': 'I am the second view.'},
            processors=[custom_proc])
    return t.render(c)

def view_3(request):
    # ...
    t = loader.get_template('template3.html')
    c = RequestContext(request, {'message': 'I am the third view.'},
            processors=[custom_proc])
    return t.render(c)

def view_4(request):
    # ...
    t = loader.get_template('template4.html')
    c = RequestContext(request, {'message': 'I am the fourth view.'},
            processors=[custom_proc])
    return t.render(c)

Let’s step through this code:

First, we define a function custom_proc. This is a context processor — it takes an HttpRequest object and returns a dictionary of variables to use in the template context. That’s all it does.
We’ve changed the four view functions to use RequestContext instead of Context. There are two differences in how the context is constructed. One, RequestContext requires the first argument to be an HttpRequest object — the one that was passed into the view function in the first place (request). Two, RequestContext takes an optional processors argument, which is a list or tuple of context processor functions to use. Here, we pass in custom_proc, the custom processor we defined above.
Each view no longer has to include app, user or ip_address in its context construction, because those are provided by custom_proc.
Each view still has the flexibility to introduce any custom template variables it might need. In this example, the message template variable is set differently in each view.

In Chapter 4, we introduced the render_to_response() shortcut, which saves you from having to call loader.get_template(), then create a Context, then call the render() method on the template. In order to demonstrate the lower-level workings of context processors, the above examples didn’t use render_to_response(), . But it’s possible — and preferable — to use context processors with render_to_response(). Do this with the context_instance argument, like so:

from django.shortcuts import render_to_response
from django.template import RequestContext

def custom_proc(request):
    "A context processor that provides 'app', 'user' and 'ip_address'."
    return {
        'app': 'My app',
        'user': request.user,
        'ip_address': request.META['REMOTE_ADDR']
    }

def view_1(request):
    # ...
    return render_to_response('template1.html',
        {'message': 'I am view 1.'},
        context_instance=RequestContext(request, processors=[custom_proc]))

def view_2(request):
    # ...
    return render_to_response('template2.html',
        {'message': 'I am the second view.'},
        context_instance=RequestContext(request, processors=[custom_proc]))

def view_3(request):
    # ...
    return render_to_response('template3.html',
        {'message': 'I am the third view.'},
        context_instance=RequestContext(request, processors=[custom_proc]))

def view_4(request):
    # ...
    return render_to_response('template4.html',
        {'message': 'I am the fourth view.'},
        context_instance=RequestContext(request, processors=[custom_proc]))

Here, we’ve trimmed down each view’s template rendering code to a single (wrapped) line.

This is an improvement, but, evaluating the conciseness of this code, we have to admit we’re now almost overdosing on the other end of the spectrum. We’ve removed redundancy in data (our template variables) at the cost of adding redundancy in code (in the processors call). Using context processors doesn’t save you much typing if you have to type processors all the time.

For that reason, Django provides support for global context processors. The TEMPLATE_CONTEXT_PROCESSORS setting designates which context processors should always be applied to RequestContext. This removes the need to specify processors each time you use RequestContext.

By default, TEMPLATE_CONTEXT_PROCESSORS is set to the following:

TEMPLATE_CONTEXT_PROCESSORS = (
    'django.core.context_processors.auth',
    'django.core.context_processors.debug',
    'django.core.context_processors.i18n',
    'django.core.context_processors.media',
)

This setting is a tuple of callables that use the same interface as our custom_proc function above — functions that take a request object as their argument and return a dictionary of items to be merged into the context. Note that the values in TEMPLATE_CONTEXT_PROCESSORS are specified as strings, which means the processors are required to be somewhere on your Python path (so you can refer to them from the setting).

Each processor is applied in order. That is, if one processor adds a variable to the context and a second processor adds a variable with the same name, the second will override the first.

Django provides a number of simple context processors, including the ones that are enabled by default:

django.core.context_processors.auth

If TEMPLATE_CONTEXT_PROCESSORS contains this processor, every RequestContext will contain these variables:

user: A django.contrib.auth.models.User instance representing the current logged-in user (or an AnonymousUser instance, if the client isn’t logged in).
messages: A list of messages (as strings) for the current logged-in user. Behind the scenes, this variable calls request.user.get_and_delete_messages() for every request. That method collects the user’s messages and deletes them from the database.
perms: An instance of django.core.context_processors.PermWrapper, which represents the permissions the current logged-in user has.

See Chapter 12 for more information on users, permissions, and messages.

django.core.context_processors.debug

This processor pushes debugging information down to the template layer. If TEMPLATE_CONTEXT_PROCESSORS contains this processor, every RequestContext will contain these variables:

debug: The value of your DEBUG setting (either True or False). You can use this variable in templates to test whether you’re in debug mode.
sql_queries: A list of {'sql': ..., 'time': ...} dictionaries representing every SQL query that has happened so far during the request and how long it took. The list is in the order in which the queries were issued.

Because debugging information is sensitive, this context processor will only add variables to the context if both of the following conditions are true:

The DEBUG setting is True.
The request came from an IP address in the INTERNAL_IPS setting.

django.core.context_processors.i18n

If this processor is enabled, every RequestContext will contain these variables:

LANGUAGES: The value of the LANGUAGES setting.
LANGUAGE_CODE: request.LANGUAGE_CODE if it exists; otherwise, the value of the LANGUAGE_CODE setting.

Appendix E provides more information about these two settings.

django.core.context_processors.request

If this processor is enabled, every RequestContext will contain a variable request, which is the current HttpRequest object. Note that this processor is not enabled by default; you have to activate it.

Guidelines for Writing Your Own Context Processors

Here are a few tips for rolling your own:

Make each context processor responsible for the smallest subset of functionality possible. It’s easy to use multiple processors, so you might as well split functionality into logical pieces for future reuse.
Keep in mind that any context processor in TEMPLATE_CONTEXT_PROCESSORS will be available in every template powered by that settings file, so try to pick variable names that are unlikely to conflict with variable names your templates might be using independently. As variable names are case-sensitive, it’s not a bad idea to use all caps for variables a processor provides.
It doesn’t matter where on the filesystem they live, as long as they’re on your Python path so you can point to them from the TEMPLATE_CONTEXT_PROCESSORS setting. With that said, the convention is to save them in a file called context_processors.py within your app or project.

Inside Template Loading

Generally, you’ll store templates in files on your filesystem, but you can use custom template loaders to load templates from other sources.

Django has two ways to load templates:

django.template.loader.get_template(template_name): get_template returns the compiled template (a Template object) for the template with the given name. If the template doesn’t exist, a TemplateDoesNotExist exception will be raised.
django.template.loader.select_template(template_name_list): select_template is just like get_template, except it takes a list of template names. Of the list, it returns the first template that exists. If none of the templates exist, a TemplateDoesNotExist exception will be raised.

As covered in Chapter 4, each of these functions by default uses your TEMPLATE_DIRS setting to load templates. Internally, however, these functions actually delegate to a template loader for the heavy lifting.

Some of loaders are disabled by default, but you can activate them by editing the TEMPLATE_LOADERS setting. TEMPLATE_LOADERS should be a tuple of strings, where each string represents a template loader. These template loaders ship with Django:

django.template.loaders.filesystem.load_template_source: This loader loads templates from the filesystem, according to TEMPLATE_DIRS. It is enabled by default.
django.template.loaders.app_directories.load_template_source: This loader loads templates from Django applications on the filesystem. For each application in INSTALLED_APPS, the loader looks for a templates subdirectory. If the directory exists, Django looks for templates there.

This means you can store templates with your individual applications, making it easy to distribute Django applications with default templates. For example, if INSTALLED_APPS contains ('myproject.polls', 'myproject.music'), then get_template('foo.html') will look for templates in this order:
- /path/to/myproject/polls/templates/foo.html
- /path/to/myproject/music/templates/foo.html
Note that the loader performs an optimization when it is first imported: it caches a list of which INSTALLED_APPS packages have a templates subdirectory.

This loader is enabled by default.
django.template.loaders.eggs.load_template_source: This loader is just like app_directories, except it loads templates from Python eggs rather than from the filesystem. This loader is disabled by default; you’ll need to enable it if you’re using eggs to distribute your application.

Django uses the template loaders in order according to the TEMPLATE_LOADERS setting. It uses each loader until a loader finds a match.

Extending the Template System

Now that you understand a bit more about the internals of the template system, let’s look at how to extend the system with custom code.

Most template customization comes in the form of custom template tags and/or filters. Although the Django template language comes with many built-in tags and filters, you’ll probably assemble your own libraries of tags and filters that fit your own needs. Fortunately, it’s quite easy to define your own functionality.

Creating a Template Library

Whether you’re writing custom tags or filters, the first thing to do is to create a template library — a small bit of infrastructure Django can hook into.

Creating a template library is a two-step process:

First, decide which Django application should house the template library. If you’ve created an app via manage.py startapp, you can put it in there, or you can create another app solely for the template library.

Whichever route you take, make sure to add the app to your INSTALLED_APPS setting. We’ll explain this shortly.
Second, create a templatetags directory in the appropriate Django application’s package. It should be on the same level as models.py, views.py, and so forth. For example:
```
books/
    __init__.py
    models.py
    templatetags/
    views.py
```
Create two empty files in the templatetags directory: an __init__.py file (to indicate to Python that this is a package containing Python code) and a file that will contain your custom tag/filter definitions. The name of the latter file is what you’ll use to load the tags later. For example, if your custom tags/filters are in a file called poll_extras.py, you’d write the following in a template:
```
{% load poll_extras %}
```
The {% load %} tag looks at your INSTALLED_APPS setting and only allows the loading of template libraries within installed Django applications. This is a security feature; it allows you to host Python code for many template libraries on a single computer without enabling access to all of them for every Django installation.

If you write a template library that isn’t tied to any particular models/views, it’s valid and quite normal to have a Django application package that contains only a templatetags package. There’s no limit on how many modules you put in the templatetags package. Just keep in mind that a {% load %} statement will load tags/filters for the given Python module name, not the name of the application.

Once you’ve created that Python module, you’ll just have to write a bit of Python code, depending on whether you’re writing filters or tags.

To be a valid tag library, the module must contain a module-level variable named register that is a template.Library instance. This template.Library instance is the data structure in which all the tags and filters are registered. So, near the top of your module, insert the following:

from django import template

register = template.Library()

Note

For a good number of examples, read the source code for Django’s default filters and tags. They’re in django/template/defaultfilters.py and django/template/defaulttags.py, respectively. Some applications in django.contrib also contain template libraries.

Once you’ve created this register variable, you’ll use it to create template filters and tags.

Writing Custom Template Filters

Custom filters are just Python functions that take one or two arguments:

The value of the variable (input)
The value of the argument, which can have a default value or be left out altogether

For example, in the filter {{ var|foo:"bar" }}, the filter foo would be passed the contents of the variable var and the argument "bar".

Filter functions should always return something. They shouldn’t raise exceptions, and they should fail silently. If there’s an error, they should return either the original input or an empty string, whichever makes more sense.

Here’s an example filter definition:

def cut(value, arg):
    "Removes all values of arg from the given string"
    return value.replace(arg, '')

And here’s an example of how that filter would be used:

{{ somevariable|cut:"0" }}

Most filters don’t take arguments. In this case, just leave the argument out of your function:

def lower(value): # Only one argument.
    "Converts a string into all lowercase"
    return value.lower()

When you’ve written your filter definition, you need to register it with your Library instance, to make it available to Django’s template language:

register.filter('cut', cut)
register.filter('lower', lower)

The Library.filter() method takes two arguments:

The name of the filter (a string)
The filter function itself

If you’re using Python 2.4 or above, you can use register.filter() as a decorator instead:

@register.filter(name='cut')
def cut(value, arg):
    return value.replace(arg, '')

@register.filter
def lower(value):
    return value.lower()

If you leave off the name argument, as in the second example, Django will use the function’s name as the filter name.

Here, then, is a complete template library example, supplying the cut filter:

from django import template

register = template.Library()

@register.filter(name='cut')
def cut(value, arg):
    return value.replace(arg, '')

Writing Custom Template Tags

Tags are more complex than filters, because tags can do nearly anything.

Chapter 4 describes how the template system works in a two-step process: compiling and rendering. To define a custom template tag, you need to tell Django how to manage both steps when it gets to your tag.

When Django compiles a template, it splits the raw template text into nodes. Each node is an instance of django.template.Node and has a render() method. Thus, a compiled template is simply a list of Node objects.

When you call render() on a compiled template, the template calls render() on each Node in its node list, with the given context. The results are all concatenated together to form the output of the template. Thus, to define a custom template tag, you specify how the raw template tag is converted into a Node (the compilation function) and what the node’s render() method does.

In the sections that follow, we cover all the steps in writing a custom tag.

Writing the Compilation Function

For each template tag it encounters, the template parser calls a Python function with the tag contents and the parser object itself. This function is responsible for returning a Node instance based on the contents of the tag.

For example, let’s write a template tag, {% current_time %}, that displays the current date/time, formatted according to a parameter given in the tag, in strftime syntax (see http://www.djangoproject.com/r/python/strftime/). It’s a good idea to decide the tag syntax before anything else. In our case, let’s say the tag should be used like this:

<p>The time is {% current_time "%Y-%m-%d %I:%M %p" %}.</p>

Note

Yes, this template tag is redundant—Django’s default {% now %} tag does the same task with simpler syntax. This template tag is presented here just for example purposes.

The parser for this function should grab the parameter and create a Node object:

from django import template

def do_current_time(parser, token):
    try:
        # split_contents() knows not to split quoted strings.
        tag_name, format_string = token.split_contents()
    except ValueError:
        msg = '%r tag requires a single argument' % token.contents[0]
        raise template.TemplateSyntaxError(msg)
    return CurrentTimeNode(format_string[1:-1])

There’s actually a lot going here:

parser is the template parser object. We don’t need it in this example.
token.contents is a string of the raw contents of the tag. In our example, it’s 'current_time "%Y-%m-%d %I:%M %p"'.
The token.split_contents() method separates the arguments on spaces while keeping quoted strings together. Avoid using token.contents.split() (which just uses Python’s standard string-splitting semantics). It’s not as robust, as it naively splits on all spaces, including those within quoted strings.
This function is responsible for raising django.template.TemplateSyntaxError, with helpful messages, for any syntax error.
Don’t hard-code the tag’s name in your error messages, because that couples the tag’s name to your function. token.split_contents()[0] will always be the name of your tag—even when the tag has no arguments.
The function returns a CurrentTimeNode (which we’ll create shortly) containing everything the node needs to know about this tag. In this case, it just passes the argument "%Y-%m-%d %I:%M %p". The leading and trailing quotes from the template tag are removed with format_string[1:-1].
Template tag compilation functions must return a Node subclass; any other return value is an error.

Writing the Template Node

The second step in writing custom tags is to define a Node subclass that has a render() method. Continuing the preceding example, we need to define CurrentTimeNode:

import datetime

class CurrentTimeNode(template.Node):

    def __init__(self, format_string):
        self.format_string = format_string

    def render(self, context):
        now = datetime.datetime.now()
        return now.strftime(self.format_string)

These two functions (__init__ and render) map directly to the two steps in template processing (compilation and rendering). Thus, the initialization function only needs to store the format string for later use, and the render() function does the real work.

Like template filters, these rendering functions should fail silently instead of raising errors. The only time that template tags are allowed to raise errors is at compilation time.

Registering the Tag

Finally, you need to register the tag with your module’s Library instance. Registering custom tags is very similar to registering custom filters (as explained above). Just instantiate a template.Library instance and call its tag() method. For example:

register.tag('current_time', do_current_time)

The tag() method takes two arguments:

The name of the template tag (string). If this is left out, the

name of the compilation function will be used.

The compilation function.

As with filter registration, it is also possible to use register.tag as a decorator in Python 2.4 and above:

@register.tag(name="current_time")
def do_current_time(parser, token):
    # ...

@register.tag
def shout(parser, token):
    # ...

If you leave off the name argument, as in the second example, Django will use the function’s name as the tag name.

Setting a Variable in the Context

The previous section’s example simply returned a value. Often it’s useful to set template variables instead of returning values. That way, template authors can just use the variables that your template tags set.

To set a variable in the context, use dictionary assignment on the context object in the render() method. Here’s an updated version of CurrentTimeNode that sets a template variable, current_time, instead of returning it:

class CurrentTimeNode2(template.Node):

    def __init__(self, format_string):
        self.format_string = format_string

    def render(self, context):
        now = datetime.datetime.now()
        context['current_time'] = now.strftime(self.format_string)
        return ''

Note that render() returns an empty string. render() should always return a string, so if all the template tag does is set a variable, render() should return an empty string.

Here’s how you’d use this new version of the tag:

{% current_time2 "%Y-%M-%d %I:%M %p" %}
<p>The time is {{ current_time }}.</p>

But there’s a problem with CurrentTimeNode2: the variable name current_time is hard-coded. This means you’ll need to make sure your template doesn’t use {{ current_time }} anywhere else, because {% current_time2 %} will blindly overwrite that variable’s value.

A cleaner solution is to make the template tag specify the name of the variable to be set, like so:

{% get_current_time "%Y-%M-%d %I:%M %p" as my_current_time %}
<p>The current time is {{ my_current_time }}.</p>

To do so, you’ll need to refactor both the compilation function and the Node class, as follows:

import re

class CurrentTimeNode3(template.Node):

    def __init__(self, format_string, var_name):
        self.format_string = format_string
        self.var_name = var_name

    def render(self, context):
        now = datetime.datetime.now()
        context[self.var_name] = now.strftime(self.format_string)
        return ''

def do_current_time(parser, token):
    # This version uses a regular expression to parse tag contents.
    try:
        # Splitting by None == splitting by spaces.
        tag_name, arg = token.contents.split(None, 1)
    except ValueError:
        msg = '%r tag requires arguments' % token.contents[0]
        raise template.TemplateSyntaxError(msg)

    m = re.search(r'(.*?) as (\w+)', arg)
    if m:
        fmt, var_name = m.groups()
    else:
        msg = '%r tag had invalid arguments' % tag_name
        raise template.TemplateSyntaxError(msg)

    if not (fmt[0] == fmt[-1] and fmt[0] in ('"', "'")):
        msg = "%r tag's argument should be in quotes" % tag_name
        raise template.TemplateSyntaxError(msg)

    return CurrentTimeNode3(fmt[1:-1], var_name)

Now do_current_time() passes the format string and the variable name to CurrentTimeNode3.

Parsing Until Another Block Tag

Template tags can work as blocks containing other tags (think {% if %}, {% for %}, etc.). To create a template tag like this, use parser.parse() in your compilation function.

Here’s how the standard {% comment %} tag is implemented:

def do_comment(parser, token):
    nodelist = parser.parse(('endcomment',))
    parser.delete_first_token()
    return CommentNode()

class CommentNode(template.Node):
    def render(self, context):
        return ''

parser.parse() takes a tuple of names of block tags to parse until. It returns an instance of django.template.NodeList, which is a list of all Node objects that the parser encountered before it encountered any of the tags named in the tuple.

So in the preceding example, nodelist is a list of all nodes between {% comment %} and {% endcomment %}, not counting {% comment %} and {% endcomment %} themselves.

After parser.parse() is called, the parser hasn’t yet “consumed” the {% endcomment %} tag, so the code needs to explicitly call parser.delete_first_token() to prevent that tag from being processed twice.

Then CommentNode.render() simply returns an empty string. Anything between {% comment %} and {% endcomment %} is ignored.

Parsing Until Another Block Tag and Saving Contents

In the previous example, do_comment() discarded everything between {% comment %} and {% endcomment %}. It’s also possible to do something with the code between block tags instead.

For example, here’s a custom template tag, {% upper %}, that capitalizes everything between itself and {% endupper %}:

{% upper %}
    This will appear in uppercase, {{ your_name }}.
{% endupper %}

As in the previous example, we’ll use parser.parse(). This time, we pass the resulting nodelist to Node:

@register.tag
def do_upper(parser, token):
    nodelist = parser.parse(('endupper',))
    parser.delete_first_token()
    return UpperNode(nodelist)

class UpperNode(template.Node):

    def __init__(self, nodelist):
        self.nodelist = nodelist

    def render(self, context):
        output = self.nodelist.render(context)
        return output.upper()

The only new concept here is self.nodelist.render(context) in UpperNode.render(). This simply calls render() on each Node in the node list.

For more examples of complex rendering, see the source code for {% if %}, {% for %}, {% ifequal %}, and {% ifchanged %}. They live in django/template/defaulttags.py.

Shortcut for Simple Tags

Many template tags take a single argument—a string or a template variable reference—and return a string after doing some processing based solely on the input argument and some external information. For example, the current_time tag we wrote earlier is of this variety. We give it a format string, and it returns the time as a string.

To ease the creation of these types of tags, Django provides a helper function, simple_tag. This function, which is a method of django.template.Library, takes a function that accepts one argument, wraps it in a render function and the other necessary bits mentioned previously, and registers it with the template system.

Our earlier current_time function could thus be written like this:

def current_time(format_string):
    return datetime.datetime.now().strftime(format_string)

register.simple_tag(current_time)

In Python 2.4, the decorator syntax also works:

@register.simple_tag
def current_time(token):
    ...

A couple of things to notice about the simple_tag helper function are as follows:

Only the (single) argument is passed into our function.
Checking for the required number of arguments has already been done by the time our function is called, so we don’t need to do that.
The quotes around the argument (if any) have already been stripped away, so we receive a plain string.

Inclusion Tags

Another common template tag is the type that displays some data by rendering another template. For example, Django’s admin interface uses custom template tags to display the buttons along the bottom of the “add/change” form pages. Those buttons always look the same, but the link targets change depending on the object being edited. They’re a perfect case for using a small template that is filled with details from the current object.

These sorts of tags are called inclusion tags. Writing inclusion tags is probably best demonstrated by example. Let’s write a tag that produces a list of choices for a simple multiple-choice Poll object. We’ll use the tag like this:

{% show_results poll %}

The result will be something like this:

<ul>
  <li>First choice</li>
  <li>Second choice</li>
  <li>Third choice</li>
</ul>

First, we define the function that takes the argument and produces a dictionary of data for the result. Notice that we need to return only a dictionary, not anything more complex. This will be used as the context for the template fragment:

def show_books_for_author(author):
    books = author.book_set.all()
    return {'books': books}

Next, we create the template used to render the tag’s output. Following our example, the template is very simple:

<ul>
{% for book in books %}
    <li> {{ book }} </li>
{% endfor %}
</ul>

Finally, we create and register the inclusion tag by calling the inclusion_tag() method on a Library object.

Following our example, if the preceding template is in a file called polls/result_snippet.html, we register the tag like this:

register.inclusion_tag('books/books_for_author.html')(show_books_for_author)

As always, Python 2.4 decorator syntax works as well, so we could have instead written this:

@register.inclusion_tag('books/books_for_author.html')
def show_books_for_author(show_books_for_author):
    ...

Sometimes, your inclusion tags need access to values from the parent template’s context. To solve this, Django provides a takes_context option for inclusion tags. If you specify takes_context in creating a template tag, the tag will have no required arguments, and the underlying Python function will have one argument: the template context as of when the tag was called.

For example, say you’re writing an inclusion tag that will always be used in a context that contains home_link and home_title variables that point back to the main page. Here’s what the Python function would look like:

@register.inclusion_tag('link.html', takes_context=True)
def jump_link(context):
    return {
        'link': context['home_link'],
        'title': context['home_title'],
    }

Note

The first parameter to the function must be called context.

The template link.html might contain the following:

Jump directly to <a href="{{ link }}">{{ title }}</a>.

Then, anytime you want to use that custom tag, load its library and call it without any arguments, like so:

{% jump_link %}

Chapter 11: Generating Non-HTML Content

Usually when we talk about developing Web sites, we’re talking about producing HTML. Of course, there’s a lot more to the Web than HTML; we use the Web to distribute data in all sorts of formats: RSS, PDFs, images, and so forth.

So far we’ve focused on the common case of HTML production, but in this chapter we’ll take a detour and look at using Django to produce other types of content.

Django has convenient built-in tools that you can use to produce some common non-HTML content:

RSS/Atom syndication feeds
Sitemaps (an XML format originally developed by Google that gives hints to search engines)

We’ll examine each of those tools a little later on, but first we’ll cover the basic principles.

The basics: views and MIME-types

Remember this from Chapter 3?

A view function, or view for short, is simply a Python function that takes a Web request and returns a Web response. This response can be the HTML contents of a Web page, or a redirect, or a 404 error, or an XML document, or an image…or anything, really.

More formally, a Django view function must

Accept an HttpRequest instance as its first argument
Return an HttpResponse instance

The key to returning non-HTML content from a view lies in the HttpResponse class, specifically the mimetype constructor argument. By tweaking the MIME type, we can indicate to the browser that we’ve returned a response of a different format.

For example, let’s look at a view that returns a PNG image. To keep things simple, we’ll just read the file off the disk:

from django.http import HttpResponse

def my_image(request):
    image_data = open("/path/to/my/image.png", "rb").read()
    return HttpResponse(image_data, mimetype="image/png")

That’s it! If you replace the image path in the open() call with a path to a real image, you can use this very simple view to serve an image, and the browser will display it correctly.

The other important thing to keep in mind is that HttpResponse objects implement Python’s standard file API. This means that you can use an HttpResponse instance in any place Python (or a third-party library) expects a file.

For an example of how that works, let’s take a look at producing CSV with Django.

Producing CSV

CSV is a simple data format usually used by spreadsheet software. It’s basically a series of table rows, with each cell in the row separated by a comma (CSV stands for comma-separated values). For example, here’s some data on “unruly” airline passengers in CSV format:

Year,Unruly Airline Passengers
1995,146
1996,184
1997,235
1998,200
1999,226
2000,251
2001,299
2002,273
2003,281
2004,304
2005,203

Note

The preceding listing contains real numbers; they come courtesy of the US Federal Aviation Administration. See http://www.faa.gov/data_statistics/passengers_cargo/unruly_passengers/.

Though CSV looks simple, it’s not a format that’s ever been formally defined. Different pieces of software produce and consume different variants of CSV, making it a bit tricky to use. Luckily, Python comes with a standard CSV library, csv, that is pretty much bulletproof.

Because the csv module operates on file-like objects, it’s a snap to use an HttpResponse instead:

import csv
from django.http import HttpResponse

# Number of unruly passengers each year 1995 - 2005. In a real application
# this would likely come from a database or some other back-end data store.
UNRULY_PASSENGERS = [146,184,235,200,226,251,299,273,281,304,203]

def unruly_passengers_csv(request):
    # Create the HttpResponse object with the appropriate CSV header.
    response = HttpResponse(mimetype='text/csv')
    response['Content-Disposition'] = 'attachment; filename=unruly.csv'

    # Create the CSV writer using the HttpResponse as the "file"
    writer = csv.writer(response)
    writer.writerow(['Year', 'Unruly Airline Passengers'])
    for (year, num) in zip(range(1995, 2006), UNRULY_PASSENGERS):
        writer.writerow([year, num])

    return response

The code and comments should be pretty clear, but a few things deserve special mention:

The response is given the text/csv MIME type (instead of the default text/html). This tells browsers that the document is a CSV file.
The response gets an additional Content-Disposition header, which contains the name of the CSV file. This header (well, the “attachment” part) will instruct the browser to prompt for a location to save the file (instead of just displaying it). This file name is arbitrary; call it whatever you want. It will be used by browsers in the Save As dialog.
Hooking into the CSV-generation API is easy: just pass response as the first argument to csv.writer. The csv.writer function expects a filelike object, and HttpResponse objects fit the bill.
For each row in your CSV file, call writer.writerow, passing it an iterable object such as a list or tuple.
The CSV module takes care of quoting for you, so you don’t have to worry about escaping strings with quotes or commas in them. Just pass information to writerow(), and it will do the right thing.

This is the general pattern you’ll use any time you need to return non-HTML content: create an HttpResponse response object (with a special MIME type), pass it to something expecting a file, and then return the response.

Let’s look at a few more examples.

Generating PDFs

Portable Document Format (PDF) is a format developed by Adobe that’s used to represent printable documents, complete with pixel-perfect formatting, embedded fonts, and 2D vector graphics. You can think of a PDF document as the digital equivalent of a printed document; indeed, PDFs are usually used when you need to give a document to someone else to print.

You can easily generate PDFs with Python and Django thanks to the excellent open source ReportLab library (http://www.reportlab.org/rl_toolkit.html). The advantage of generating PDF files dynamically is that you can create customized PDFs for different purposes — say, for different users or different pieces of content.

For example, we used Django and ReportLab at KUSports.com to generate customized, printer-ready NCAA tournament brackets.

Installing ReportLab

Before you do any PDF generation, however, you’ll need to install ReportLab. It’s usually pretty simple: just download and install the library from http://www.reportlab.org/downloads.html.

The user guide (naturally available only as a PDF file) at http://www.reportlab.org/rsrc/userguide.pdf has additional installation instructions.

Note

If you’re using a modern Linux distribution, you might want to check your package management utility before installing ReportLab. Most package repositories have added ReportLab.

For example, if you’re using the (excellent) Ubuntu distribution, a simple apt-get install python-reportlab will do the trick nicely.

Test your installation by importing it in the Python interactive interpreter:

>>> import reportlab

If that command doesn’t raise any errors, the installation worked.

Writing Your View

Like CSV, generating PDFs dynamically with Django is easy because the ReportLab API acts on filelike objects.

Here’s a “Hello World” example:

from reportlab.pdfgen import canvas
from django.http import HttpResponse

def hello_pdf(request):
    # Create the HttpResponse object with the appropriate PDF headers.
    response = HttpResponse(mimetype='application/pdf')
    response['Content-Disposition'] = 'attachment; filename=hello.pdf'

    # Create the PDF object, using the response object as its "file."
    p = canvas.Canvas(response)

    # Draw things on the PDF. Here's where the PDF generation happens.
    # See the ReportLab documentation for the full list of functionality.
    p.drawString(100, 100, "Hello world.")

    # Close the PDF object cleanly, and we're done.
    p.showPage()
    p.save()
    return response

A few notes are in order:

Here we use the application/pdf MIME type. This tells browsers that the document is a PDF file, rather than an HTML file. If you leave off this information, browsers will probably interpret the response as HTML, which will result in scary gobbledygook in the browser window.
Hooking into the ReportLab API is easy: just pass response as the first argument to canvas.Canvas. The Canvas class expects a filelike object, and HttpResponse objects fit the bill.
All subsequent PDF-generation methods are called on the PDF object (in this case, p), not on response.
Finally, it’s important to call showPage() and save() on the PDF file (or else you’ll end up with a corrupted PDF file).

Complex PDFs

If you’re creating a complex PDF document (or any large data blob), consider using the cStringIO library as a temporary holding place for your PDF file. The cStringIO library provides a file-like object interface that is written in C for maximum efficiency.

Here’s the previous “Hello World” example rewritten to use cStringIO:

from cStringIO import StringIO
from reportlab.pdfgen import canvas
from django.http import HttpResponse

def hello_pdf(request):
    # Create the HttpResponse object with the appropriate PDF headers.
    response = HttpResponse(mimetype='application/pdf')
    response['Content-Disposition'] = 'attachment; filename=hello.pdf'

    temp = StringIO()

    # Create the PDF object, using the StringIO object as its "file."
    p = canvas.Canvas(temp)

    # Draw things on the PDF. Here's where the PDF generation happens.
    # See the ReportLab documentation for the full list of functionality.
    p.drawString(100, 100, "Hello world.")

    # Close the PDF object cleanly.
    p.showPage()
    p.save()

    # Get the value of the StringIO buffer and write it to the response.
    response.write(temp.getvalue())
    return response

Other Possibilities

There’s a whole host of other types of content you can generate in Python. Here are a few more ideas and some pointers to libraries you could use to implement them:

ZIP files: Python’s standard library ships with the zipfile module, which can both read and write compressed ZIP files. You could use it to provide on-demand archives of a bunch of files, or perhaps compress large documents when requested. You could similarly produce TAR files using the standard library tarfile module.
Dynamic images: The Python Imaging Library (PIL; http://www.pythonware.com/products/pil/) is a fantastic toolkit for producing images (PNG, JPEG, GIF, and a whole lot more). You could use it to automatically scale down images into thumbnails, composite multiple images into a single frame, or even do Web-based image processing.
Plots and charts: There are a number of incredibly powerful Python plotting and charting libraries you could use to produce on-demand maps, charts, plots, and graphs. We can’t possibly list them all, so here are a couple of the highlights:
- matplotlib (http://matplotlib.sourceforge.net/) can be used to produce the type of high-quality plots usually generated with MatLab or Mathematica.
- pygraphviz (https://networkx.lanl.gov/wiki/pygraphviz), an interface to the Graphviz graph layout toolkit (http://graphviz.org/), can be used for generating structured diagrams of graphs and networks.

In general, any Python library capable of writing to a file can be hooked into Django. The possibilities really are endless.

Now that we’ve looked at the basics of generating non-HTML content, let’s step up a level of abstraction. Django ships with some pretty nifty built-in tools for generating some common types of non-HTML content.

The Syndication Feed Framework

Django comes with a high-level syndication-feed-generating framework that makes creating RSS and Atom feeds easy.

What’s RSS? What’s Atom?

RSS and Atom are both XML-based formats you can use to provide automatically updating “feeds” of your site’s content. Read more about RSS at http://www.whatisrss.com/, and get information on Atom at http://www.atomenabled.org/.

To create any syndication feed, all you have to do is write a short Python class. You can create as many feeds as you want.

The high-level feed-generating framework is a view that’s hooked to /feeds/ by convention. Django uses the remainder of the URL (everything after /feeds/) to determine which feed to return.

To create a feed, you’ll write a Feed class and point to it in your URLconf (see Chapters 3 and 8 for more about URLconfs).

Initialization

To activate syndication feeds on your Django site, add this URLconf:

(r'^feeds/(?P<url>.*)/$',
 'django.contrib.syndication.views.feed',
 {'feed_dict': feeds}
),

This line tells Django to use the RSS framework to handle all URLs starting with "feeds/". (You can change that "feeds/" prefix to fit your own needs.)

This URLconf line has an extra argument: {'feed_dict': feeds}. Use this extra argument to pass the syndication framework the feeds that should be published under that URL.

Specifically, feed_dict should be a dictionary that maps a feed’s slug (short URL label) to its Feed class. You can define the feed_dict in the URLconf itself. Here’s a full example URLconf:

from django.conf.urls.defaults import *
from myproject.feeds import LatestEntries, LatestEntriesByCategory

feeds = {
    'latest': LatestEntries,
    'categories': LatestEntriesByCategory,
}

urlpatterns = patterns('',
    # ...
    (r'^feeds/(?P<url>.*)/$', 'django.contrib.syndication.views.feed',
        {'feed_dict': feeds}),
    # ...
)

The preceding example registers two feeds:

The feed represented by LatestEntries will live at feeds/latest/.
The feed represented by LatestEntriesByCategory will live at feeds/categories/.

Once that’s set up, you’ll need to define the Feed classes themselves.

A Feed class is a simple Python class that represents a syndication feed. A feed can be simple (e.g., a “site news” feed, or a basic feed displaying the latest entries of a blog) or more complex (e.g., a feed displaying all the blog entries in a particular category, where the category is variable).

Feed classes must subclass django.contrib.syndication.feeds.Feed. They can live anywhere in your code tree.

A Simple Feed

This simple example, taken from chicagocrime.org, describes a feed of the latest five news items:

from django.contrib.syndication.feeds import Feed
from chicagocrime.models import NewsItem

class LatestEntries(Feed):
    title = "Chicagocrime.org site news"
    link = "/sitenews/"
    description = "Updates on changes and additions to chicagocrime.org."

    def items(self):
        return NewsItem.objects.order_by('-pub_date')[:5]

The important things to notice here are as follows:

The class subclasses django.contrib.syndication.feeds.Feed.
title, link, and description correspond to the standard RSS

Table of Contents

Chapter 1: Introduction to Django

What Is a Web Framework?

The MVC Design Pattern

Django’s History

How to Read This Book

Required Programming Knowledge

Required Python Knowledge

New Django Features

Getting Help

What’s Next

Chapter 2: Getting Started

Installing Python

Installing Django

Installing an Official Release

Installing Django from Subversion

Setting Up a Database

Using Django with PostgreSQL

Using Django with SQLite 3

Using Django with MySQL

Using Django Without a Database

Starting a Project

The Development Server

What’s Next?

Chapter 3: The Basics of Dynamic Web Pages

Your First View: Dynamic Content

Mapping URLs to Views

How Django Processes a Request

How Django Processes a Request: Complete Details

URLconfs and Loose Coupling

404 Errors

Your Second View: Dynamic URLs

A Word About Pretty URLs

Wildcard URLpatterns

Django’s Pretty Error Pages

What’s next?

Chapter 4: The Django Template System

Template System Basics

Using the Template System

Creating Template Objects

Rendering a Template

Multiple Contexts, Same Template

Context Variable Lookup

Method Call Behavior

How Invalid Variables Are Handled

Playing with Context Objects

Basic Template Tags and Filters

Tags

if/else

for

ifequal/ifnotequal

Comments

Filters

Philosophies and Limitations

Using Templates in Views

Template Loading

render_to_response()

The locals() Trick

Subdirectories in get_template()

The include Template Tag

Template Inheritance

What’s next?

Chapter 5: Interacting with a Database: Models

The “Dumb” Way to Do Database Queries in Views

The MTV Development Pattern

Configuring the Database

Your First App

Defining Models in Python

Your First Model

Installing the Model

Basic Data Access

Adding Model String Representations

Inserting and Updating Data

Selecting Objects

Filtering Data

Retrieving Single Objects

Ordering Data

Chaining Lookups

Slicing Data

Deleting Objects

The `include` Template Tag