Have you ever created a Web application? If so, it’s very likely that you have at one time or another faced “the security problem”; whether to create and maintain a homegrown security sub-system, or to learn to use framework-specific security mechanisms (which may not be as flexible as you wish).

Securing Web applications shouldn’t be a problem. This article explores a highly extensible alternative which you can learn once and use in arbitrary applications, regardless of the Web framework used (if any!).

Application security is a broad field within software development, covering topics ranging from low-level network transmission security and encryption, up to application-level data security and input validation. In this article, we will focus on two of the most basic elements of application security: authentication and authorization.

Authentication vs. Authorization

Even experienced software developers often confuse these two related but not equivalent terms, so we’ll start by explaining what they are.

Authentication, often shortened as “authn”, is what you do when you check the credentials provided by the user to verify that he’s really who he claims to be. The most widely-used credentials are a set made up of a user identifier (like user name, or email address) and a password.

Authorization, often shortened as “authz”, is what you do when you check whether the user has permission to make the request or perform the transaction he’s currently making. Most of the time this depends on who the subject is, but it may also depend on what is requested, when it is requested, and/or how it is requested.

A common shortening for “authentication and authorization” is “auth”.

And there’s one more related term that comes into play: Identification, which is to check whether he was successfully authenticated previously to avoid challenging him again unnecessarily.

Authentication, identification and authorization in Python Web applications

Web applications powered by Python may take advantage of repoze.who to handle authentication and identification, and repoze.what to handle authorization. Both are security frameworks that can be used on top of your application, or integrated with your application’s Web framework if you are using one.

You can use them as long as your application is WSGI-compliant. WSGI is a Python standard which defines an interface between HTTP servers and Python applications, similar to CGI, which eases the writing of cross Web server libraries (which can be framework-independent) and applications.

The WSGI standard mandates the availability of a Python dictionary, referred to as the “WSGI environment”. This dictionary stores CGI environment variables, as well as others specific to the WSGI standard; other components used by your application may also use the WSGI environment to store data.

Any framework or raw application that exposes the WSGI environment dictionary can use both repoze.who and repoze.what. The popular WSGI-compliant frameworks CherryPy, Django, Pylons, TurboGears and Werkzeug expose this variable.

How repoze.who and repoze.what work

Figure 1 illustrates how a typical request is processed in WSGI and when repoze.who comes into play.

Figure 1

repoze.who is a WSGI middleware. A WSGI middleware is a layer that wraps your application, processing each request before your application does. WSGI middleware can also post-process your application’s response. WSGI applications can have zero or more of these middleware layers.

As shown in Figure 2, when a request is made and before it reaches your application, repoze.who will try to check if the user is already logged in or if he’s currently trying to log in. If the user is trying to log in and has supplied the right credentials, authentication will succeed and the user will be “remembered” in future requests by default. The request is then passed to the next WSGI middleware, if any, and eventually on to your WSGI application.

Figure 2

The following fully customizable components can be used at this stage:

• The request classifier: It matters if the agent is a browser, a Subversion client or a library which access a Web Service; you wouldn’t want to display a login form if the agent is a Subversion client, for example. Thus, this component classifies the current request so that only the appropriate plugins (out of your pre-selected ones) are used.
• Identifier plugins: These are the components that “identify” the user. That is, they are able to tell if the user is already logged from in a previous request, or if he is trying to log in in the current request. When authentication succeeds, these plugins are in charge of “remembering” the user for future requests (e.g., by defining a session cookie). Likewise, when a challenge is required, they “forget” the user (discarding a session cookie).
• Authenticator plugins: When the user is trying to log in in the current request, these components verify the user-supplied credentials (such as username and password) against a database, LDAP server, .htaccess file, etc.
• Metadata provider plugins: If the user was authenticated, metadata providers can load data about the current user (email address, full name, etc.) so that such data is ready to be used by your application.

After your application issues a response and before it reaches the HTTP server, as shown in Figure 3, repoze.who will check if a challenge is necessary; that is, ask the user in some way to identify himself. If so and the user was previously authenticated, the previous authentication will be forgotten, and the related session cookie will be discarded. After the challenge is complete, your application’s response will be replaced with a new response which will allow the user to log in (for example, sending a “WWW-Authenticate” header or displaying a login form).

repoze.who’s response-handling functionality is also driven by customizable components:

• The challenge decider: This is the component which will determine whether a challenge is required. The default challenge decider will order a challenge if and only if the response’s HTTP status code is 401.
• Challenger plugins: When a challenge is required, those challenger plugins which support the current request type will be run until the first of them returns a valid response.

repoze.who may seem hard to use at first sight, with all its components and terminology; so much flexibility comes at the cost of being a little hard to understand. But one of the goals of this article is to help people who had never heard of repoze.who to deal with basic and advanced authentication settings.

And how does repoze.what work?

repoze.what is mostly used inside of your application, where you define the access rules that must be met for a given routine to be performed.

Access rules in repoze.what are made of atomic units called “predicate checkers”, re-usable objects that check whether a given condition is met. Predicate checkers can be single or compound to form complex access rules. For example, a single condition (or predicate) might be “The user is not anonymous”, while a compound predicate could be “The user is not anonymous and their IP address is X.X.X.X”. You can write your own predicate checkers, usually in just a few lines of code.

repoze.what has built-in support for the widely-used authorization pattern whereby you assign groups to your users and then grant permissions to such groups; it ships with a comprehensive set of checkers for the relevant predicates (e.g., “The current user belongs to the ‘directors’ group”, “The current user is allowed to edit user accounts”). If you use this authorization pattern, the groups and permissions for the authenticated user will be loaded by a repoze.who metadata provider; in repoze.what 1.1, they’ll be loaded on demand by repoze.what itself, so you wouldn’t need repoze.who. Nevertheless, it is entirely optional and you can use any authorization pattern that best suits your needs.

Predicate checkers allow you to control access based not only on who makes the request, but also on how the request is made and what exactly is requested. For example, if you have a blog application, you might use the simple predicate “The user is allowed to remove posts”, focusing on who the user is to control access to the post edition routine. Or you can have the more specific compound predicate “The user is allowed to remove posts, as long as the user is the post’s author”. This predicate focuses on both who makes the request and what is requested. You can have an even more complex access rule for the article edition routine, such as “The user is allowed to remove posts, as long as the user is the post’s author — except post #1 which nobody can remove”.

repoze.what 1.0.X supports only blacklist-based authorization, that is, authorization is granted unless explicitly denied. Whitelist-based authorization, in which authorization is denied unless explicitly granted, will be supported as of version 1.1 (under development as of this writing).

Creating a Web application protected with repoze.who and repoze.what

It’s time to go practical! We’re going to create a WSGI application powered by repoze.who and repoze.what.

I’ll use the TurboGears 2 Web Application framework to make this simple application, but after reading the article you should be able to put what you just learned into practice in other frameworks, or standalone applications.

(For the sake of demonstration, we will skip typical overhead features such as input validation, so we can focus the examples on the repoze.who and repose.what’s integration of authentication and authorization.)

Let’s get started!

We’re going to develop this application using an isolated Python environment using a rather famous utility called “virtualenv”. This is very handy because everything you install or remove won’t affect your system-wide Python environment. To install it, run:

easy_install virtualenv

You may need administration rights to install it, depending on where you install it.

Next, create an environment for our application and activate it; on Unix systems, the commands for this are:

virtualenv --no-site-packages appenv
source appenv/bin/activate

On Windows systems, enter the following in a directory whose path doesn’t contain spaces:

C:\Python25\python.exe "C:\Path-to-VE\virtualenv.py" appenv
C:\Path-to-newly-created-environment\Scripts\activate.bat

When you’re done and want to deactivate it, you should run the command “deactivate” on Unix systems. For Windows, use “C:\Path-to-newly-created-environment\Scripts\deactivate.bat”.

I’ve called my virtual environment “appenv”, but you can use any name you like.

For help installing virtualenv, you can check its documentation at http://pypi.python.org/pypi/virtualenv.

Now to install TurboGears 2:

easy_install -i http://www.turbogears.org/2.0/downloads/current/index tg.devtools

Generating an application

TurboGears allows you to start coding using a minimal application, so that you don’t have to start from scratch (unless you really want to). We’ll use just that minimal application so that we can get off to a quick start, which you can optionally download.

The Web application we’re going to create is a classifieds service, like craiglist.org or gumtree.com; for lack of imagination, I’ll call it “classifieds”. To start coding it from a minimal application, we’ll ask TurboGears to generate it with the following command:

paster quickstart classifieds

Then you’ll be asked a couple of questions. The first will ask you for the package name to be generated in this project; hit ENTER to accept “classifieds” as the default package name. The second question will ask if you need authentication and authorization features for this project, hit ENTER to accept the default answer of “yes”. By answering “yes” to the second question, repoze.who and repoze.what will be used in the application by default. Now switch to the application’s directory, install it in development mode and set it up:

cd classifieds
python setup.py develop
paster setup-app development.ini


We’re not going to start coding yet — I’d like you to try the generated application, so that you can see repoze.who and repoze.what in action. So, start the application (the “reload” switch will restart the application whenever one of its files is modified):

paster serve --reload development.ini

and then open the following URL in your browser: http://localhost:8080/.

You should keep this simple procedure in mind because we’ll use it very often. Then, when you need to stop the development server, you have to hit Ctrl+C.

Now try, at least, the following (in order):

1. Log in: Visit http://localhost:8080/login or click on the “Login” link in the upper-right side of any Web page of our application. Enter “manager” and “managepass” in the login and password fields. You’ll get redirected to the main page and a welcome message will be displayed.
2. Visit a page you shouldn’t see: Like http://localhost:8080/editor_user_only; you should get a 403 page and a message which reads “Only for the editor”.
3. Log out: Visit http://localhost:8080/logout_handler or click on the “Logout” link in the upper-right side of any Web page of our application. You’ll get redirected to the main page and a goodbye message will be displayed.
4. Visit a private page as anonymous: If you’re currently logged in, log out first. Then visit http://localhost:8080/manage_permission_only; you should be redirected to the login form, where also the message “Only for managers” is displayed. Log in with the previous credentials (“manager”/”managepass”) and you’ll be redirected to the page you requested initially.

What you’ve seen is repoze.who, repoze.what and some of their official plugins in action; those notification messages are TurboGears-specific, though, but it shouldn’t be hard to port them to other frameworks or raw applications.

Before moving forward, I’ll describe some of the files and directories that the “paster quickstart” command above generated:

• devdata.db: The sqlite database file used for development.
• classifieds/: Your application’s package itself.
• classifieds/config/middleware.py: The file where the extra WSGI middleware for your application is added.
• classifieds/controllers/root.py: Your application’s root controller. Sub-controllers should be attached to the one defined in this file; below I’ll explain how.
• classifieds/model/: The application’s model definitions powered by SQLAlchemy.
• classifieds/model/auth.py: The model definitions specific to authentication, identification and authorization.
• classifieds/templates/: The application’s XHTML templates powered by Genshi.
• classifieds/websetup.py: Contains the function which will get run when the application is set up (used by the paster setup-app command). By default it just adds rows to the database.

On the other hand, this is how authentication, identification and authorization is configured right now:

• We have a table for our application’s users (tg_user), which contains the self-explanatory fields user_name and password, among others. repoze.who is configured to authenticate by using these two fields.
• repoze.what is configured to use its groups/permission-based pattern. The groups and permissions are stored in the database, in the tg_group and tg_permission tables.
• One user can belong to zero or more groups; one group can be granted zero or more permissions.
• Right now we have two users, “manager” and “editor” (with passwords “managepass” and “editpass”). “manager” belongs to the only group defined so far, “managers”. The group “managers” is granted the only permission defined so far, “manage”.

Let’s start coding

“At last!”, I heard you say.

While implementing repoze.who and repoze.what in an application which doesn’t come with them enabled out-of-the-box, the first thing you have to do is add their middleware to your application.

However, we’ll skip that part and keep the default settings for now, so that we can go to fun part right away: Learning how to protect areas in your Web application. The setup will be addressed later on, where you’ll learn how to configure repoze.who and repoze.what the TurboGears-independent way.

This is what we’re going to implement in our classifieds application:

• A simple user registration system.
• A classified visualization mechanism, for any user (anonymous or authenticated).
• A classified addition mechanism, for registered users.
• A classified edition mechanism, for registered users to edit their own classifieds.

So it’s finally time to fire up your Python editor, a terminal and a window of your browser!

First of all, let’s add groups and permissions for authorization in our application, instead of sticking to the ones created by default. We’re going to add one more group called “posters” and the “add-classifieds” permission to be granted to posters. (Before continuing, you should stop the server running in the terminal — With Ctrl+C).

To add the posters group, add this code to the setup_app() function, defined in classifieds/websetup.py:

posters = model.Group(group_name=u'posters', display_name=u'Classified posters')
model.DBSession.add(posters)

The addclassifieds permission is created with this code:

addclassifieds = model.Permission(permission_name=u'add-classifieds', description=u'Allowed to add classifieds')
addclassifieds.groups.append(posters)
model.DBSession.add(addclassifieds)

Add these sections right before the following lines:

model.DBSession.flush()
transaction.commit()

Now, since we have a classifieds application, we have to define the SQLAlchemy model for the classifieds. To keep things simple, we’ll define just four columns:

• classified_id: The classified’s identifier.
• poster_id: The poster’s user id.
• classified_title: The classified title.
• classified_contents: The classified contents.

Listing 1 implements this model definition. You have to create classifieds/model/posts.py and store that definition in there. Then you have to import that model at the end of classifieds/model/__init__.py:

from classifieds.model.posts import Classified

To apply the changes, let’s remove the development database, re-create it with our new model and rows, and finally start the server again:

rm devdata.db
paster setup-app development.ini
paster serve --reload development.ini


That’s it, now we have the model definition for the classifieds. Let the fun part begin!

Creating a user registration system

We’re going to create a simple registration system made up of two controller actions: One to display the registration form and another to process the submitted form.

We’re going to implement them in the root controller for our application, the class RootController found in the source file classifieds/controllers/root.py.

To define the action for the registration form, first import the repoze.what predicate checker that verifies that the current user is anonymous:

from repoze.what.predicates import is_anonymous

Next, add the following method in RootController:

@expose('classifieds.templates.register')
@require(is_anonymous(msg='Only one account per user is allowed'))
def register(self):
"""Display the user registration form"""
return {'page': 'user registration'}

What the two decorators above do is specify that the template for the “register” action is classifieds/templates/register.html and that access is granted to users who don’t have an account, respectively.

@require is a TurboGears-specific decorator that evaluates a repoze.what predicate before calling the action in question. When such a predicate is not met, the action is not called and a 401 response is returned (403 if the user is already logged in). Then repoze.who’s default challenge decider will find the 401 response and will replace it with the challenge.

Internally, the predicate is evaluated by calling its check_authorization() method (which raises the repoze.what.predicates.NotAuthorizedError exception if the predicate isn’t met, whose message is the user-friendly explanation) or is_met() (which returns a boolean). Listing 2 illustrates a fictitious implementation, using the decorator package; you’d find it useful if you want to use repoze.what in another framework or raw application (Pylons users may want to check repoze.what-pylons).

Going back to the creation of the action, you’ll have to create the template that will display the form: Create classifieds/templates/register.html with the contents of Listing 3.

Now it’s time to create the action which will process the contents of the submitted form by adding the user to the database, including them in the “posters” group and finally redirecting them to the login form so that they can use the newly created account. For that, you have to define the add_user method shown in Listing 4 (in RootController).

That’s it! Our registration system is done. Now you can try it by visiting http://localhost:8080/register.

The classifieds visualization mechanism

We’re going to write the part of the interface which will allow us to see the classifieds hosted by our service. This will be accomplished by means of two controller actions: One to see all the classifieds available, in the main page of the application, and another to see individual classifieds.

For the first, we have to re-write the “index” action of the root controller, to make it look like this:

@expose('classifieds.templates.index')
def index(self):
query = DBSession.query(model.Classified)
all_classifieds = query.all()
return dict(page='index', classifieds=all_classifieds)

Then its template, classifieds/templates/index.html, should be replaced with the contents of Listing 5, which lists the available classifieds with a link to their individual pages.

The second action, the one to show the classifieds individually, will be implemented as “view” and will be defined in the root controller too:

@expose('classifieds.templates.view')
def view(self, classified):
query = DBSession.query(model.Classified)
classified_obj = query.get(classified)
# Is the user allowed to edit the classified?
checker = user_is_poster()
can_edit = checker.is_met(request.environ)
return {'page': 'Classified page', 'classified': classified_obj, 'classified_id': classified, 'can_edit': can_edit}

Note that in the “view” action we do something new: Handle a predicate checker directly. Sometimes it is necessary to evaluate them directly and get a boolean result to know whether it’s met or not thanks to the is_met() method of the predicate checker. For example, right now we use it to display a link to edit that predicate, if and only if the current user is allowed to edit it.

Then the template for “view”, classifieds/templates/index.html, is defined as shown in Listing 6.

Implementing the classified submission mechanism

To allow users publish classifieds, we’ll write two controller actions (once again, one to display the form and the other to process it).

The action that will display the form should first check that the user is allowed to add a classified; this is, we should use repoze.what’s has_permission predicate so that it checks whether they are granted the “add-classified” permission. You have to import it at the top of classifieds/controllers/root.py:

from repoze.what.predicates import has_permission

And then write the action as shown below:

@expose('classifieds.templates.add')
@require(has_permission('add-classifieds'))
def add(self):
return {'page': 'Classified submission'}


This action uses the template classifieds/templates/add.html, which we have to create with the contents of Listing 7.

Finally, Listing 8 shows how the action that processes the submitted form is implemented. There we use something we hadn’t used before: The repoze.who identity dictionary. Do you remember that I said that repoze.who optionally uses so-called “metadata providers”, which are plugins that load data about the current user into the request? Well, that data is kept in the WSGI environment dictionary, under the repoze.who.identity key.

Then we use one of the items of that dictionary, “user”, which contains the database object for the current user. It is loaded by the metadata provider defined in the repoze.who SQLAlchemy plugin (repoze.who.plugins.sa), a component that is enabled by default in TurboGears.

Implementing the classified edition mechanism with custom predicate checkers

So far we’ve used a few repoze.what predicate checkers, which are all very basic. Very often you have to write your own checkers. For example, if the URL where classifieds are edited looks like http://localhost:8080/edit/X (where “X” represents the classified identifier), and we want that classifieds can only be edited by their posters, we’ll need a predicate checker that finds the poster of the “X” classified and checks if it is the current user.

This predicate checker is implemented in Listing 9, which should be stored in classifieds/lib/authz.py (a file you should create). That’s a good sample checker, which helps us to understand how a predicate checker is defined:

• It must extend the repoze.what.predicates.Predicate class.
• It must define a user-friendly explanation in the message attribute, which may be shown to the user when the predicate is not met.
• Its logic is defined in the evaluate() instance method, which must call the unmet() method when the predicate is not met; keyword arguments passed to this method will replace the placeholders defined in message (if any), although that message can be replaced on-the-fly with a string passed as the first positional argument. evaluate() receives the WSGI environment and the repoze.what credentials dictionaries as arguments.
• If the checker relies on arguments such as GET or POST variables, or other arguments available in the URL, the parse_variables() method should be used to retrieve them. It will return a dictionary whose items are: “get” for variables in the query string and “post” for POST variables, plus “named_args” and “positional_args” for named and positional arguments in the URL (which must be set by a routing software like Selector or Routes).

Because this predicate relies on a named argument in the URL (“classified”), we have to use a URL router software compliant with the wsgiorg.routing_args standard; we’ll use Routes. To configure Routes in TurboGears 2, you have to insert the following contents in classifieds/config/app_cfg.py (right after the imports) and then replace the line base_config = AppConfig() with base_config = ClassifiedsConfig():

class ClassifiedsConfig(AppConfig):
def setup_routes(self):
"""Customize routing"""
from tg import config
from routes import Mapper
dir = config['pylons.paths']['controllers']
map = Mapper(directory=dir, always_scan=config['debug'])
# Defining our custom routes:
map.connect('/{action}/{classified:\d+}', controller='root')
# Required by TurboGears:
map.connect('*url', controller='root', action='routes_placeholder')
config['routes.map'] = map


At this point we’re ready to use the user_is_poster checker.

Now import the custom checker into classifieds/controllers/root.py:

from classifieds.lib.authz import user_is_poster

Next, use the following contents to define the “edit” action and the contents of Listing 10 for its template (classifieds/templates/edit.html):

@expose('classifieds.templates.edit')
@require(user_is_poster())
def edit(self, classified):
query = DBSession.query(model.Classified)
classified_obj = query.get(classified)
return {'page': 'Classified edition page', 'classified': classified_obj, 'classified_id': classified}

Finally, for the action that processes the submitted form, we can use:

@expose()
@require(user_is_poster())
def edit_classified(self, classified, title, contents):
# Fetching and updating the classified object:
query = DBSession.query(model.Classified)
classified_obj = query.get(classified)
classified_obj.classified_title = title
classified_obj.classified_contents = contents
DBSession.update(classified_obj)
# Notifying the user:
flash('Classified "%s" updated!' % title)
redirect(url('/view/%s' % classified))


You can now play with the classifieds edition mechanism, to see by yourself how authorization is denied when somebody tries to edit somebody else’s classified, thanks to our “user_is_poster” checker.

Configuring it all by ourselves

At this point you’re able to deal with identification (using the repoze.who identity dictionary, which contains user data) and control authorization in your application with repoze.what predicate checkers (even how to write your own!), and you should also be able to put this knowledge to the test in frameworks other than TurboGears.

What we’re missing now is to know how to configure repoze.who and repoze.what by ourselves, since we skipped that part initially because TurboGears configures them for us. But you have to know this to use both packages with other frameworks or even to continue with TurboGears in a more advanced setup.

Therefore we’re going to stop TurboGears from configuring repoze.who and repoze.what, so that we have full control on how they are configured and learn how to do it in other frameworks.

To disable the automatic setup of repoze.who and repoze.what, go to classifieds/config/app_cfg.py and set the variable base_config.auth_backend to None. Then all those variables that start by base_config.sa_auth will be ignored, so you can remove them if you want.

Now let’s handle the configuration by adding the middleware to our WSGI application. I’ll use a function called add_auth() (defined in classifieds/config/auth.py) which receives the WSGI application as the only argument and returns it with the middleware added, as shown in Listing 11.

Because add_auth() configures repoze.who and repoze.what the same way we’ve been using them, but with the hidden details revealed, we’ll be able to identify their components.

In this function we see that repoze.who is configured with the following plugins:

• AuthTktCookiePlugin, an identifier which remembers and forgets authenticated users using cookies (using the string “secret” as the encryption key and “authtkt” as the cookie name).
• FriendlyFormPlugin, the component that handles our login form and logouts. As an identifier, when the user is logging in it extracts the “login” and “password” from the request so that the authenticator(s) can use such data, and when the user tries to log out, it asks AuthTktCookiePlugin to forget the user. As a challenger, it redirects the user to the login form (at “/login”).
• SQLAlchemyAuthenticatorPlugin, as the only authenticator used. It connects to the “tg_user” table to check if there’s a match for the supplied “login” and “password”.
• SQLAlchemyUserMDPlugin, the metadata provider that loads the current user’s SQLAlchemy object into the repoze.who identity item “user”.
• Because we didn’t specified request classifiers or challenge deciders, the default ones will be used.

Meanwhile, repoze.what is configured using the groups/permissions-based authorization pattern, where the groups and permissions are retrieved thanks to the following adapters:

• SqlGroupsAdapter, which loads the groups from the “tg_group” table.
• SqlPermissionsAdapter, which loads the groups from the “tg_permission” table.

If we don’t use this authorization pattern, our app_with_mw variable would have been defined without passing groups/permission adapters:

app_with_mw = setup_auth(app, **who_args)

And what about using repoze.who but not repoze.what? Listing 12 shows how to configure repoze.who just like we did above, but without repoze.what. Note that this time we had to pass the request classifier and challenge decider explicitly.

The opposite, using repoze.what without repoze.who, is not yet possible as of this writing because repoze.what 1.0’s credentials are loaded through a repoze.who metadata provider. repoze.what 1.1 will be completely repoze.who-independent, optionally.

Finally, it’s time to use add_auth(). It can be used like any other WSGI middleware, so in the case of this TurboGears 2 application, it is in classifieds/config/middleware.py; if using another framework, consult the relevant documentation to find the equivalent. First, you should import the function:

from classifieds.config.auth import add_auth

And then use it inside the make_app() function, under the specified line:

# Wrap your base TurboGears 2 application with custom (...)
app = add_auth(app)

And voila! Now our classifieds service behaves the same way as before, with its authentication, identification and authorization settings now being controlled by our own code instead of the generated defaults.

Going beyond with repoze.who and repoze.what

The topics covered in this article are just the tip of the iceberg. Both Repoze packages are created with extensibility in mind; their core is minimalist but they already have many ready-to-use plugins, not only the repoze.who and repoze.what SQLAlchemy plugins we used here.

There are repoze.who plugins for OpenId, LDAP, .htaccess and RADIUS authentication, as well as a built-in challenger plugin for HTTP authentication — just to name some of the available plugins. And you can easily create your own plugins, following the patterns described in this article.

Although repoze.what is a relatively new piece of software as of this writing, it has several ready-to-use plugins as well. It has plugins to store the groups and permissions in XML files or .ini files, not only in databases, as well as a plugin called repoze.what-quickstart which allows us to have repoze.who and repoze.what working quickly (that’s what TurboGears uses to set them up).

Although they were not used in our classifieds service, just mentioned in the beginning, you can have so-called compound predicates. Access rules aren’t always as simple as “The user must be logged in” or “The user belongs to the ‘posters’ group”. For example, in our classified edition mechanism we way want to allow administrators to edit classifieds, even those not posted by themselves; to do so, instead of using our user_is_poster checker alone, we could use it along with the Any and in_group checkers:

from repoze.what.predicates import Any, in_group
p = Any(in_group('manager'), user_is_poster())

In this article we didn’t even use half of the built-in repoze.what predicate checkers. A full list is available in the repoze.what manual.

Settings are very flexible. It is even possible to configure repoze.who and repoze.what through .ini files, so that those settings can be adjusted while deploying the application (which would be a replacement for our add_auth() function defined above).

It is also worth noting that both projects are actively developed, well documented, well tested and supported by the Repoze community. As a result, it is most likely that you’ll have a good experience using them.

repoze.who and repoze.what aim to solve “the security problem” we Web developers face so often, and they have proved to be the right choice in many scenarios. The likelihood of them being the right choice for your next Web application is strong, specially now that you’re familiar with them!

To ask questions about repoze.who and/or repoze.what, please use the repoze-dev mailing list. For everything else that is related to this article, please leave a comment below.

svnadmin dump /path/to/your/repository > ./repository.dump

And how to restore it:

mkdir /new/path/to/your/repository
svnadmin create /new/path/to/your/repository
svnadmin load /new/path/to/your/repository < ./repository.dump

But don’t tell you what to do to solve the problem that will arise when you try to restore it:

svnadmin: Dump stream contains a malformed header (with no ':') at '* Dumped revision 0.'

The solution is too simple; just run:

grep --binary-files=text -v '^* Dumped revision' ./repository.dump > ./repository.clear-dump

Now you should be able to restore your subversion repository with no problems:

svnadmin load /new/path/to/your/repository < ./repository.clear-dump

PS: I don’t use hotcopy because I prefer to have a single file for these backups.

I’m going to show you how to install it the easiest way, from my experience with the GLM Translation Service under Debian 4.0 (this guide might also work under Ubuntu, though). I’ll use the sudo utility, so I assume that you’ve already installed and configured sudo accordingly for your user.

Installing dependencies

Some of the Pootle’s dependencies are available in Debian repositories, so we may install them by running:
sudo aptitude install python-dev python-pysqlite2 python-celementtree subversion bzip2 g++ curl

We’re now going to install the other software packages required by Pootle. First, select the directory where you want to download and compile your software, for example:
cd /usr/src

To get the source code of the packages and extract them, run the following commands:
curl http://www.kid-templating.org/dist/0.9.6/kid-0.9.6.tar.gz | tar xzv
curl http://dfn.dl.sourceforge.net/sourceforge/translate/translate-toolkit-1.1.1.tar.bz2 | tar xjv
curl http://garr.dl.sourceforge.net/sourceforge/translate/Pootle-1.1.0.tar.bz2 | tar xjv
curl http://jtoolkit.sourceforge.net/snapshots/jToolkit-0.7.8/jToolkit-0.7.8.tar.gz | tar xzv


Installing the applications:
sudo python translate-toolkit-1.1.1/setup.py install
sudo python kid-0.9.6/setup.py install
sudo Pootle-1.1.0/pootlesetup.py install
sudo jToolkit-0.7.8/jToolkitSetup.py install


Do not install PyLucene

Pootle is supposed to work better with PyLucene, but if you (like me) hate headaches, you won’t install PyLucene on Debian Etch. If you try to install it, you’d get this error:
install: cannot stat `/usr/lib/security/libgcj.security': No such file or directory
make: *** [install] Error 1

There’s nothing you could install that would create that file; don’t you even try to create it as an empty file: You’d be able to install PyLucene, but Pootle won’t work and you’d get this error when you’ll try to run it:
GC Warning: Repeated allocation of very large block (appr. size 65536):
May lead to memory leak and poor performance.

If it’s too late, and you’ve already installed PyLucene with libgcj.security as an empty file (as suggested in some mailing-lists), you may fix it by running the following commands:
sudo mv /usr/lib/python2.4/site-packages/_PyLucene.so /usr/lib/python2.4/site-packages/_PyLucene.so.old
sudo mv /usr/lib/python2.4/site-packages/PyLucene.py /usr/lib/python2.4/site-packages/PyLucene.py.old
sudo mv /usr/lib/python2.4/site-packages/PyLucene.pyc /usr/lib/python2.4/site-packages/PyLucene.pyc.old

Don’t you have a mail server?

If your mail server is not in the same host as Pootle, you’ll need to install one. Just run the command below to install Postfix and then select “Internet website” when asked about what you’ll use postfix for:
sudo apt-get install postfix

Configuring Pootle

To make it easy for you to configure and play with Pootle, I suggest you create softlinks to Pootle’s configuration files in a folder like /etc/pootle:
sudo mkdir /etc/pootle
sudo ln /usr/lib/python2.4/site-packages/Pootle/pootle.prefs -s /etc/pootle/pootle.conf
sudo ln /usr/lib/python2.4/site-packages/Pootle/users.prefs -s /etc/pootle/users.conf
sudo ln /usr/lib/python2.4/site-packages/Pootle/html -s /etc/pootle/html
sudo ln /usr/lib/python2.4/site-packages/Pootle/templates -s /etc/pootle/templates

This way, any file you might ever need to edit will be in /etc/pootle.

Now let’s edit the main configuration file in Pootle: sudo nano /etc/pootle/pootle.conf

It’s well-documented, so I won’t talk a lot about it. I just suggest you only set the following parameters accordingly:

• description: Describe your website powered by Pootle, with a text aimed at (potential) translators.
• fromaddress.
• supportaddress: The email address for translators to make questions. This might be a mailing list.
• defaultrights: I suggest you set it to defaultrights = "view, suggest, archive, pocompile, translate, commit"
• podirectory: It’s the path to your translation files. For this HOWTO I’ll use /var/translations.

Your pootle user

I suggest you create a user for running pootle:
sudo adduser --disabled-password --disabled-login pootle

Setting up service scripts for Pootle

Create the file /etc/init.d/pootle as root with the following contents:
#!/bin/bash
# /etc/init.d/pootle: start and stop the Pootle Server
# Pootle runs as user -pootle- via sudo
# This script is based on the one published here http://translate.sourceforge.net/wiki/pootle/service_scripts
PREFIX=/usr/local/pootle
test -x /usr/bin/PootleServer || exit 0
. /lib/lsb/init-functions
case "$1" in
start)
log_begin_msg "Starting Pootle Server..."
/usr/bin/sudo -u pootle $PREFIX/start_pootle &
log_end_msg 0
;;
stop)
log_begin_msg "Stopping Pootle Server..."
$PREFIX/stop_pootle >/dev/null 2>&1
log_end_msg 0
;;
restart)
log_begin_msg "Stopping Pootle Server..."
$PREFIX/stop_pootle >/dev/null 2>&1
log_end_msg 0
log_begin_msg "Starting Pootle Server..."
sudo -u pootle $PREFIX/start_pootle &
log_end_msg 0
;;
status)
$PREFIX/status_pootle
;;
*)
log_success_msg "Usage: /etc/init.d/pootle {start|stop|restart|status}"
exit 1
esac
exit 0

Now let’s create, as root, the auxiliary scripts required by the script above:

/usr/local/pootle/start_pootle

#!/bin/bash
# $PREFIX/start_pootle
# Start script in port 8080, separates errors from normal logs
# based on a script published here: http://translate.sourceforge.net/wiki/pootle/service_scripts
export HOME="/home/pootle"
/usr/bin/PootleServer -p 8080 $@ >> /var/log/pootle/`date "+%F"` 2>> /var/log/pootle/`date "+%F"`.err

/usr/local/pootle/stop_pootle


#!/bin/bash
# Stop script $PREFIX/stop_pootle
# based on a script published here: http://translate.sourceforge.net/wiki/pootle/service_scripts
echo "Killing Pootle"
pidpootle=`ps -ef |grep PootleServer |grep python | awk '{print $2}'`
pidpootle2=`ps -ef |grep start_pootle |grep bin | awk '{print $2}'`
kill $pidpootle2 >/dev/null 2>&1
kill $pidpootle >/dev/null 2>&1

/usr/local/pootle/status_pootle

#!/bin/bash
# Pootle Status $PREFIX/status_pootle
# based on a script published here: http://translate.sourceforge.net/wiki/pootle/service_scripts
pidpootle=`ps -ef |grep PootleServer |grep python | awk '{print $2}'`
if [ "$pidpootle" != "" ]
then
echo "Pootle Server running in pid ($pidpootle)"
else
echo "Pootle Server is not running"
fi

Make them work!

Now we have to make these scripts useful:
sudo chmod +x /etc/init.d/pootle
sudo chmod +x /usr/local/pootle/s*
sudo update-rc.d pootle

Configuring Pootle to use your repository

Pootle supports several VCSs, but I’ll only teach you how to setup a Subversion working copy that Pootle may update and commit. If you don’t use a version control system, I recommend you do so. If you’re sure you don’t need it, you may skip this section. If you’re using another VCS, you’ll need to go to the Pootle’s website to learn how to configure it with Pootle.

Setup your working copy

Let’s create the working copy at, say, /var/translations. To do that in GNU/Linux Matters, we would run:
sudo svn co https://svn.gnulinuxmatters.org:81/i18n /var/translations
You should replace the URL by yours.
And don’t forget to set the pootle user as the owner:
sudo chown pootle -R /var/translations

Create basic cronjobs

We should run from time to time two scripts:

• One to commit the changes made by translators; ideally, these changes would be committed by translators themselves, but from my experience you should not count on that (most of them forget about it or just don’t know they can do that).
• Another to update your PO templates from the repository.

So we’re going to ask our pootle user to do this by creating cronjobs. First, become pootle:
sudo su - pootle
Then run crontab -e and add the following two lines:
*/15 * * * * /usr/bin/svn update /var/translations >> /dev/null 2>> /home/pootle/update.err
0 0,8,16 * * * /usr/bin/svn commit -m "Uncommitted translations in the last 8 hours" /var/translations >> /dev/null 2>> /home/pootle/commit.err

Pootle is running on port 8080

For security reasons, if you followed this HOWTO you’ll be running as the pootle user, not as root. And due to a UNIX misfeature, only root may open ports below 1024. So, unless you love appending the port number to your URLs, and if port 80 is available on your system, you’d proxy Pootle through a webserver.

The proxy trick is recommended, but if you cannot do that or just don’t want, you can use iptables to redirect traffic from port 8080 to 80. My suggestion would be to use the next iptables ruleset:
# Generated by iptables-save v1.3.6 on Thu Jan 24 16:40:59 2008
*nat
REROUTING ACCEPT [0:0]
OSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080
COMMIT
# Completed on Thu Jan 24 16:40:59 2008
# Generated by iptables-save v1.3.6 on Thu Jan 24 16:40:59 2008
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [495:60715]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state INVALID -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8080 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
-A INPUT -j REJECT --reject-with icmp-proto-unreachable
-A FORWARD -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Thu Jan 24 16:40:59 2008

For more information about this basic iptables-based firewall, you may read my Setting up your first server HOWTO.

Criticism? Suggestions? Complaints?

Leave a comment!

• XVidCap: The workaround is supposed to be disabling sound, but didn’t work.
• RecordMyDesktop (plus its two front-ends): Crashes with no error message. From time to time I was able to record videos.
• Istanbul: Crashes if you select a window to be recorded.

So, the only solution is to install a newer version of Istanbul for Debian:
http://ftp.fr.debian.org/debian/pool/main/i/istanbul/istanbul_0.2.2-4_i386.deb
or, if you have a 64-bit box, http://ftp.fr.debian.org/debian/pool/main/i/istanbul/istanbul_0.2.2-4_amd64.deb

After making the appropriate bug reports, I can finally start recording the screencasts with Istanbul!

PS: Forget about the above. That Debian package doesn’t work either. I’m installing KDE4 to check whether I can record the screencast with it; I think I saw such an option when I tried it out.

PPS: The KDE4 built-in screencasting component uses a weird format (cps?) which cannot be opened with VLC, Mplayer nor Kaffeine (but there’s a dirty workaround). Anyways, it records the whole screen and I just want to record a single window.

PPPS: I’m not alone.

Unless you have good reasons to use another GNU/Linux distribution, I recommend you to install Debian. It has a lot of ready-to-install applications, is very stable and it’s perhaps the distribution with more tutorials around.

Please notice that this is a very basic tutorial and has only been tested on Debian.

Connect to your server

First of all, log in as root:
ssh root@123.123.123.123 # where 123.123.123.123 is your server's IP address

Some hosting providers disable ssh root access, so you will need to replace root by your user name. If this is the case, after you log in you should become root:

su -

Update your system

aptitude update
aptitude upgrade
aptitude dist-upgrade

Add your user

If your hosting provider disables root access, then you should skip this step.

adduser emacs

Replace emacs by VI VI VI if you don’t believe in Saint IGNUcius.

Sudo setup

sudo is a very useful utility, and I recommend you to use it.

First, let’s install it:

aptitude install sudo

Then, we add your user to the list of sudoers, by running visudo and then adding the following line at the end of the file emacs ALL=(ALL) ALL.

Now you become yourself:

su emacs -

Shared key ssh authentication

At this point you should use shared key ssh authentication, but for that there’s a great tutorial at ammonlauritzen.com.

Configuring the SSH daemon

Open /etc/ssh/sshd_config with your favorite text editor, say:

sudo nano /etc/ssh/sshd_config

And make sure the following lines are set this way, if not, add or modify them accordingly:
PermitRootLogin no
PasswordAuthentication no
X11Forwarding no
UsePAM no
AllowUsers emacs # separate two or more usernames by spaces

Finally, apply your modifications:
sudo /etc/init.d/ssh reload

Don’t log out yet, we need to check that you will be able to access your server via ssh (this is, that you didn’t break anything on the /etc/ssh/sshd_config file). To check if everything is OK, try to log in:
ssh emacs@123.123.123.123

If you’re able to access, then it’s well configured and you may close the second session. If not, then you should check your modifications and try again.

Setting up a basic firewall

We are going to setup a very basic firewall with the powerful netfilter/iptables. For this step you need to be root:
sudo -s

First, store the current iptables rules, in case something goes wrong with ours:
iptables-save > /etc/iptables.conf.old

Now, create the file /etc/iptables.conf and add the following contents:
# boring stuff for someone new to server administration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [495:60715]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state INVALID -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# this is the port used by the SSH daemon
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
-A INPUT -j REJECT --reject-with icmp-proto-unreachable
-A FORWARD -j REJECT --reject-with icmp-port-unreachable
COMMIT

Please pay attention to this line:
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

You should use a line similar for every open port that you want to be accessible from the Internet. This is, if you have a webserver, you should copy that line but replace “22″ by “80″ (or any other port):
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

This is how you “enable” ports.

Then we load the configuration (and don’t log out until we test it!):
iptables-restore < /etc/iptables.conf

Testing the rules

To test the rules, open another terminal and try to access your server:

ssh emacs@123.123.123.123

If you could access, then the rules should be OK. If not, reload the original rules until you find help:
iptables-restore < /etc/iptables.conf.old

Loading the rules when the server stars

If the rules we defined work, then our the firewall should be loaded when the server starts:

Create the file /etc/network/if-pre-up.d/iptables with the following contents:
#!/bin/bash
/sbin/iptables-restore < /etc/iptables.conf

Then make it executable:
chmod +x /etc/network/if-pre-up.d/iptables

We can now go back to our normal user:
exit

Your server is ready!

At this point, you are ready to start installing applications on your brand-new server!

What’s next?

• Make sure you already know the basics of GNU/Linux.
• Learn more about Debian.
• Find the HOWTO you were looking for.