Stein Magnus Jodal

Speeding up a Django web site without touching the code

2011-10-19T00:00:00+02:00

I’ve recently been tweaking my server setup for a Django 1.3 web site with the goal of making it a bit faster. Of course, there is a lot of speed to gain by improving e.g. the number of database queries needed to render a web page, but the server setup also has an effect on the web site performance. This is a log of my findings.

All measurements have been done using the ab tool from Apache using the arguments -n 200 -c 20, which means that each case have been tested with 20 concurrent requests up to 200 requests in total. The tests was run from another machine than the web server, with around 45ms RTT to the server. This is not a scientific measurement, but good enough to let me quickly test my assumptions on what increases or decreases performance.

The Django app isn’t particularly optimized in itself, so I don’t care much about the low number of requests per second (req/s) that it manages to process. The main point here is the relative improvement with each change to the server setup.

The baseline setup is a Linode 1024 VPS (Referral link: I get USD 20 off my bill if you sign up and remain a customer for 90 days), running Apache 2.2.14 with mpm-itk, mod_wsgi in daemon mode with maximum 50 threads and restart every 10000 requests, SSL using mod_ssl, and PostgreSQL 8.4.8 as the database. For the given Django app and hardware, this setup is strolling along at 4.0 req/s.

With this blog post as reference, I switched from Apache+mod_wsgi to using nginx 0.7.5 as SSL terminator, for serving static media, and as a proxy in front of Gunicorn 0.13.4. Gunicorn is a WSGI HTTP server, hosting the Django site. The Linode VPS got access to four CPU cores (n=4), so I set up nginx with 4 workers (n) and Gunicorn with 9 workers (2n+1). Different values for these settings are sometimes recommended, but this is what I’m currently using. This setup resulted in an increase to 9.0 req/s.

A nice improvement, but I changed multiple components here, so I don’t know exactly what helped. It would be interesting to test e.g. Apache with mod_proxy in front of Gunicorn, as well as different number of nginx and Gunicorn workers. The nginx version is also a bit old, because I used the one packaged in Ubuntu 10.04 LTS. I should give nginx 1.0.x a spin.

Next up, I added pgbouncer 1.3.1 (as packaged in Ubuntu 10.04 LTS, latest is 1.4.2) as a PostgreSQL connection pooler. I let pgbouncer do session pooling, which is the safest choice and the default. Then I changed the Django app settings to use pgbouncer at port 6432, instead of connecting directly to PostgreSQL’s port 5432. This increased the performance further to 10.5 req/s.

Then, I started looking at SSL performance, without this being the bottleneck at all. I learned a lot about SSL performance, but didn’t improve the test results at all. Some key points was:

nginx defaults to offering Diffie-Hellman Ephemeral (DHE) which takes a lot of resources. Notably, the SSL terminators stud and stunnel does not use DHE. See this blog post for more details and how to turn off DHE in nginx.
If you’re using AES, you can process five times as many requests with a 1024 bit key compared to a 2048 bit key. I use a 2048 bit key.
64-bit OS and userland doubles the connections per second compared to 32-bit. My VPS is stuck at 32-bit for historical reasons.
SSL session reuse eliminates one round-trip for subsequent connections. I set this up, but my test setup only use fresh connections, so this improvement isn’t visible in the test results.
Browsers will go a long way to get hold of missing certificates in the certificate chain between known CA certificates and the site’s certificate. To avoid having the browser doing requests to other sites to find missing certificates, make sure all certificates in the chain are provided by your server.

If you’re switching from Apache to Nginx, note that Apache uses separate files for your SSL certificate and the SSL certificate chain, while Nginx wants these two files to be concatenated to a single file, with your SSL certificate first.

Next, I read about transaction management and the use of autocommit in Django. The Django site I’m testing is read-heavy, with almost no database writes at all. It doesn’t use Django’s transaction middleware, which means that each select/update/insert happens in its own transaction instead of having one database transaction spanning the entire Django view function.

Since I’m using PostgreSQL >= 8.2, which supports INSERT ... RETURNING, I can turn on autocommit in the Django settings, and keep the transaction semantics of a default Django setup without the transaction middleware. Turning on autocommit makes PostgreSQL wrap each query with a transaction, instead of Django adding explicit BEGIN, and COMMIT or ROLLBACK statements around each and every query. Somewhat surprisingly, this reduced the performance to 9.2 req/s. Explanations as to why this reduced the performance are welcome.

Reverting the autocommit change, I got back to 10.5 req/s. Then I tried tuning the PostgreSQL configuration using the pgtune tool. I went for the web profile, with autodetection of the amount of memory (1024 MB):

pgtune -i /etc/postgresql/8.4/main/postgresql.conf -o postgresql-tuned.conf -T Web
mv postgresql-tuned.conf /etc/postgresql/8.4/main/postgresql.conf

pgtune changed the following settings:

maintenance_work_mem = 60MB        # From default 16MB
checkpoint_completion_target = 0.7 # From default 0.5
effective_cache_size = 704MB       # From default 128MB
work_mem = 5MB                     # From default 1MB
wal_buffers = 4MB                  # From default 64kB
checkpoint_segments = 8            # From default 3
shared_buffers = 240MB             # From default 28MB
max_connections = 200              # From default 100

After restarting PostgreSQL with the updated settings, the performance increased to 11.7 req/s.

To summarize: in a few hours, I’ve learned a lot about SSL performance tuning, and–without touching any application code–I’ve almost tripled the amount of requests that the site can handle. The performance still isn’t great, but it’s a lot better than what I started with, and the setup is still far from perfect.

To get further speed improvements, I would mainly look into three areas: adding page (or block) caching where appropriate, log database queries and tweak the numerous or slow ones, and look further into tweaking the PostgreSQL settings. But, that’s for another time.

If you have suggestions for other server setup tweaks, please share them in the comments, and I’ll try them out.

Updated: Removed the “mean response time” numbers, which simply is (time of full test run) / (number of requests). It just told us the same as req/s in a less intuitive way. The other interesting number here is the perceived latency for a single user/request. I’ll make sure to include it in future posts.

Traversable attributes in Pykka

2011-10-06T00:00:00+02:00

In Pykka 0.13–which was released almost two weeks ago–traversing the attributes of an actor is about 8.3 times faster than it used to be. To paraphrase Apple: “8.3X faster. That’s amazing!” (Update: This was written a couple of hours before the news of Jobs’ passing arrived. May he continue to inspire us.)

So, what is “traversable attributes”? Let’s take a few steps back.

If we were a conservative actor adhering strictly to the actor model, we surely wouldn’t share our attributes with anybody else. We would expect other actors to send serializable messages to us, asking nicely to get the value of the attribute, or maybe asking for something else. Of course, the other actors wouldn’t even know the attribute existed unless we told them, and even then they wouldn’t ever dream of requesting a reference to our attribute or altering the attribute directly. It would be indecent. It would break the rules of the actor model. It would be unsafe.

When using Pykka, you can keep to the traditional way of passing messages back and forth between the actors. You start the actor by calling the Actor.start() class method, which returns an ActorRef object. This object can safely be passed around and even shared between threads. The ActorRef object got two methods for sending messages to the actor, called send_one_way() and send_request_reply(). This is nice enough by itself, and it gives you a way to build concurrent applications which is easier to reason about–just as promised by advocates of the actor model–than when you do the thread and lock management dance. You can quickly hack together a simple actor implementation like this from scratch for each and every application you make using e.g. Thread and Queue. I’ve done this a couple of times, and it works.

But, I wanted a bit more, so I created Pykka.

First, I wanted to get rid of verbose dict messages all over my code base. I just wanted to call regular methods and access regular attributes on regular objects. Pykka provides a safe way of doing this, called ActorProxy. An ActorProxy is nothing more than a wrapper around an ActorRef. It does all it’s magic by sending messages to the actor, just like you used to do yourself.

from pykka.actor import ThreadingActor

class X(ThreadingActor):
    y = 1

>>> x = X.start().proxy()
>>> x.y.get()
1
>>> x.stop()

Second, I wanted to be able to organize actors like regular code, e.g. by splitting them into multiple classes. Imagine a running actor a which have the attribute b. The “subobject” b have the method c().

from pykka.actor import ThreadingActor

class B(object):
    def c(self):
        return 1 + 1

class A(ThreadingActor):
    b = B()

If you call a.b.c(), the following happens:

We send a message to actor a requesting attribute b, and immediately get a future object back which is our handle to the result which will be available in the future.
Actor a gets the message, looks up attribute b, and returns a copy of the object referenced by the b attribute.
We call c() on the future, but the Future class doesn’t have an attribute called c, so it fails. Alternatively, we use the future correctly and call get() on the future to get the real result, a copy of b. Then we call c() on the copy of b. The method c() is now running, but it is running in the caller’s thread, and not in the actor a like I wanted it to do.

>>> a = A.start().proxy()
>>> a.b.c()
AttributeError: 'ThreadingFuture' object has no attribute 'c'
>>> a.b.get().c()
2   # Result calculated in the caller's thread
>>> a.stop()

The simple attribute access that the ActorProxy provides isn’t enough to make this work.

To make the a.b.c() method call be executed in the actor a instead of the caller’s thread, we need to traverse attribute b without having it returned to us, so that we can get to c() while still inside the actor a, and call its method c(). We need what we in Pykka call traversable attributes.

To make an attribute traversable, the only thing we need to do is to mark it as such by adding the attribute pykka_traversable to the traversable attribute:

from pykka.actor import ThreadingActor

class B(object):
    pykka_traversable = True
    def c(self):
        return 1 + 1

class A(ThreadingActor):
    b = B()

>>> a = A.start().proxy()
>>> a.b.c().get()
2   # Result calculated by the actor `a`
>>> a.stop()

When you access a regular attribute of a Pykka actor, you just get a future object, which, when you call get() on it, will return a copy of the attribute. When you access a traversable attribute of a Pykka actor, you get a brand new ActorProxy which wraps the same ActorRef, but method calls and attribute accesses on the new proxy object will work on the actor’s attribute instead of the actor itself.

Speeding up access to traversible attributes

If you’re still following, you’re maybe wondering how we sped up access to traversable attributes with a factor of 8.3. The answer is a few lines up: “you get a brand new ActorProxy.”

So, why should that matter?

If you split your actor into multiple classes using traversable attributes, you’re probably going to use each traversable attribute more than once. Maybe really often. Turns out, creating brand new ActorProxy objects for the same attribute over and over again is kind of wasteful.

How did you find out?

John Bäckstrand was irritated by Mopidy being almost unusable on his slow system, and attacked the problem in the scientific way: by measuring where the bottleneck was. John quickly pointed out that access to second-level attributes, which required the traversal of a traversable attribute, was five times slower than access to first-level attributes, which didn’t involve traversable attributes. This observation made it obvious that the creation of new ActorProxy objects whenever we accessed traversable attributes–even though the proxy objects didn’t contain any state and was fully reusable–probably needed refinement.

To be sure we fixed the issue, we started by writing a performance test which compared attribute access with and without the traversal of a traversable attribute.

# Using Pykka 0.12.4
test_direct_plain_attribute_access took 0.958s
test_direct_callable_attribute_access took 0.977s
test_traversible_plain_attribute_access took 8.259s
test_traversible_callable_attribute_access took 8.344s

Then, the fix was short and easy: Cache and reuse ActorProxy objects.

diff --git a/pykka/proxy.py b/pykka/proxy.py
index 27c075b..4c6b908 100644
--- a/pykka/proxy.py
+++ b/pykka/proxy.py
@@ -58,6 +58,7 @@ class ActorProxy(object):
         self.actor_ref = actor_ref
         self._attr_path = attr_path or tuple()
         self._known_attrs = None
+        self._actor_proxies = {}
 
     def _update_attrs(self):
         self._known_attrs = self.actor_ref.send_request_reply(
@@ -88,7 +89,10 @@ class ActorProxy(object):
         if attr_info['callable']:
             return _CallableProxy(self.actor_ref, attr_path)
         elif attr_info['traversable']:
-            return ActorProxy(self.actor_ref, attr_path)
+            if attr_path not in self._actor_proxies:
+                self._actor_proxies[attr_path] = ActorProxy(
+                    self.actor_ref, attr_path)
+            return self._actor_proxies[attr_path]
         else:
             message = {
                 'command': 'pykka_getattr',

The result was immediate: The performance test for traversable attribute access showed an 8.3X improvement.

# Using Pykka 0.13
test_direct_plain_attribute_access took 0.953s
test_direct_callable_attribute_access took 0.988s
test_traversible_plain_attribute_access took 0.984s
test_traversible_callable_attribute_access took 1.006s

Mopidy use Pykka’s traversable attributes heavily to organize its backend code. Obviously, we try to avoid wiring up lots of actors in Mopidy’s unit tests, but we’ve been lazy and use some actors in the tests. These five lines of code inserted at the right place in a dependency made Mopidy’s test suite run 20% faster, and made John’s use case run 166% faster.

We could use more of five-line patches like that :-)

Hva alle utviklere må vite om tegnsettenkoding

2011-09-08T00:00:00+02:00

English: This is slides and video from the Norwegian lightning presentation on character encoding I did at the JavaZone 2011 conference today.

Takk for nok en knall konferanse :-)

Her er slidene mine fra lyntalen om tegnsettenkoding jeg holdt på JavaZone 2011 i dag. PDF-versjon er også tilgjengelig, by popular demand.

Og her er videoen ute, mindre enn åtte timer senere. Smidig! :-)

pyspotify 1.2 released

2011-06-08T00:00:00+02:00

pyspotify is a Python wrapper for libspotify, which give developers access to the Spotify music streaming service.

Today, I tagged pyspotify 1.2 at GitHub and pushed the new release to PyPI. I’ve also made deb packages of libspotify and pyspotify available at apt.mopidy.com.

The 1.2 release brings pyspotify up to date with libspotify 0.0.8, which was released by Spotify a couple of weeks ago. It also fixes a bunch of memory issues. pyspotify does not implement the full libspotify API, but a significant and usable part of it. With pyspotify 1.2 all the old and broken unit tests have been fixed, and a bunch of new tests have been developed. For the first time, pyspotify is documented, and it comes with a new web site.

This is the first release of pyspotify to PyPI since Doug Winter’s release of 1.1 in April 2010. Since Doug hasn’t found the time to maintain pyspotify further, Johannes Knutsen and myself have been patching pyspotify somewhat ad hoc, adding just what we needed to keep pyspotify barely working with the new releases of libspotify and Mopidy. Back in January, I explored the use of Cython for a new alternative libspotify binding, spoticy. Using Cython for this was a great success as far as I took it, but I had other Mopidy related side projects to complete first (aka Pykka). Because Mopidy depends heavily on pyspotify, I also asked Doug to transfer the maintenance of pyspotify to the Mopidy project, which he finally decided to do now in May.

In February, Antoine Pierlot-Garcin started sending us patches for our branch of pyspotify. Since then, he has been steadily improving pyspotify. All the improvements listed above is due to Antoine’s work, and the 1.2 release is to his credit.

I hope this will make the situation around pyspotify clearer, and that we’ll soon see more projects using pyspotify to do great stuff with the Spotify service.

Le pyspotify est mort, vive le pyspotify.

Log from the debugging of a segfault

2011-05-24T00:00:00+02:00

The following is a cleaned up log I wrote for myself while debugging a bug. Writing a log while working helps me keep track of the debugging effort in case I’m interrupted (life, sleep, work, etc.). It also requires me to explain all findings to myself in fully spelled out sentences, making my thoughts considerably easier to follow.

In addition to serving as an example of a personal debug log, I hope it can be useful as an introduction to debugging segfaults or other low-level bugs.

How to reproduce

I have three computers running Ubuntu 11.04. Mopidy revision 9c23949 consistently crashes with a segfault on one of them when I do the following:

Start Mopidy.
Connect with an MPD client, e.g. ncmpcpp.
Search for anything, e.g. foo.
Wait less than 10 seconds for the segfault to happen. It always happens directly after a log message from Mopidy’s Spotify backend stating that it is Updating metadata.

Rule out the obvious

I check that I’m actually using the latest versions of the important pieces of software, not just some old version I installed by hand and forgot about.

I try uninstalling pyspotify v1.1+mopidy20110405 installed from my own Debian package, and install the latest revision (c6e2a02) from Git instead. Still the same error.

Nobody has reported the same problem, even though both other developers and users should have been on the same mix of versions for the last couple of months.

Digging in

Luckily I can consistently reproduce the segfault in less than 20 seconds of work. I’m not too familiar with debugging C programs and especially the infamous segfaults, but I’m taking on this fight.

I expect the problem to be in pyspotify, as the rest of the code is either pure Python or more well-tested and broadly used libraries. Also the segfault consistently happens directly after a log message from Mopidy’s Spotify backend.

Antoine Pierlot-Garcin, the new main contributor to pyspotify, says to rebuild pyspotify with CFLAGS="-g -O0". -g will include debug information that gdb will understand in the resulting binary. -O0 will override the -O2 default of the pyspotify build system, and turn off any optimizations to ease the debugging.

Googling “debugging segfaults” yields a nice howto on debugging segfaults.

In summary: Get a core dump, inspect it with gdb.

Possible causes: “There are four common mistakes that lead to segmentation faults: dereferencing NULL, dereferencing an uninitialized pointer, dereferencing a pointer that has been freed (or deleted, in C++) or that has gone out of scope (in the case of arrays declared in functions), and writing off the end of an array.”

Rerunning Mopidy nothing more interesting happens, except the usual segfault. No core dump is produced.

Hum, how to get a core dump? I’ve done it before, but I can’t remember. Google helps.

I try ulimit -c 50000 to get a core dump of maximum 50MB, and then reproduce the segfault.

Run python mopidy -v, connect with ncmpcpp, search for foo, wait less than 10 seconds. Kaboom:

DEBUG    2011-05-23 20:07:02,030 [7659:SpotifySMThread] mopidy.backends.spotify.session_manager
  Metadata updated
  Segmentation fault (core dumped)

Loads the core dump up in gdb:

gdb /usr/bin/python core

gdb complains that the core dump is truncated, and that is should be approximately 180MB.

man bash and search for ulimit. Aha. I can specify unlimited. But I’m not allowed to? Restart shell, run ulimit -c unlimited again. Works.

Rerun Mopidy to get an untruncated core dump.

Analyzing the core dump

Yay! gdb loads debug symbols from all the linked libraries that got them available. Notably, the proprietary libspotify library does not include debug symbols. So what does gdb say?

Core was generated by `python mopidy -v'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fef50dae5f0 in sp_album_add_ref () from /usr/lib/libspotify.so.7
(gdb)

Verifying a hypothesis

According to the libspotify docs for the album subsystem sp_album_add_ref takes an sp_album struct and increases the reference count of the album. Looking back at the list of reasons for segfaults from the segfault debugging howto, it may sound as we increase the reference count of NULL, which obviously isn’t good. Let’s see if that hypothesis is correct…

Looking in the howto for ways to proceed, backtrace reveals the events happening just before the segfault:

(gdb) backtrace
#0  0x00007fef50dae5f0 in sp_album_add_ref () from /usr/lib/libspotify.so.7
#1  0x00007fef51048801 in Track_album (self=0x7fef4d155a50) at src/track.c:72
#2  0x00000000004970ef in call_function (f=<value optimized out>, throwflag=<value optimized out>) at ../Python/ceval.c:3997
#3  PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at ../Python/ceval.c:2666

Moving one step up the stack, we get to the pyspotify code:

(gdb) up
#1  0x00007fef51048801 in Track_album (self=0x7fef4d155a50) at src/track.c:72
72	    sp_album_add_ref(album);

Our guess was that we’re increasing the reference count on an album that is null. Let’s see what album actually was…

(gdb) print album
$1 = (sp_album *) 0x0

A pointer to an sp_album struct at the address 0x0, also known as NULL. Hypothesis confirmed.

Finding a solution

Let’s take a look at the pyspotify code in question:

static PyObject *Track_album(Track *self) {
    sp_album *album;

    album = sp_track_album(self->_track);
    Album *a = (Album *)PyObject_CallObject((PyObject *)&AlbumType, NULL);
    sp_album_add_ref(album);
    a->_album = album;
    return (PyObject *)a;
}

We create a pointer to a sp_album struct. Given a Spotify track, we request a reference to the related album, and assign the result to our pointer. Then we create an Album Python object, before we increase the reference count on the sp_album and give the reference to the Python object. Finally, the Python object is returned.

Lets try just returning NULL if we get NULL from sp_track_album:

diff --git a/src/track.c b/src/track.c
index f0d5956..3284029 100644
--- a/src/track.c
+++ b/src/track.c
@@ -68,6 +68,9 @@ static PyObject *Track_album(Track *self) {
     sp_album *album;
 
     album = sp_track_album(self->_track);
+    if (album == NULL) {
+        return NULL;
+    }
     Album *a = (Album *)PyObject_CallObject((PyObject *)&AlbumType, NULL);
     sp_album_add_ref(album);
     a->_album = album;

If we rebuild pyspotify and try to reproducing the segfault, we now get a familiar Python traceback instead:

/usr/lib/python2.7/threading.py:828: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
  return _active[_get_ident()]
ERROR    2011-05-23 22:33:05,214 [10571:SpotifySMThread] mopidy.utils.process
  error return without exception set
Traceback (most recent call last):
  File "/home/jodal/dev/mopidy/mopidy/utils/process.py", line 21, in run
    self.run_inside_try()
  File "/home/jodal/dev/mopidy/mopidy/backends/spotify/session_manager.py", line 40, in run_inside_try
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/pyspotify-1.1-py2.7-linux-x86_64.egg/spotify/manager.py", line 48, in connect
    self.loop(sess) # returns on disconnect
  File "/usr/local/lib/python2.7/dist-packages/pyspotify-1.1-py2.7-linux-x86_64.egg/spotify/manager.py", line 58, in loop
    self.timer = threading.Timer(timeout/1000.0, self.awoken.set)
  File "/usr/lib/python2.7/threading.py", line 731, in Timer
    return _Timer(*args, **kwargs)
  File "/usr/lib/python2.7/threading.py", line 742, in __init__
    Thread.__init__(self)
  File "/usr/lib/python2.7/threading.py", line 446, in __init__
    self.__daemonic = self._set_daemon()
  File "/usr/lib/python2.7/threading.py", line 470, in _set_daemon
    return current_thread().daemon
  File "/usr/lib/python2.7/threading.py", line 828, in currentThread
    return _active[_get_ident()]
  File "/home/jodal/dev/mopidy/mopidy/backends/spotify/session_manager.py", line 73, in metadata_updated
    self.refresh_stored_playlists()
  File "/home/jodal/dev/mopidy/mopidy/backends/spotify/session_manager.py", line 131, in refresh_stored_playlists
    SpotifyTranslator.to_mopidy_playlist(spotify_playlist))
  File "/home/jodal/dev/mopidy/mopidy/backends/spotify/translator.py", line 62, in to_mopidy_playlist
    if str(Link.from_track(t, 0))],
  File "/home/jodal/dev/mopidy/mopidy/backends/spotify/translator.py", line 34, in to_mopidy_track
    if dt.MINYEAR <= int(spotify_track.album().year()) <= dt.MAXYEAR:
SystemError: error return without exception set

Note that we get a SystemError and not e.g. AttributeError: 'NoneType' object has no attribute 'year' which we would expect if album() returned None. This is because we return NULL from Track_album, which Python considers an “error return”, without setting an exception code first. If we want to return None on failure instead of throwing an exception, we can change track.c as follows:

diff --git a/src/track.c b/src/track.c
index f0d5956..82897de 100644
--- a/src/track.c
+++ b/src/track.c
@@ -68,6 +68,8 @@ static PyObject *Track_album(Track *self) {
     sp_album *album;
 
     album = sp_track_album(self->_track);
+    if (!album)
+        Py_RETURN_NONE;
     Album *a = (Album *)PyObject_CallObject((PyObject *)&AlbumType, NULL);
     sp_album_add_ref(album);
     a->_album = album;

Rebuilding pyspotify again, and reproducing the error, we now get the expected Python error which we can handle nicely:

File "/home/jodal/dev/mopidy/mopidy/backends/spotify/session_manager.py", line 73, in metadata_updated
    self.refresh_stored_playlists()
  File "/home/jodal/dev/mopidy/mopidy/backends/spotify/session_manager.py", line 131, in refresh_stored_playlists
    SpotifyTranslator.to_mopidy_playlist(spotify_playlist))
  File "/home/jodal/dev/mopidy/mopidy/backends/spotify/translator.py", line 62, in to_mopidy_playlist
    if str(Link.from_track(t, 0))],
  File "/home/jodal/dev/mopidy/mopidy/backends/spotify/translator.py", line 34, in to_mopidy_track
    if dt.MINYEAR <= int(spotify_track.album().year()) <= dt.MAXYEAR:
AttributeError: 'NoneType' object has no attribute 'year'

By this point, I consider the bug squashed and ready to be reported.

The only part left is to decide how to best fix it. E.g. if album() should return None, throw an exception, or maybe return an empty album object. That’s another story.

FizzBuzz in Haskell

2011-03-31T00:00:00+02:00

I’ve recently started reading the free Haskell tutorial Learn You A Haskell for Great Good! while commuting. So far I’ve enjoyed the tutorial, and I’ve had a couple moments where I’ve smiled to myself on the bus due to typeclasses, etc., which I guess isn’t quite normal behaviour among the general population ;-)

I just discovered that the tutorial is going to be released as a book in a couple of weeks. You should definitely pick it up if you’re interested in learning a different programming language.

Here’s my first shot at some Haskell code, implementing the FizzBuzz kata using guard statements:

fizzBuzz :: Int -> String
fizzBuzz x
    | x `rem` 15 == 0 = "FizzBuzz"
    | x `rem` 3 == 0  = "Fizz"
    | x `rem` 5 == 0  = "Buzz"
    | otherwise       = show x

If you save this code to a file named FizzBuzz.hs, you can load and run it the interactive Haskell interpreter:

$ ghci
Prelude> :l FizzBuzz
[1 of 1] Compiling Main             ( FizzBuzz.hs, interpreted )
Ok, modules loaded: Main.

Now the file is loaded, compiled, and the fizzBuzz function is available in my current namespace. I use a list comprehension and a range to call fizzBuzz 20 times over the integers from 1 to 20, collecting the returned strings in a list:

*Main> [fizzBuzz x | x <- [1..20]]
["1","2","Fizz","4","Buzz","Fizz","7","8","Fizz","Buzz",
"11","Fizz","13","14","FizzBuzz","16","17","Fizz","19","Buzz"]

Updated 2011-04-10: Removed special case for 0, as there is no such rule in FizzBuzz.

Pykka, and porting Pykka to Python 3

2011-03-29T00:00:00+02:00

This is somewhat of an introduction to Pykka, a Python library implementing the actor model, which I’ve been working on now and then since November. And it’s still just 300 lines of code!

The goal of the actor model is to make it easier to develop concurrent programs by removing all shared state and use messaging for all communication. Removing shared state and doing lots of messaging doesn’t make anything easier by itself, but it avoids common problems in concurrent programs like proper lock usage and data corruption (due to the absence of proper lock usage). Also, I believe the actor model makes it easier to reason about concurrent programs, which is a nice property since software development often feels like 10% development and 91% debugging.

The goal of Pykka is to implement the actor model for Python as well as building useful concurrency abstractions on top of the actor model, most notably Pykka’s actor proxies. Actor proxies let you call the actor’s methods and read/write to the actor’s public attributes through a regular API, as opposed to passing messages using the usual send_one_way(message) and send_request_reply(message, block, timeout) methods. Accessing the actor’s public attributes like this may sound like the reintroduction of shared state and not in line with the actor model, but the actor proxy has no more access to the actor than anyone with a regular reference to the running actor has: it can send messages to the actor and patiently wait for an answer. The request for reading or writing to the attribute is processed like any other message to the actor: one at the time. The only difference between a regular object API and the API of the actor proxy is that all attribute reads and method calls on the proxy returns futures. You can either just get() the future right away, to block until the result is available and effectively serialize the execution your program, or you can pass the future around your program, delaying get() until the moment where you really need the future’s encapsulated value.

Currently Pykka has two actor implementations. One is based on threads, with one thread per actor. The other is based on gevent, with one lightweight greenlet per actor. In the gevent variant, all actors share a single thread, but are scheduled by a libevent loop. libevent is a cross-platform library wrapping whatever event mechanism that is the most efficient one available on your platform. libevent is also used by the popular Node.js framework. The big disadvantage of the gevent actors are that they don’t play well with non-gevent threads. Sometimes you can’t convert all threads in your application to actors, especially if those threads are embedded in support libraries you use, like GStreamer and libspotify, as is the case for my other current project, the Mopidy music server. For those cases, the threading-based actors are there to help you.

For some examples and more detailed descriptions and documentation, check out Pykka’s documentation.

Porting to Python 3

gevent isn’t available under Python 3 yet, but it seems like a port of gevent to Python 3 is being worked on. The upcoming 0.12 release of Pykka brings Python 3 support to the thread-based actor implementation.

Porting Pykka to Python 3 was no large feat:

Pykka is a quite new project, mainly made to scatch my own itch (aren’t all projects?). I know some people have been playing with it, but I’m not aware of anybody other than myself using it for a “real” project yet. Thus, I’m quite lenient about changing the API, at least in minor ways. The current goal is to make the library strike a nice balance between being useful and consistent. If I see ways to improve the API, I mostly just do it, but make sure to increase the version number in the next release. That’s how I got to 0.12 in four months. ;-) To keep overhead low, I’ve not yet started to maintain a changelog or migration documentation outside the git log. If you start using Pykka for anything more than an hour of play, please notify me, and I’ll straighten up. My point is, not having to maintain backwards compatability simplifies the world greatly. That said, I don’t think I’ve changed any of Pykka’s API to add Python 3 support.

As my main/original target for Pykka was to use it for Mopidy, which requires 2.6+, Pykka also requires Python 2.6+. 2.6 brings e.g. with statements, which are nice when working with locks. All the three latest Ubuntu releases defaults to 2.6, and so does OS X as of Snow Leopard. (Heck, some Linux distributions like Arch Linux are already using Python 3 as the default.) If you increase the requirement to Python 2.7 (a bit early yet, I think) you can even use lots of the new modules from Python 3 which has been backported to Python 2.x. Not having to support older Python versions makes it easier to write idiomatic Python code that will work mostly unchanged on Python 3. Keep the range of supported versions as narrow as possible, gradually getting rid of support for older Python versions, while not excluding users of any recent OS distribution.

Given this setting, three things have been really helpful for porting to and supporting a project on Python 3:

Good test coverage. Pykka is currently at 100% line coverage, and I believe the branch coverage is quite all right too. 1.2 seconds after running the tests I’m confident that everything works. That confidence is a great feeling. The tools I’ve used is unittest (not unittest2, yet), the nosetests test runner, and the nose coverage plugin.
Have all the Python version you target available on your system. I recently upgraded to the alpha version of Ubuntu 11.04, which works nicely so far. Ubuntu 11.04 changes the default Python version from 2.6.6 to 2.7.1, but more importantly: it brings Python 3.2 to the table. In other words Python 2.6, 2.7, 3.1, and 3.2 on a single system is just sudo apt-get install python-all python3-all away.
A test runner which makes testing on multiple Python versions a piece of cake: tox. Thanks to tox, it is actually 6 characters less effort to run the tests on four Python versions instead of running them one Python version (tox vs nosetests). Of course, it takes more than four times as long to run, but when the time for a single Python version is 1.2 seconds, I’ll survive the wait, and I’ll even run the tests on all targeted versions before every commit.

Porting to Python 3 is not just a one time effort – an effort which probably is quite small for many idiomatic and well-tested Python projects – but also an ongoing effort. There is little gain in porting if your going to break the Python 3 support with your next commit. Having confidence in your tests and a couple of nice tools is all you need to support multiple versions in parallel.

If you’re not writing tests, I’m sorry. I can’t help you.

Update 2011-03-30: Pykka 0.12 has now been released.

multiprocessing.Connection is not free

2011-03-20T00:00:00+01:00

I’ve previously written about how to wrap multiprocessing.Connection objects in a class that makes Connection objects picklable, and thus transferable over another Connection. That approach is still valid, but don’t use it–or Connection objects at all–for lots of use-once-and-throw-away connections.

For Pykka 0.9 and 0.10 I used wrapped Connection objects for implementing futures. It worked great until I tried using it for my ongoing rewrite of Mopidy to use Pykka actors. As soon as I had updated the majority of Mopidy’s tests to use actors, the test suite started failing after a given number of tests with the error “too many open files”. This error is due to multiprocessing using UNIX sockets in /tmp to implement it’s connections. The reason for backing connections with sockets is to make them work between multiple processes, a core feature of multiprocessing, but not a feature that I need.

In Pykka 0.11 I’ve rewritten the ThreadingFuture implementation from using multiprocessing.Connection to using Queue.Queue (or queue.Queue if your are using Python 3.x). As Queue.Queue use thread’s shared memory and locks instead of sockets on disk, it will not hit the “too many open files” problem, and it should also be a bit faster: A given test class in Mopidy took 2.7-2.9 seconds to run using Pykka 0.10 and multiprocessing.Connection. The same test class took 1.8-1.9 seconds using Pykka 0.11 and Queue.Queue. That’s an improvement of about one third, and a nice side effect of fixing the “too many open files” issue.

Traits in Python using multiple inheritance

2011-03-15T00:00:00+01:00

As you probably know, if a class needs to handle two mostly unrelated concerns, the class should probably be split into two. This way, your will achieve higher cohesion) in your code, which is generally considered a good thing. Though, there are times where you need to address two mostly unrelated concerns in a single class.

The Scala programming language has a concept they call traits. Scala traits can be compared to Java interfaces, except that Scala traits may be partially or fully implemented, like abstract classes. As with Java interfaces, you can mixin multiple traits in a class. If any of the traits has colliding method signatures, the last trait mixed in overrides the earlier ones. Just as with regular inheritance and interfaces, the class must implement any methods that has not been implemented by the traits, and may override any methods that already has been implemented.

trait Similarity {
  def isSimilar(x: Any): Boolean
  def isNotSimilar(x: Any): Boolean = !isSimilar(x)
}

class Point(xc: Int, yc: Int) extends Similarity {
  var x: Int = xc
  var y: Int = yc
  def isSimilar(obj: Any) =
    obj.isInstanceOf[Point] &&
    obj.asInstanceOf[Point].x == x
}

Traits as a concept for separating reusable parts of your code is not new. Like most good ideas, someone has been thinking about them previously. According to Wikipedia) traits originated in the Self programming language from 1987. Today, multiple programming languages has variants of traits as native constructs in the language. One example being Scala traits, another being module mixins in Ruby.

Given almost 25 years of history, one should think traits should be well known by now and more widely used for achieving good and reusable code. My first meeting with traits wasn’t until during an otherwise unrelated course in late 2009 where Jim Coplien introduced me to the DCI architecture during lunch, and then again a bit later when I picked up the Scala language. I ask myself: Why didn’t I learn stuff like this in university?

Example: Mixing in the actor life cycle in Mopidy

Back to Python. I’m currently introducing Pykka’s actor model implementation in the Mopidy music server as a replacement for Mopidy’s existing thread management and inter-thread communication code. I’m replacing untested application wiring–some reused throughout the application, some custom for specific problems–with a fully tested library. Using the actor model I apply the same simple semantics to all concurrent components in the application, making it simpler to reason about, at least compared to having custom solutions all over the place. As an added bonus, the current git diff --stat is at about -700/+300, leaving 400 fewer lines of code to maintain.

Mopidy has several extension points where one can add new components such as music library backends, audio outputs, audio mixers, and frontend servers. When you implement an extension, you must subclass the corresponding base class, e.g. the BaseOutput class, which defines and documents the API that the extension must implement for it to be usable by the rest of the system.

Many of these components interface with external libraries or systems, but we can’t block the entire system while waiting for these interactions to complete. We want to be able to respond to requests from MPD clients while we adjust the volume on the NAD amplifier, or queue the next block of audio data for playback. Thus, the extensions need to either run in its own thread, or to dispatch work to a worker thread.

Clearly, the components got two separate sets of concerns. First of all, they have some application logic which implements the API. E.g. methods like play_uri() and get_volume(). Secondly, the components got a life cycle, consiting of methods like start() and stop().

An example from Mopidy is the Last.fm scrobbler, which previously was split in two; a main class which implements the frontend API, and a worker class, which is a thread which contains most of the application logic.

import multiprocessing

from mopidy.frontends.base import BaseFrontend
from mopidy.utils.process import BaseThread

class LastfmFrontend(BaseFrontend):
    def __init__(self, *args, **kwargs):
        super(LastfmFrontend, self).__init__(*args, **kwargs)
        (self.connection, other_end) = multiprocessing.Pipe()
        self.thread = LastfmFrontendThread(self.core_queue, other_end)

    def start(self):
        self.thread.start()

    def destroy(self):
        self.thread.destroy()

    def process_message(self, message):
        if self.thread.is_alive():
            self.connection.send(message)

class LastfmFrontendThread(BaseThread):
    def __init__(self, core_queue, connection):
        # ...

    def run_inside_try(self):
        # ...

    def setup(self):
        # ...

    def process_message(self, message):
        # ...

    def started_playing(self, track):
        # ...

    def stopped_playing(self, track, stop_position):
        # ...

After refactoring the code is down to a single class just implementing core application logic, which no library can replace. The trick? Using multiple inheritance to mix in the actor life cycle.

from pykka.actor import ThreadingActor

from mopidy.frontends.base import BaseFrontend

class LastfmFrontend(ThreadingActor, BaseFrontend):
    def __init__(self):
        # ...

    def post_start(self):
        # ...

    def react(self, message):
        # ...

    def started_playing(self, track):
        # ...

    def stopped_playing(self, track, stop_position):
        # ...

I think this is a beautiful way of separating concerns that needs to be part of the same class. I also think the code reads well:

The above example is an frontend extension, which has the trait of also being an actor. It has some application logic, but also happens to have an actor life cycle.

Disclaimer: Before refactoring all your code to use multiple inheritance in Python, get familiar with Python’s Method Resolution Order (MRO) and the semantics of super() when using multiple inheritance.

Pickling multiprocessing Connection objects

2011-03-06T00:00:00+01:00

For safe message-based communication between threads and processes in Python, I tend to use multiprocessing’s Queue and Pipe. A pattern often seen is using a queue for sending messages from multiple producers to a single consumer.

When a producer wants a response to its message, I create a Pipe and piggy-back one end of the Pipe (a Connection object) to the message. I use Python dicts as messages, and use the string “reply_to” as the dictionary key for the connection objects.

When the queue consumer processes a message, it doesn’t know who the sender is or how to reach him. Though, if the message has an attached Connection object, the consumer can–almost magically–respond to the sender, across thread and process boundaries.

All good? Nope.

Any message sent through the queues and pipes must be serializable, or picklable as we say in Pythonesque. The multiprocessing.Connection objects can be serialized, but not unserialized, which means that you will not see an exception when you create your message, but some time later in the consumer that tries to respond. The exception does not tell you much, unless you’ve seen it before:

Traceback (most recent call last):
  File "pipetest2.py", line 10, in <module>;
    print c1.recv();
TypeError: function takes at least 1 argument (0 given)

This has been a known bug in Python for two years. Googling the exception leads to StackOverflow question asking for a workaround.

I’ve usually added a version of the workaround to some util package in my projects; one function for pickling a connection, and one function for unpickling a connection. In my code I’ve been forced to manually pickle/unpickle Connection objects before putting them on a Queue or Pipe.

from multiprocessing.reduction import reduce_connection
import pickle

def pickle_connection(connection):
    return pickle.dumps(reduce_connection(connection))

def unpickle_connection(pickled_connection):
    (func, args) = pickle.loads(pickled_connection)
    return func(*args)

This works great most of the time, but not this time. In the Python actor model library Pykka I use Connection objects to implement futures for thread-based actors, similar to how I use gevent’s AsyncResult for gevent-based actors. When someone sets a value on the future, it is written to one end of a Pipe. When someone tries to read the future’s value, they block on the other end of the Pipe until there is something to get or a timeout is reached. The problem appeared when I tried to nest futures, which is likely to happen if an actor, in response to your message, returns a future result from another actor. I no longer have the opportunity to babysit every Connection object that goes into or comes out of another Connection. They need to be able to watch over themselves. As the Connection class is implemented in C and is rather closed to changes, my solution was to wrap the Connection objects:

import multiprocessing.reduction

class ConnectionWrapper(object):
    """
    Wrapper for :class:`multiprocessing.Connection` objects to make them
    picklable.
    """

    def __init__(self, connection):
        self._connection = connection

    def __reduce__(self):
        (conn_func, conn_args) = multiprocessing.reduction.reduce_connection(
            self._connection)
        wrapper_func = _ConnectionWrapperRebuilder(conn_func)
        return (wrapper_func, conn_args)

    def __getattr__(self, name):
        return getattr(self._connection, name)


class _ConnectionWrapperRebuilder(object):
    """
    Internal class used by :class:`ConnectionWrapper` to rewrap
    :class:`multiprocessing.Connection` objects when they are depickled.

    A function defined inside :meth:`ConnectionWrapper.__reduce__` which takes
    :attr:`conn_func` from its scope cannot be used, as functions must be
    defined at the module's top level to be picklable.
    """

    def __init__(self, inner_func):
        self._inner_func = inner_func

    def __call__(self, *args):
        connection = self._inner_func(*args)
        return _ConnectionWrapper(connection)

The ConnectionWrapper class simply implements __reduce__ on the wrapped Connection object using multiprocessing’s own reduce_connection function. To work like a real Connection object, it dispatches any attribute access to the wrapped connection by implementing __getattr__.

To make sure the connection remains wrapped even after a trip through pickle.dumps() and pickle.loads(), _ConnectionWrapperBuilder is used for rebuilding the connection and rewrapping it on deserialization.

Given this wrapper, you can make your own Pipe function which creates a new pipe and wraps the connection objects for you.

Hopefully this trick will be of help until the bug is fixed in Python.