PyCon Sprints Part 1: The realtime web with gevent, socket io, redis and django

TL;DR: Get the source here.

One of the major themes of PyCon was realtime web utilizing websockets or long polling and coroutines.

Websockets is a new feature in modern browsers which allows bi-directional communication between your server and the users web browser but it isn't a reliable way to do realtime communication since not everyone upgrades as quickly as us web developers would like and older browsers don't support it.

So you either have to fallback to long polling if they don't have websockets or just only support long polling since it works on all browsers. Longpolling is opening a connection to the server and keeping it open with javascript so that the server can send messages back to the client.

Luckily, the fine folks that developed Socket.io have done the legwork to do feature fallback and so it'll work on all browsers by testing for websockets, flash sockets, and then as a last resort long polling.

Page postbacks and refreshes are a thing of the past, the future of the web is realtime communication.

To show you how to achieve this we are going to build a quick Tic Tac Toe application using SocketIO, gevent, redis, and Django.

Lets setup the development environment first:

$ virtualenv --no-site-packages tictactoe
$ cd tictactoe/
$ source bin/activate
$ mkdir src
$ cd src/

Before you continue make sure you install libevent and redis-server using your operating systems package manager.

Then you want to create a pip requirements.txt file that will provide you all the packages you want, the version of redis is very important since they didn't have a publish/subscribe model until 2.2.2:

django==1.3
gevent
redis==2.2.2
simplejson
mock
hg+https://bitbucket.org/Jeffrey/gevent-websocket#egg=gevent-websocket
hg+https://bitbucket.org/Jeffrey/gevent-socketio#egg=gevent-socketio

Then use pip to install everything:

$ pip install -r requirements.txt

Now we need to create the django application. We are going to organize our apps into the apps/ folder, this is a personal preference of mine but I recommend you do the same:

$ django-admin.py startproject tictactoe
$ cd tictactoe/
$ mkdir apps
$ django-admin.py startapp core
$ mv core/ apps/

Open up settings.py and add 'core' to INSTALLED_APPS and create utility attribute we'll use to figure out the directory we are working out of, you should also configure your database settings and at this time:

import os
PROJECT_ROOT = os.path.dirname(__file__)

Then open up manage.py and add a this sys.path line so that it knows we are storing our apps in the apps/ folder.

import sys, os
sys.path.insert(0, os.path.join(settings.PROJECT_ROOT, "apps"))

Now that we've got a base development system configured we can start setting up our environment for doing realtime development. The first step is to setup a WSGI webserver that supports coroutines, since the standard django development server is single threaded. We will use gevent to monkey patch the standard library to get async support:

Create a file called run.py that will replace manage.py runserver:

#!/usr/bin/env python
from gevent import monkey
from socketio import SocketIOServer
import django.core.handlers.wsgi
import os
import sys

# import the django settings file to get PROJECT_ROOT
import settings

# use gevent to patch the standard lib to get async support
monkey.patch_all()

PORT = 9000
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
application = django.core.handlers.wsgi.WSGIHandler()

# add our project directory to the path
sys.path.insert(0, os.path.join(settings.PROJECT_ROOT, "../"))

# add our apps directory to the path 
sys.path.insert(0, os.path.join(settings.PROJECT_ROOT, "apps"))

if __name__ == '__main__':
    # Launch the redis server in the background
    os.popen('redis-server &')
    print('Listening on http://127.0.0.1:%s and on port 843 (flash policy server)' % PORT)
    SocketIOServer(('', PORT), application, resource="socket.io").serve_forever()

Now that we have a server that can support the asynchronous calls that we'll be making we can start designing our application.

Lets first setup the apps/core/models.py to describe the database tables we'll need:

from django.db import models
from django.contrib.auth.models import User
import pickle

class Game(models.Model):
    player1 = models.ForeignKey(User, related_name='player1_set')
    player2 = models.ForeignKey(User, related_name='player2_set')
    last_move = models.CharField(max_length=1, null=True, blank=True)
    board = models.CharField(max_length=100,
            default=str(pickle.dumps([''] * 9)))

    def __unicode__(self):
        board = pickle.loads(str(self.board))
        return '%s vs %s (%s)' % (self.player1.username,
                self.player2.username, board)

    def make_move(self, player, move):
        """
        player is X or O and move is a number 0-9
        """
        board = pickle.loads(self.board)
        board[move] = player
        self.board = pickle.dumps(board)
        self.last_move = player
        self.save()

    def get_valid_moves(self):
        """
        Returns a list of valid moves. A move can be passed to get_move_name to
        retrieve a human-readable name or to make_move/undo_move to play it.
        """
        board = pickle.loads(self.board)
        return [pos for pos in range(9) if board[pos] == '']

    def all_equal(self, list):
            """
            Returns True if all the elements in a list are equal, or if
            the list is empty. 
            """
            return not list or list == [list[0]] * len(list)

    def get_winner(self):
        """
        Determine if one player has won the game. Returns X, O, '' for Tie,
        or None
        """
        board = pickle.loads(self.board)

        winning_rows = [[0,1,2], [3,4,5], [6,7,8], # horizontal
                        [0,3,6], [1,4,7], [2,5,8], # vertical
                        [0,4,8], [2,4,6]]          # diagonal

        for row in winning_rows:
            if board[row[0]] != '' and self.all_equal([board[i] for i in row]):
                return board[row[0]]

        # No winner found, check for a tie
        if not self.get_valid_moves():
            return ''

        return None

Then open up apps/core/tests.py and create tests to make sure getting a winner and getting valid moves works properly, because code without tests is broken code!

from django.test import TestCase
from core.models import Game
from django.contrib.auth.models import User
import pickle

class TestTicTacToeBoard(TestCase):
    def setUp(self):
        self.player1 = User.objects.create(username='X')
        self.player2 = User.objects.create(username='O')

        self.player1.save()
        self.player2.save()

    def test_finds_winner(self):
        """
        Tests that getting a winner works properly
        """
        board = pickle.dumps(['X', 'X', 'X', '', '', '', '', '', ''])
        game = Game(player1=self.player1, player2=self.player2,
                board=board)
        winner = game.get_winner()

        self.assertEqual(winner, 'X')

    def test_doesnt_find_winner(self):
        board = pickle.dumps(['', '', '', '', '', '', '', '', ''])
        game = Game(player1=self.player1, player2=self.player2,
                board=board)
        winner = game.get_winner()

        self.assertEqual(winner, None)

    def test_find_tie(self):
        """
        Tests that you get a tie when the board is full
        """
        board = pickle.dumps(['X', 'X', 'O',
                              'O', 'O', 'X',
                              'X', 'O', 'X'])

        game = Game(player1=self.player1, player2=self.player2,
                board=board)
        winner = game.get_winner()

        self.assertEqual(winner, '')

    def test_make_move(self):
        """
        Tests that you can make moves 
        """
        board = pickle.dumps(['', '', '', '', '', '', '', '', ''])

        game = Game(player1=self.player1, player2=self.player2,
                board=board)

        game.make_move('X', 0)
        board = pickle.dumps(['X', '', '', '', '', '', '', '', ''])

        self.assertEqual(board, game.board)
        self.assertEqual('X', game.last_move)

    def test_gets_valid_moves(self):
        """
        Tests that getting valid moves works properly
        """
        board = pickle.dumps(['', '', '', '', '', '', '', '', ''])
        game = Game(player1=self.player1, player2=self.player2,
                board=board)
        moves = game.get_valid_moves()

        self.assertEqual(len(moves), 9)

    def test_doesnt_gets_valid_moves(self):
        """
        Tests that getting valid moves works properly
        """
        board = pickle.dumps(['X', 'X', 'X', 'X', 'X', 'X', 'X', 'X', 'X'])
        game = Game(player1=self.player1, player2=self.player2,
                board=board)
        moves = game.get_valid_moves()

        self.assertEqual(len(moves), 0)

Now that we have a tic tac toe board that is tested, we just need to create our views to allow the 2 users to play a game. We are going to need a view for viewing the game, a view for asynchronously making moves, and a view that our clients can keep an open socket to for the bi-directional communication.

Create the urls.py:

from django.conf.urls.defaults import (patterns, include, url, handler500,
        handler404)

# we'll use admin for creating users and games
from django.contrib import admin
admin.autodiscover()

from core.views import (
    create_move,
    view_game,
    socketio,
)
urlpatterns = patterns('',
    url(
        regex=r'^create_move/(?P<game_id>\d+)/$',
        view=create_move,
        name='create_move'
    ),
    url(
        regex=r'^view_game/(?P<game_id>\d+)/$',
        view=view_game,
        name='view_game'
    ),
    url(
        regex=r'^socket\.io',
        view=socketio,
        name='socketio'
    ),
    (r'^admin/', include(admin.site.urls)),
)

# Django 1.3 Features, allows us to serve static files easily
from django.contrib.staticfiles.urls import staticfiles_urlpatterns
urlpatterns += staticfiles_urlpatterns()

Then our views.py:

from django.views.decorators.http import require_http_methods
from django.contrib.auth.decorators import login_required
from django.shortcuts import get_object_or_404, render_to_response
from django.http import HttpResponse, Http404
from django.template import RequestContext
from django.conf import settings

from redis import Redis
from gevent.greenlet import Greenlet

from core.models import Game

REDIS_HOST = getattr(settings, 'REDIS_HOST', 'localhost')

def _sub_listener(socketio, chan):
    """
    This is the method that will block and listen
    for new messages to be published to redis, since
    we are using coroutines this method can block on
    listen() without interrupting the rest of the site
    """
        red = Redis(REDIS_HOST)
        red.subscribe(chan)

        for i in red.listen():
            socketio.send({'message': i})

def socketio(request):
    """
    This view will handle the 'subscribe' message
    from the client and spawn off greenlet coroutines
    to monitor messages on redis
    """
    socketio = request.environ['socketio']

    while True:
        message = socketio.recv()

        if len(message) == 1:
            message = message[0].split(':')

            if message[0] == 'subscribe':
                print 'spawning sub listener'
                g = Greenlet.spawn(_sub_listener, socketio, message[1])

    return HttpResponse()

@require_http_methods(["POST"])
@login_required
def create_move(request, game_id):
    """
    Creates a move for the logged in player on a specific game
    """
    game, player, opponent_id = _get_game_data(request, game_id)
    move = int(request.POST['move'])
    game.make_move(player, move)

    # Announce to the opponent that the current player made a move
    red = Redis(REDIS_HOST)

    _announce_player_moved(red, game_id, opponent_id, player, move),
    winner = game.get_winner()

    if winner:
        # Announce to current player and the opponent that the game is over
        _announce_game_over(red, game_id, request.user.id, player)
        _announce_game_over(red, game_id, opponent_id, player)

    return HttpResponse()

def view_game(request, game_id, template_name='core/view_game.html'):
    """
    Renders a tic tac toe board to be played for a specific game
    """
    game, player, opponent_id = _get_game_data(request, game_id)

    if game.last_move:
        current_player = 'X' if game.last_move == 'O' else 'O'
    else:
        current_player = 'X'

    board = game.get_board()
    winner = game.get_winner()

    context = { 'game_id': game_id,
                'board': board,
                'player': player,
                'current_player': current_player,
                'winner': winner,
                'game_over': False if winner == None else True
              }

    return render_to_response(template_name, context,
            context_instance=RequestContext(request))

def _announce_player_moved(red, game_id, to_user_id, player, move):
    """
    Publishes a message to redis to to_user_id that their opponent has
    completed a move
    """
    red.publish(to_user_id, ['opponent_moved', int(game_id), player, move])

def _announce_game_over(red, game_id, to_user_id, winner):
    """
    Publishes a message to redis to to_user_id that the game as finished
    """
    red.publish(to_user_id, ['game_over', int(game_id), winner])

def _get_game_data(request, game_id):
    """
        Grabs the game, current player, and its opponent
    """
    game = get_object_or_404(Game, pk=game_id)

    if game.player1 == request.user:
        player = 'X'
        opponent_id = game.player2.id
    elif game.player2 == request.user:
        player = 'O'
        opponent_id = game.player1.id
    else:
        raise Http404

    return game, player, opponent_id

The special part here is in the socketio view, we are launching greenlet coroutines that subscribe to a "channel" on the redis server.

Like we established before, it its not tested, it doesn't work. So we need to write more tests to verify that the views we just created work:

class TestGameViews(TestCase):
    def setUp(self):
        self.player1 = User.objects.create(username='X')
        self.player1.set_password('test')
        self.player2 = User.objects.create(username='O')

        self.player1.save()
        self.player2.save()

        self.game = Game(player1=self.player1, player2=self.player2)
        self.game.save()

        self.client = Client()
        self.client.login(username='X', password='test')

    def test_create_move_publishes_to_redis(self):
        """
        Tests that we are publishing to redis when we create moves
        """
        request = Mock(name='request')
        request.user = self.player1

        redis = Mock(name='redis')
        redis.publish = Mock()

        self.game.board = pickle.dumps(['X', '', '',
                                        '', 'X', '',
                                        '', '', ''])
        self.game.save()

        with patch('core.views.Redis') as mock_redis:
            mock_redis.return_value = redis

            move = 8
            player = 'X'
            response = self.client.post('/create_move/%d/' % self.game.id,
                        {
                            'move': move
                        }
                    )

            redis.publish.assert_called_with(self.player2.id,
                    ['game_over', self.game.id, player])

            _pop_last_call(redis.publish)

            redis.publish.assert_called_with(self.player1.id,
                    ['game_over', self.game.id, player])

            _pop_last_call(redis.publish)

            redis.publish.assert_called_with(self.player2.id,
                    ['opponent_moved', self.game.id, player, move])

    def test_winning_move_publishes_to_redis(self):
        """
        Tests that we are publishing to redis when we create moves
        """
        request = Mock(name='request')
        request.user = self.player1

        redis = Mock(name='redis')
        redis.publish = Mock()

        with patch('core.views.Redis') as mock_redis:
            mock_redis.return_value = redis

            move = 0
            player = 'X'
            response = self.client.post('/create_move/%d/' % self.game.id,
                        {
                            'move': move
                        }
                    )

            redis.publish.assert_called_once_with(self.player2.id,
                    ['opponent_moved', self.game.id, player, move])

    def test_create_move_makes_move(self):
        """
        Tests that we are creating moves in the db when we call create_move
        """
        redis = Mock(name='redis')
        redis.publish = Mock()

        with patch('core.views.Redis') as mock_redis:
            mock_redis.return_value = redis

            move = 0
            player = 'X'
            response = self.client.post('/create_move/%d/' % self.game.id,
                        {
                            'move': move
                        }
                    )

            game = Game.objects.get(pk=self.game.id)
            board = game.get_board()
            self.assertEqual(board[0], player)

    def test_make_move_wins(self):
        pass

def _pop_last_call(mock):
    if not mock.call_count:
        raise AssertionError('Cannot pop last call: call_count is 0')

    mock.call_args_list.pop()

    try:
        mock.call_args = mock.call_args_list[-1]
    except IndexError:
        mock.call_args = None
        mock.called = False

    mock.call_count -=1

There is a copy fancy things going on here, if you aren't familiar with Mock we are using it to tell our view whenever it tries to use the Redis class to use our special mocked object instead so we can record its actions. Then we have the _pop_last_call function that is just to work around the fact that Mock only records the last call of a function and we want to make sure it fired the publish 3 times.

Now that we've proven that our db model and our views are working with our tests, the final piece is to create a template and write the javascript to make the call backs!

We need to get back into settings.py to define our static file directories for loading the javascript:

STATICFILES_DIRS = (
    os.path.join(PROJECT_ROOT, 'static'),
)

Now create your static folder and put jquery 1.5.1 (jquery 1.5 has a bug in it) and socket.io.js in there.

Then we'll define the javascript we need to interact with the the socketio view we defined earlier in a file called game.js:

socket.connect();

socket.on("message", function(obj){
    alert(obj);
    if (obj.message.type == "message") {
        var data = eval(obj.message.data);

        if (data[1] == game_id) {
            if (data[0] == "game_over") {
                    winner = data[2];
                    game_over = true;

                    if (data[2] == "") {
                        SetMessage("Its a tie!");
                    }
                    else {
                        SetMessage("The winner is: " + data[2]);
                    }
                }
            else if (data[0] == "opponent_moved") {
                $('#cell' + data[3]).html(data[2]);
                SwapUser();
            }
        }
    }
});

function MakeMove(sender, move) {
    if (player == current_player && game_over == false) {
        if ($(sender).text().trim() == "") {
            $(sender).html(player);
            SwapUser();

            $.post(create_url, {'move': move},
                function(data) {
                    // successfully made a move
                }
            )
        }
    }
}

function SwapUser() {
    var computer = player == "X" ? "O" : "X";

    if (current_player == player) {
        current_player = computer;
        SetMessage("Your opponents turn!");
    } else {
        current_player = player;
        SetMessage("Your turn!");
    }
}

socket.send("subscribe:" + user_id);

function SetMessage(message) {
    $("#messages").html("<div>" + message + "</div>");
    $("#messages").show();
}

The import parts here are sending the subscribe message to the server so that it will start listening on that specific channel and then handling the message event from the socket so we can do something when the server sends us a new message.

To tie it all togethers lets create templates/core/view_game.html and a django.js file that allows us to call the webserver from javascript with CSRF protection:

django.js:

$('html').ajaxSend(function(event, xhr, settings) {
    function getCookie(name) {
        var cookieValue = null;
        if (document.cookie && document.cookie != '') {
            var cookies = document.cookie.split(';');
            for (var i = 0; i < cookies.length; i++) {
                var cookie = jQuery.trim(cookies[i]);
                // Does this cookie string begin with the name we want?
                if (cookie.substring(0, name.length + 1) == (name + '=')) {
                    cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
                    break;
                }
            }
        }
        return cookieValue;
    }
    if (!(/^http:.*/.test(settings.url) || /^https:.*/.test(settings.url))) {
        // Only send the token to relative URLs i.e. locally.
        xhr.setRequestHeader("X-CSRFToken", getCookie('csrftoken'));
    }
});

templates/core/view_game.html:

<html>
    <head>
        <title>TicTacToe</title>
        <link rel="stylesheet" type="text/css" href="{{ STATIC_URL }}style.css" />
        <script type="text/javascript" src="{{ STATIC_URL }}jquery-1.5.1.min.js"></script>
        <script type="text/javascript" src="{{ STATIC_URL }}django.js"></script>
        <script type="text/javascript" src="{{ STATIC_URL }}socket.io.js"></script>
    </head>
    <body>

    <script type="text/javascript">
        var player = "{{ player }}";
        var current_player = "{{ current_player }}";
        var game_id = {{ game_id }};
        var user_id = {{ request.user.id }};
        var game_over = {{ game_over|lower }};
        var create_url = "{% url create_move game_id %}";

        var socket = new io.Socket(null, {port: {{ request.environ.SERVER_PORT }}, rememberTransport: false});
    </script>
    <script type="text/javascript" src="{{ STATIC_URL }}game.js"></script>
    You are: {{ player }}
    <div id="messages">
        {% if winner == None %}
            {% if player == current_player %}
                Your turn
            {% else %}
                Your opponents turn
            {% endif %}
        {% else %}
            {% if winner == "" %}
                Its a tie!
            {% else %}
                {{ winner }} wins!
            {% endif %}
        {% endif %}
    </div>
    {% for row in board %}
        <div class="cell" id="cell{{ forloop.counter0 }}" onclick="MakeMove(this, '{{ forloop.counter0 }}')">
            {{ row }}
        </div>
        {% if forloop.counter|divisibleby:3 %}
            <br class="clear" />
        {% endif %}
    {% endfor %}
    </body>
</html>

and then static/style.css file:

.cell {
    border: solid 1px #000;
    width: 100px;
    height: 100px;
    float: left;
    text-align: center;
    font-size: 70px;
}
.clear {
    clear: both;
}

and you now have a realtime tic tac toe application.

To play the game you just have to launch the redis server and your WSGI webserver:

$ redis-server
$ python run.py

Watch a video of it in action here