Alexandre Bourget

Have you heard of EventSource ? It’s very cool.

Here is some Pylons code to implement some sweet HTML5 server-pushing-events to web clients.

The example is based off this post by Rick Waldron. It uses the HTML5 features in Chromium version 6. Follow on his post for install instructions.

So in a random controller, dump this code:

import time
class EventsourceController(BaseController):

    def page(self):
        """Serve a fully fledged HTML page"""
        return '<script src="%s"></script>Loaded...' % url.current(action='js')

    def js(self):
        """Serve the JavaScript referenced by the previous action"""
        response.content_type = 'text/javascript'
        return """
document.addEventListener('DOMContentLoaded', function () {
  
  var eventSrc  = new EventSource('%s');
  
  eventSrc.addEventListener('open', function (event) {
    console.log("OPEN");
    console.log(event.type);
  });
  
  eventSrc.addEventListener('message', function (event) {
    console.log("RECEIVED");
    console.log(event.type);
    console.log(event.data);
  
  });
  console.log("Loaded...");

}, false);
""" % url.current(action='event_sender')

    def event_sender(self):
        """Implement a full-blown Event collector and dispatcher"""
        response.headers['content-type'] = 'text/event-stream'
        # Don't use this! It will append a charset=utf-8 and it won't work.
        #response.content_type = 'text/event-stream'
        response.status_int = 200
        def go():
            for x in range(10):
                msg = "data: %s %s\n\n" % (x, time.time())
                time.sleep(2)
                yield msg
        return go()

With a generator, you can have any code wait for some external events (other users’ input, UPS failures, mouse movements on the server) and send some events to your web client.

Make sure you take the ErrorHandler middleware out in config/middleware.py, because it buffers the output: it will keep the data from being flushed to the web user until all the content is finished processing. Another option would be to turn debugging off in your .ini file.

def make_app(global_conf, full_stack=True, static_files=True, **app_conf):
    ...
    # CUSTOM MIDDLEWARE HERE (filtered by error handling middlewares)

    if asbool(full_stack):
        # Handle Python exceptions
        #app = ErrorHandler(app, global_conf, **config['pylons.errorware'])
    ...

Point your browser to your http://.../{controller}/page with Chromium 6, and watch the events being printed to the control (Ctrl+Shift+J).

Now put some Gevent and Greenlet goodness in there, and you’ll have some real-time app server in no time!

Hello world,

Just a note to tell you that we, at Savoir-faire Linux now offer a Pylons Training Course both in French and in English. Follow the links directly on our website or see the pylonshq.com announcement.

We can give the course in our Montreal, Ottawa or Quebec City offices or anywhere you’re willing to fly us.

If you need help migrating from Pylons 0.9.7 (or previous) to the latest and greatest (version 1.0), we can accomodate with custom courses or consulting services.

Here is my presentation at the Montreal Python user group gathering on Decembre 9th 2009 at UQAM.

From a web point of view, if you want to stop a user from doing stupid things (like submitting a form before reviewing it, sending a heinous e-mail late at night, or clicking a button that launches a nuclear missile), you might want to do one one of three things:

• Pop-up a confirmation dialog
• Go to a second page, with yet another submit button asking for confirmation, with a summary of the submitted data
• Just let it go, but offer a timed “UNDO” or “CANCEL” link, like Gmail implemented for e-mails

Of course, the best of all three is the latter. The difference here is major. It’s the difference between pro-active annoyingness (pop-ups asking confirmation when you know very well you want to send the e-mail), or reactive flexibility. It also gives your user a great boost of confidence in your application, because the user can do stupid things and knows he’ll be able to fix them in a timely manner.

Now here’s how to do it in Pylons, using the just-released WebUndo lib. You can find it on PyPI, so just pip install WebUndo from your project.

We’re lucky Pylons runs as a single application with threads, since that’s what we’ll use to delay our operations and give the user a chance to cancel. PHP apps for instance, wouldn’t give us that flexibility, since everything vanishes once the request is sent. Take a look at the source code to know how it’s built.

Let’s consider this piece of Pylons controller:

class ThController(BaseController):
    def index(self):
        return render('th.mako')

    def save(self):
        open('/tmp/dump.txt', 'w').write(request.params.get('stuff'))
        return "Everything saved"

where th.mako contains a simple form with a text field named stuff. In this case, once you submit the form in th.mako, it’s done: the file is written to disk, and there’s no way you can cancel it.

Now let’s have a look at this version:

class ThController(BaseController):
    def index(self):
        return render('th.mako')

    def save(self):
        req = request
        def do_save():
            open('/tmp/dump.txt', 'w').write(req.params.get('stuff'))
        key = webundo.launch_cancelable_job(do_save, 5)
        return 'Saving, click to <a href="%s">cancel</a>' % h.url_for(controller='th', action='cancel', id=key)

    def cancel(self, id):
        if webundo.cancel_job(id):
            return "Thread cancelled successfully"
        else:
            return "Too late, thread finished already."

We added some pieces of webundo, the save() action returns a confirmation message, and a link to cancel the operation. We also added a def do_save() wrapper, a call to the launch_cancelable_job function and the passing on of the key. Notice also the req = request, used to make sure the request object gets closed in the do_save() function. The fun part here, is that the operation will not be carried out until the 5 seconds (second parameter to launch_cancelable_job) are elapsed. The last thing is obviously, the cancel action, which will deal with the cancelation.

Using this feature, even if you close your browser window after calling the save action, the do_save() function will be executed normally, only delayed by 5 seconds.

For the Pylons addicts who read that code and say “hey, that request object can’t traverse threads, it’s a StackedObjectProxy”! See the details and answer in my other blog post.

Of course, there are cases where you want a certain operation to be carried out directly (when calling save() for example) but you want some special code to be executed if someone click an undo button for example. The WebUndo lib provides you with another pair of functions to do just that: launch_undoable_job and undo_job.

Take a look at the documentation for full details. Happy undoing!

UPDATED March 20th: typos following feedback

I was working on a feature to implement Gmail’s “Undo” feature in Pylons (which I’ll present on this blog soon), and was faced with a challenge regarding the passage of variables (or functions) to a separate thread. The reason is that Pylons resolves the current request object according to the local thread. You won’t find the same request object in a different thread from which it originated.

I need to be able to resolve those objects (to get the actual object, not the proxy object) before spawning a new thread. Though, I don’t want to require the user to know about all this. He should only create a function that should work in the current thread context, and my code will make sure the good objects get passed on to the new thread.

In Python terms, I wanted to be able to modify the func_closure attribute of a function object (it’s called __closure__ from Python 2.6 onwards). I searched around, only to find that it’s really a read-only attribute, unless you want to do risky things.

So, if you do something like:

def a():
    b = 123
    def myclosure():
        print b
    return myclosure
myfunc = a()

the value of b gets referenced (as a cell object) in the myfunc.func_closure tuple. But there’s no way to change it. What I need is to tweak the myfunc function’s closure, and replace the internal value of b for something else. So when I’d call myfunc(), it actually prints something else than 123, let’s say 'mynewvalue'.

At first, I tried the mix-and-match suggested here, which looks like:

import new
def getitem(a):
    def retclosure():
        return a
    return retclosure
tmpfunc = getitem('mynewvalue')
cell = tmpfunc.func_closure[0]
rewrittenfunc = new.function(myfunc.func_code , myfunc.func_globals, myfunc.func_name, myfunc.func_defaults, (cell,))

Basically, it takes the code, globals, name and defaults from the myfunc function, and uses the cells from another function, mixing two different functions into one.

That’s great for a single reference in the closure, but it’s not very flexible. I want my code to go through all of the closed in objects, and verify is they are of a certain type (namely StackedObjectProxy), and resolve those that need local resolution. It should work with a random number of arguments.

Here is my solution:

def unproxy_closure(f):
    """Rewrite function with resolved closure references (StackedObjectProxy from current thread)"""
    import new
    def create_cell(obj):
        return (lambda: obj).func_closure[0]
    return new.function(func.func_code , func.func_globals, func.func_name,
                        func.func_defaults,
                        tuple(create_cell(cell.cell_contents._current_obj())
                                  if hasattr(cell.cell_contents, '_current_obj')
                                       and
                                     callable(cell.cell_contents._current_obj)
                                  else cell
                              for cell in func.func_closure))

First, there’s that create_cell function, which will return a cell object. It seems to be the only way to create those types of objects: creating a closure and stealing it’s cell object.

Using the new module, it creates a new function borrowing code (globals, defaults, name, etc.) from the first function, and it goes through the cells of the passed-in closure to modify them if necessary. That way, we keep the actual object in the closure, not the proxy object that won’t resolve correctly in the other thread I’m going to launch.

Voilà! That opens the road to a very cool feature I’ll talk about shortly. Comments are welcome.

UPDATE 18h22: rephrased a little bit clunky sentences

UPDATE Match 18th 2010: Added link to what I’ve talk about shortly after

UPDATE March 20th 2010: fixed code block, thanks to John

So this is the short story of my presentation at the Montreal Python user groups on january 25th 2010.

I’m preparing to give a talk on “Pylons, web development done right” (or something alike), at the Confoo conference in March. I wanted to test-drive my presentation, so I decided to jump in after seeing a couple of presentation over there - some of which were pretty so-so, thus pushing me to risk myself.

I wanted the presentation to be a performance (since this is my primary training, as a pianist), a lightning bolt filled with impressive demonstrations, speedy typing, cool technologies using a sweet programming language. I wanted to show off Pylons (http://www.pylonshq.com) and a bunch of WSGI sweetnesses (memento, WPHP, CleverCSS, Beaker, Routes, Mako, SQLAlchemy, SqlSoup, etc.)

So I packed my Karmic system with a Squid3 proxy server, to fake having an Internet connection (while everyone in the room was struggling with the bad/non-existant Wi-Fi connection). I wrote some Python scripts that would do all sorts of background mangling to accelerate the live demos* and organised the presentation in modules, so that I could dynamically add or remove pieces of the demonstration based on interests I sized in the audience.

Finally, the presentation was greatly appreciated if I believe the comments I have received. It was a very good experience for me and I received a gift, which is in fact the subject/object of this post: the book “Friends with benefits“.

Read on...

Tired of using paster serve –reload development.ini and waiting for your code to reload ?

memento allows you to dynamically reload your code – without restarting your whole server – or parts of your code.

Install it:

$ easy_install memento

To integrate with Pylons, follow these steps:

Go to your config/middleware.py, and add:

import memento

Then, a bit lower, replace:

def make_app(...):
    ...
    # The Pylons WSGI app
    app = PylonsApp()
    ...

by:

def make_app(...):
    ...
    # The Pylons WSGI app
    #app = PylonsApp()
    app = memento.Assassin('pylons.wsgiapp:PylonsApp()', ['yourpackage'])
    ...

The second parameter to Assassin() is a list of packages you want reloaded on each request. You probably want your whole package to be reloaded, or you can be more granular.

NOTE: don’t forget the () after the PylonsApp, because it has to call the object to get a WSGI app. You’ll get errors otherwise

The neat thing is I think you can disable that in real-time also, by changing the value of app.mode (’on’ = ‘on and ‘off’ = ‘off, as you’ve guessed), so I guess you’d like to keep a copy of that ‘app’ before you lose it’s reference lower in middleware.py.