Alexandre Bourget

Have you heard of EventSource ? It’s very cool.

Here is some Pylons code to implement some sweet HTML5 server-pushing-events to web clients.

The example is based off this post by Rick Waldron. It uses the HTML5 features in Chromium version 6. Follow on his post for install instructions.

So in a random controller, dump this code:

import time
class EventsourceController(BaseController):

    def page(self):
        """Serve a fully fledged HTML page"""
        return '<script src="%s"></script>Loaded...' % url.current(action='js')

    def js(self):
        """Serve the JavaScript referenced by the previous action"""
        response.content_type = 'text/javascript'
        return """
document.addEventListener('DOMContentLoaded', function () {
  
  var eventSrc  = new EventSource('%s');
  
  eventSrc.addEventListener('open', function (event) {
    console.log("OPEN");
    console.log(event.type);
  });
  
  eventSrc.addEventListener('message', function (event) {
    console.log("RECEIVED");
    console.log(event.type);
    console.log(event.data);
  
  });
  console.log("Loaded...");

}, false);
""" % url.current(action='event_sender')

    def event_sender(self):
        """Implement a full-blown Event collector and dispatcher"""
        response.headers['content-type'] = 'text/event-stream'
        # Don't use this! It will append a charset=utf-8 and it won't work.
        #response.content_type = 'text/event-stream'
        response.status_int = 200
        def go():
            for x in range(10):
                msg = "data: %s %s\n\n" % (x, time.time())
                time.sleep(2)
                yield msg
        return go()

With a generator, you can have any code wait for some external events (other users’ input, UPS failures, mouse movements on the server) and send some events to your web client.

Make sure you take the ErrorHandler middleware out in config/middleware.py, because it buffers the output: it will keep the data from being flushed to the web user until all the content is finished processing. Another option would be to turn debugging off in your .ini file.

def make_app(global_conf, full_stack=True, static_files=True, **app_conf):
    ...
    # CUSTOM MIDDLEWARE HERE (filtered by error handling middlewares)

    if asbool(full_stack):
        # Handle Python exceptions
        #app = ErrorHandler(app, global_conf, **config['pylons.errorware'])
    ...

Point your browser to your http://.../{controller}/page with Chromium 6, and watch the events being printed to the control (Ctrl+Shift+J).

Now put some Gevent and Greenlet goodness in there, and you’ll have some real-time app server in no time!

Hello world,

Just a note to tell you that we, at Savoir-faire Linux now offer a Pylons Training Course both in French and in English. Follow the links directly on our website or see the pylonshq.com announcement.

We can give the course in our Montreal, Ottawa or Quebec City offices or anywhere you’re willing to fly us.

If you need help migrating from Pylons 0.9.7 (or previous) to the latest and greatest (version 1.0), we can accomodate with custom courses or consulting services.

Here is my presentation at the Montreal Python user group gathering on Decembre 9th 2009 at UQAM.

I wanted to create still frames to be used in Cinelerra, working with SD material, but Gimp and Inkscape produce images that are 1:1 on pixel aspect ratio. Video frames however, use 4:3 or 16:9 pixel aspect ratio. I’ve searched and tried a lot of things before sticking to this recipe:

If you’re working in 4:3, open a new image of 640x480 pixels in either Inkscape or the Gimp. If you’re working in the Gimp, I suggest you at least double the precision (so 1280x960), and we’ll resize it at the end, no matter what. In Inkscape, you can just increase the output resolution when exporting. If you’re working in 16:9, have your image 865x480 pixels WxH.

Work with your favourite tool, viewing the correct output. Then export, or save at a higher resolution.

Last thing, from the ImageMagick suite, run:

convert filename.png -resize 720x480! outputfilename.png

This should do the job. Now import it in Cinelerra. You’ll note the bad resizing algorithm used to display the images in the viewer and compositor. These artifacts should go away when you export to some file, and view in a real video player.

From a web point of view, if you want to stop a user from doing stupid things (like submitting a form before reviewing it, sending a heinous e-mail late at night, or clicking a button that launches a nuclear missile), you might want to do one one of three things:

• Pop-up a confirmation dialog
• Go to a second page, with yet another submit button asking for confirmation, with a summary of the submitted data
• Just let it go, but offer a timed “UNDO” or “CANCEL” link, like Gmail implemented for e-mails

Of course, the best of all three is the latter. The difference here is major. It’s the difference between pro-active annoyingness (pop-ups asking confirmation when you know very well you want to send the e-mail), or reactive flexibility. It also gives your user a great boost of confidence in your application, because the user can do stupid things and knows he’ll be able to fix them in a timely manner.

Now here’s how to do it in Pylons, using the just-released WebUndo lib. You can find it on PyPI, so just pip install WebUndo from your project.

We’re lucky Pylons runs as a single application with threads, since that’s what we’ll use to delay our operations and give the user a chance to cancel. PHP apps for instance, wouldn’t give us that flexibility, since everything vanishes once the request is sent. Take a look at the source code to know how it’s built.

Let’s consider this piece of Pylons controller:

class ThController(BaseController):
    def index(self):
        return render('th.mako')

    def save(self):
        open('/tmp/dump.txt', 'w').write(request.params.get('stuff'))
        return "Everything saved"

where th.mako contains a simple form with a text field named stuff. In this case, once you submit the form in th.mako, it’s done: the file is written to disk, and there’s no way you can cancel it.

Now let’s have a look at this version:

class ThController(BaseController):
    def index(self):
        return render('th.mako')

    def save(self):
        req = request
        def do_save():
            open('/tmp/dump.txt', 'w').write(req.params.get('stuff'))
        key = webundo.launch_cancelable_job(do_save, 5)
        return 'Saving, click to <a href="%s">cancel</a>' % h.url_for(controller='th', action='cancel', id=key)

    def cancel(self, id):
        if webundo.cancel_job(id):
            return "Thread cancelled successfully"
        else:
            return "Too late, thread finished already."

We added some pieces of webundo, the save() action returns a confirmation message, and a link to cancel the operation. We also added a def do_save() wrapper, a call to the launch_cancelable_job function and the passing on of the key. Notice also the req = request, used to make sure the request object gets closed in the do_save() function. The fun part here, is that the operation will not be carried out until the 5 seconds (second parameter to launch_cancelable_job) are elapsed. The last thing is obviously, the cancel action, which will deal with the cancelation.

Using this feature, even if you close your browser window after calling the save action, the do_save() function will be executed normally, only delayed by 5 seconds.

For the Pylons addicts who read that code and say “hey, that request object can’t traverse threads, it’s a StackedObjectProxy”! See the details and answer in my other blog post.

Of course, there are cases where you want a certain operation to be carried out directly (when calling save() for example) but you want some special code to be executed if someone click an undo button for example. The WebUndo lib provides you with another pair of functions to do just that: launch_undoable_job and undo_job.

Take a look at the documentation for full details. Happy undoing!

UPDATED March 20th: typos following feedback

I was working on a feature to implement Gmail’s “Undo” feature in Pylons (which I’ll present on this blog soon), and was faced with a challenge regarding the passage of variables (or functions) to a separate thread. The reason is that Pylons resolves the current request object according to the local thread. You won’t find the same request object in a different thread from which it originated.

I need to be able to resolve those objects (to get the actual object, not the proxy object) before spawning a new thread. Though, I don’t want to require the user to know about all this. He should only create a function that should work in the current thread context, and my code will make sure the good objects get passed on to the new thread.

In Python terms, I wanted to be able to modify the func_closure attribute of a function object (it’s called __closure__ from Python 2.6 onwards). I searched around, only to find that it’s really a read-only attribute, unless you want to do risky things.

So, if you do something like:

def a():
    b = 123
    def myclosure():
        print b
    return myclosure
myfunc = a()

the value of b gets referenced (as a cell object) in the myfunc.func_closure tuple. But there’s no way to change it. What I need is to tweak the myfunc function’s closure, and replace the internal value of b for something else. So when I’d call myfunc(), it actually prints something else than 123, let’s say 'mynewvalue'.

At first, I tried the mix-and-match suggested here, which looks like:

import new
def getitem(a):
    def retclosure():
        return a
    return retclosure
tmpfunc = getitem('mynewvalue')
cell = tmpfunc.func_closure[0]
rewrittenfunc = new.function(myfunc.func_code , myfunc.func_globals, myfunc.func_name, myfunc.func_defaults, (cell,))

Basically, it takes the code, globals, name and defaults from the myfunc function, and uses the cells from another function, mixing two different functions into one.

That’s great for a single reference in the closure, but it’s not very flexible. I want my code to go through all of the closed in objects, and verify is they are of a certain type (namely StackedObjectProxy), and resolve those that need local resolution. It should work with a random number of arguments.

Here is my solution:

def unproxy_closure(f):
    """Rewrite function with resolved closure references (StackedObjectProxy from current thread)"""
    import new
    def create_cell(obj):
        return (lambda: obj).func_closure[0]
    return new.function(func.func_code , func.func_globals, func.func_name,
                        func.func_defaults,
                        tuple(create_cell(cell.cell_contents._current_obj())
                                  if hasattr(cell.cell_contents, '_current_obj')
                                       and
                                     callable(cell.cell_contents._current_obj)
                                  else cell
                              for cell in func.func_closure))

First, there’s that create_cell function, which will return a cell object. It seems to be the only way to create those types of objects: creating a closure and stealing it’s cell object.

Using the new module, it creates a new function borrowing code (globals, defaults, name, etc.) from the first function, and it goes through the cells of the passed-in closure to modify them if necessary. That way, we keep the actual object in the closure, not the proxy object that won’t resolve correctly in the other thread I’m going to launch.

Voilà! That opens the road to a very cool feature I’ll talk about shortly. Comments are welcome.

UPDATE 18h22: rephrased a little bit clunky sentences

UPDATE Match 18th 2010: Added link to what I’ve talk about shortly after

UPDATE March 20th 2010: fixed code block, thanks to John

Voici ma réponses à ces quelques articles trouvés ça et là.

Je trouve qu’elle est absurde cette poursuite – ce brevet aussi d’ailleurs. C’est une sur-utilisation des brevets logiciels et un bon exemple pour démontrer pourquoi on devrait les abolir.

Je ne sais pas si vous connaissez cette technologie d’Apple, mais c’est d’une simplicité déconcertante. C’est de splitter un fichier vidéo en plusieurs petits fichiers et les fournir via HTTP. Je trouverais aberrant qu’on empêche cette méthode simple de distribution du vidéo (qui à mon avis devrait plutôt être adoptée par TOUS comme un standard web) à cause d’un mec qui a dit y avoir pensé avant. Tout le monde y a pensé à ça déjà! Il fallait juste quelqu’un pour le déployer, le mettre en place, le *standardiser*.

Sincèrement. La spec de Apple fait à peine 15 pages, elle est écrite grosse et bourrée d’espaces. On va empêcher le monde entier d’avoir un principe finalement simple pour diffuser de la vidéo live et en différé, à cause d’un bozo à l’autre bout du monde qui a décidé lui aussi de couper ses fichiers en p’tits morceaux ?

C’est ça l’innovation ? Dans ce domaine, on innove si on rejoint les masses et qu’une technologie sert finalement à tous. Qui veut d’un système “d’innovation” qui nous empêche d’utiliser le clique-droit d’une souris ?!

En réponse au Journal du Geek, qui affirme:

C’est clair qu’à ce rythme et connaissant le respect de la vie privée de Google, on va bien finir par avoir un Google Street View à domicile (surement du genre témoin de Jéhovah avec des caméras 360° sur la tête).

dans son article Google Street View à la montagne, dans les magasins et bientôt chez vous !, je réponds:

Pourquoi est-ce qu’on essaie toujours de pousser l’argument de la vie privée alors qu’il n’a rien à voir ?

Je sais que l’insinuation était presqu’inaudible, passagère dans l’article, mais 20% des commentaires ont repris sur ce thème.

L’information et la vue des montagnes est une information publique, et même une information de grande valeur. Il n’y a simplement pas d’atteinte à la vie privée. Si Google fait quelque chose d’admirable – du genre, rendre disponible des choses hors de notre portée jusqu’alors – alors pourquoi ne pas simplement être admiratif.

Et je sais pas si vous savez, mais Google, ils la respectent la vie privée. Ils se posent la question sur plein de sujets chaque jour, pour s’assurer de la vie privée. Ils la méprisent pas, et peut-être qu’ils la comprennent beaucoup mieux que nous qui s’inquiétons quand nos montagnes se font prendre en photo.

Vous avez peut-être vu qu’ils brouillent les visages sur Street View la plupart du temps, sinon sur demande. Ils brouillent les adresses pour pas permettre des associations trop directes. Dans Google Goggles, ils ont volontairement désactivé la reconnaissance faciale, chose possible technologiquement, pour éviter des croisés trop directs.

Est-ce qu’on se plaint de nos commis de banque qui ont accès à toutes les informations sur nos comptes ? Et pourtant, ils ont l’information directe, associée devant ma personne, de ce que je gagne. Ils nous rendent un service, on leur laisse l’information nécessaire.. et ils font sans *aucun* doute des statistiques sur ce qu’ils ont. Pourquoi pas de constipation là ?

On s’alarme de profilage. Depuis la nuit des temps, le marketing fait du profilage, dans les magasins, dans les centres commerciaux. Pourtant, quand on entre chez [insérez votre magasin], on a déjà été profilé, par les article de décorations mis de l’avant, par les yeux de tel ou tel commis qui va nous diriger (en nous *regardant*, en voyant notre style, notre langage) vers ce qu’il croit être le mieux ou le plus approprié pour nous .. et *oui*, pour nous *vendre* ! Incroyable ce commis qui viole comme ça ma vie privée! Il me regarde, et *apprends* des choses sur moi, et même il les utilise pour me vendre des choses, quel arrogant.

Sur Internet, on ne se voit pas. L’information est utilisée pour faire du profilage, ça n’est pas différent, et même beaucoup moins direct.

Cessons de faire les paranoïaques.

Here’s a very good article on why Google Chrome might be the next Web Developer browser:

Why Web Developers Should Switch to Google Chrome - Nettuts

So this is the short story of my presentation at the Montreal Python user groups on january 25th 2010.

I’m preparing to give a talk on “Pylons, web development done right” (or something alike), at the Confoo conference in March. I wanted to test-drive my presentation, so I decided to jump in after seeing a couple of presentation over there - some of which were pretty so-so, thus pushing me to risk myself.

I wanted the presentation to be a performance (since this is my primary training, as a pianist), a lightning bolt filled with impressive demonstrations, speedy typing, cool technologies using a sweet programming language. I wanted to show off Pylons (http://www.pylonshq.com) and a bunch of WSGI sweetnesses (memento, WPHP, CleverCSS, Beaker, Routes, Mako, SQLAlchemy, SqlSoup, etc.)

So I packed my Karmic system with a Squid3 proxy server, to fake having an Internet connection (while everyone in the room was struggling with the bad/non-existant Wi-Fi connection). I wrote some Python scripts that would do all sorts of background mangling to accelerate the live demos* and organised the presentation in modules, so that I could dynamically add or remove pieces of the demonstration based on interests I sized in the audience.

Finally, the presentation was greatly appreciated if I believe the comments I have received. It was a very good experience for me and I received a gift, which is in fact the subject/object of this post: the book “Friends with benefits“.

Read on...