Alexandre Bourget

geek joy

Posts in category: Python

Fabric and Pyramid: automatic deployment issues with paster serve

February 29, 2012 at 04:00 PM

I'm writing a post because I want to keep a memory of the issue I had, and the solution I found.

This is the issue I had:

When running either an /etc/init.d script or directly paster serve --daemon via Fabric (using run() or sudo(), on a remote location that is, more often than not, the process would display: Entering daemon mode but then never run (or die directly).

Here is a description of the problem:

When paster serve enters the part of its code where it starts to daemonize, it will call os.fork(). Then, the next thing it does, is exit the parent process, and let the child go in the background. Its next statement is os.setsid(), which sets the Session ID and Group ID of the process. The problem is that Fabric wraps all of its calls in an /bin/sh -c call, running each command in a separate sub-shell. The moment the parent process (the one who issued the os.fork()) exits, so does the /bin/sh process, because it is finished. But the shell process always sends a SIGHUP to its sub-processes. Some times, the shell *does* kill the child (which is your paster process) before it had time to settle its own signal handlings (through setsid? through other means, I didn't get into it).

If you hook SIGHUP to a no-op (using signal.signal(signal.SIGHUP, signal.SIG_IGN)) just before the call to os.fork() in the paster daemonization procedure, you'll be able to restart your service via Fabric: it won't be killed by the exiting shell. This was annoying to me because it required to patch paster, which I didn't want to do. The other solution, also in Paster, was to add a slight delay to the parent, so that the child had the time to settle its SIGHUP handlers, or was detached correctly. Didn't want to do that either.

I tried to use nohup but this thing sucks, as it requires to redirect everything inside out: I still wanted to have a confirmation of the daemonization proc. success.

So, I found a couple of solutions and settled with one. First you could use screen to launch your daemonization process in the background. I haven't tested that one because I didn't have time to tweak the parameters so that it worked. If you have a working command, send it in the comments I'll add it here. The second one, was simply to add a sleep 1 just after my call to the init.d script, or paster serve --daemon, like that (a fabfile.py snippet):

run("/etc/init.d/myapp-prod restart; sleep 1")

There you go. No patching, simple solution, kind of a hack, just don't forget it!

Read and Post Comments

Introduction to Python in/à Montreal

January 17, 2012 at 05:00 PM

To all Montrealers! There will be an introduction to Python workshop, at UQAM, on January 23rd 2012. It's organized by the Montreal Python User Group and I'll be giving the first workshop. It will be in French though. An English training will be given by a colleague in the following week at McGill.

The cool thing is that Montreal Python is kind of instituting a series of workshops that will happen both in French and in English, in sync with the university sessions. Since Python is not part of the Computer Science programs near Montreal, we'll make sure no one can miss this awesome language.

Head up to the Montreal Python website for more details and registration.

Read and Post Comments

Job in Montreal, for Python and web lovers

May 12, 2011 at 03:40 PM

Savoir-faire Linux

Are you a fan of the web ? Fond of the latest technologies ? You love Python, and maybe you've tried PHP in the past ? You're a FLOSS lover and know Linux personally ? Savoir-faire Linux is the place for you! Being the Free Software reference in Canada, we're a small and friendly S.M.B. that's growing fast.

We have some great work for you, a great environment, a nice office. We're located in Montreal, Canada -- a very nice place if you haven't visited yet.

We are a FLOSS service company that offers expertise on a variety of open source / free software products. We also do some in-house software development and integration of OSS bricks.

We're looking to hire on two fronts: PHP and Python development, both heavily oriented towards the web.

First, we're looking for some good PHP developers to add to our task force. We currently have projects running with Drupal and Symfony.

Second, we're seeking talented Python developers, to do some web oriented development, as well as system-level programming and integration.

Candidates must know the HTML/CSS/JS stack thoroughly. Being in some small and cross-functional teams, candidates will be called to work on all aspects of web apps (design, usability, integration, back-end programming, database and server-side components integration, etc), depending on the interests and capacities of the team members. Candidates must be eager to learn a lot of new stuff (our teams like a constant dose of innovation). We use the Scrum methodology in house and invest a lot in our employees to help them get up to speed with development best practices.

If you're a student, then obviously you don't need to know all these things already, but you should be ready to work hard, on real projects and learn quite a bunch of new things.

Candidates will have opportunities - if desired - to teach what they learn in some of our training classes, to act as consultants at our customers' place, to learn from our support center and infrastructure development team. We are not a huge company so you won't be a number here. Work atmosphere is great, the office is cool (even in the summer :), we sometimes can travel for the job, we've got flexible schedules, a social club, we often have "techno-lunches" (midi-technos) to share our knowledge, and there's plenty of good food around (we're located near the Marché Jean-Talon).

For more details, see: http://www.savoirfairelinux.com

We're waiting for your resume here: rh@savoirfairelinux.com and mention this blog in the body of your message.

UPDATE June 7th: corrected typos

Read and Post Comments

New and hot, part 1: Meta presentation

April 01, 2011 at 09:50 AM

This is part 1 of my March 9th 2011 Confoo presentation. Refer to the Table of Contents for the other parts.

This is a meta-presentation. It's a presentation of the tools and things I've used to build and perform the actual presentation I gave. It is not directly related to Pyramid or Python.

Presentation of my setup and the tools used.

Some people come to me and express wonder on the flow of my live presentation, because it all seems to just work all the time, even without Internet access. Here is what I've done to make things look like they actually work:

Scripts to handle different tasks with keyboard shortcuts

Here is a script I've written that lists a bunch of events with associated chunks of code to be executed when I launch them. The script allows me to hit Ctrl+Spacebar and type in a couple of letters, and then launches the associated piece of code.

They require the Ubuntu xclip and beep packages to be installed. The scripts are available here:

apps.py, gtkdrive.glade, gtkdrive.py, mod_confoo2011.py, presmod.py

This required setting some global hotkeys like this:

$ # Support for compiz (use `ccsm` to activate the "Commands" plugin)
$ gconftool-2 --type string --set /apps/compiz/plugins/commands/command0 "echo 1 > /tmp/btntrigger"
$ gconftool-2 --type string --set /apps/compiz/plugins/commands/run_command0_key "<Super>space"
$ # Support for metacity:
$ gconftool-2 --type string --set /apps/metacity/keybinding_commands/command_1 "echo 1 > /tmp/btntrigger" 
$ gconftool-2 --type string --set /apps/metacity/global_keybindings/run_command_1 "<Super>space"

Afterwards, hitting <Super>space will pop up a little window like this:

Little box

In there, typing one of the commands that were listed when running the presentation will run the associated method in mod_confoo2011.py.

Emacs tweaks

I've added this snippet of code to my ~/.emacs file to adjust the font size, making sure people can see something from a distance and fits a 1024x768 resolution at 80 characters wide (grabbed here):

;; When doing a presentation:
(set-face-attribute 'default nil :height 150)

Also to speed up things when dealing with boilerplate, I'm using yasnippet for Emacs

;; .emacs
(add-to-list 'load-path "~/.emacs.d/plugins/yasnippet-0.6.1c")
(require 'yasnippet)
(yas/initialize)
(yas/load-directory "~/.emacs.d/snippets")

and these yasnippets in my ~/.emacs.d/snippets directory. You'll find in there only the ones I actually used in the presentation, and not the standard ones.

Bash shortcuts

I've added a couple of aliases to speed things up in the presentation. Here is a list of what I've used:

# To use throughout the code to refer to my custom-built FFmpeg. See the post on FFmpeg
export FFMPEG=/home/abourget/build/ffmpeg-0.6.1/bin/ffmpeg

# To make sure I load ipython using the local virtualenv
alias ipython="python `which ipython`"

# PIP download cache, don't download things twice.
export PIP_DOWNLOAD_CACHE=~/.pip/download_cache

Using PIP_DOWNLOAD_CACHE allowed me to install things more quickly. If I had downloaded the package already, it would take it from my download cache. This doesn't prevent *all* Internet access, as pip will still go look on the PyPI if it's the latest version for example.

You could also have set up a local mirror with the packages you needed, and install everything following this method.

Squid caching proxy

At first, I wanted to use Squid caching to fake I had an Internet connection, but in the end, the presentation room did have Internet and I had a cell phone with tethering enabled so I didn't need the proxy. But still, here is how I did it the year before:

$ sudo apt-get install squid3

Tweak the squid3 proxy config:

# Add:
refresh_pattern .               1440    90%     4320 reload-into-ims ignore-no-cache ignore-reload override-expire
# Comment out:
#refresh_pattern -i (/cgi-bin/|\?) 0    0%      0    
# Enable:
offline_mode on

Restart the server, and you should be all set. In any console you wish to use the proxy, run:

$ export http_proxy=http://localhost:3128

Pre-fetched downloads

I've downloaded the HTML5 boilerplate code from http://html5boilerplate.com/.

I also had a copy of socket.io.js from here.

Video and sound recordings

To do video recording of the screen, I've used this ffmpeg command:

~/build/ffmpeg-0.6.1/bin/ffmpeg -y -f x11grab -r 12 -s 1024x768 -i :0.0 -vcodec libx264 -vpre veryfast Presentation.mkv

Sound recording was made with an old SIM-less HTC Dream (USA's G1), running a free Android application named Virtual Recorder, recording in PCM WAVE using the microphone-earbuds thing that came with the phone.

Someone to tape the presentation, using an external camcorder.

I was using a SANYO Xacti camera, which spat some already-encoded H.264/AVC, that was sitting on a tripod.

Final rendering

The final rendering was done with Cinelerra, using ReframeRT to adjust the frame-rate of the Presentation.mkv file, produced by FFmpeg's x11grab. I also needed to adjust the frame-rate for some reasons

I used the audio from the Android device, the video from tx11grab

YouTube publishing

I've used YouTube to publish the video parts, and this Blogofile blog for the tutorials themselves. Here is the YouTube result:

Read and Post Comments

New and hot, part 6: Redis, Publish and Subscribe

March 31, 2011 at 04:40 PM

This is part 6 of my March 9th 2011 Confoo presentation. Refer to the Table of Contents for the other parts.

The original announcement for my Confoo presentation was to use RabbitMQ to do real-time communications, but then, after finding a way to use Redis to do it, I found it was simpler and more straight-forward for a live demo.

Redis Integration

In this part, we will use Redis's PubSub features to communicate in real-time between the different web browsers connected with a Socket.IO (or a Websocket) to our Gevent-based application.

Redis is a FOSS database which supports the Publish/Subscribe metaphore. It is written in C, and is quite robust. It also comes pre-setup on your Ubuntu machine.

redis Show the install instructions for redis-server.

To install Redis, simply run:

$ sudo apt-get install redis-server

and you're ready to go! To get things into production, you'll want to learn how to secure your installation.

You can try the redis server locally if you kill the system-wide instance (through init.d/service scripts):

$ redis-server
...
[21194] 08 Mar 22:08:54 * Server started, Redis version 1.3.15
...
[21194] 08 Mar 22:08:54 * The server is now ready to accept connections on port 6379
[21194] 08 Mar 22:08:54 - 0 clients connected (0 slaves), 533436 bytes in use

Python Redis bindings and PubSub commands

Let's setup what is needed for messaging using Redis.

First of all, we'll want to connect to the Redis server and publish a new message. We'll add some code to the views.py and set up the publisher. We'll name the channel foo:

...
import redis
from json import loads, dumps
...
def encode_video(filename, request):
    ...
    if p.returncode == 0:
        r = redis.Redis()
        msg = {'type': 'video', 'fileid': str(fileid),
               'url': request.route_url('video', fileid=fileid)}
        r.publish("foo", dumps(msg))

Now, we'll want to have our Socket.IO socket listen for any incoming message from the Redis subscription:

class ConnectIOContext(SocketIOContext):
    def msg_connect(self, msg):
        ...
        def listener():
            r = redis.Redis()
            r.subscribe(['foo'])
            for m in r.listen():
                if not self.io.connected():
                    return
                print "From Redis:", m
                if m['type'] == 'message':
                    self.io.send(loads(m['data']))
        self.spawn(listener)

When we receive a message as the m object, we'll just pass it directly to the current Socket.IO socket. We've subscribed to the foo channel, so we'll receive a copy of any message sent to it.

We will also want the client to handle the new message, so we'll tweak the call to go() and the definition for the go() function, in index.html:

      <script>
        function go(url) {
          $('#video').html('<div>Video ready: ' + url + '</div><video controls preload src="'+ url + '" />');
          ...
        }
      </script>

      ...
      socket.on('message', function(obj) {
        ...
        if (obj.type == "video") {
          go(obj.url);
        }
        ...

In here, if the message is of type video, like the one sent by the encode_video() function, it will trigger the playback immediately.

Let's make sure a background process is spawned when someone asks us to encode a video. The encoding process itself could be totally decoupled from this server, sent to a render farm for example. In views.py, we would tweak:

def get_iframe(request):
    if 'file' in request.params:
        ...
        gevent.spawn(encode_video, '/tmp/video.mp4', request)

That's it! Hope you liked the series! I might add the Android app demo if I have time, so please be patient.

If you would like us to implement anything you've seen here in a real-life project, or to kick start some of your projects, don't hesitate to send out an e-mail at contact@savoirfairelinux.com and mention this blog.

Read and Post Comments

Next Page »