Alexandre Bourget

I'm writing a post because I want to keep a memory of the issue I had, and the solution I found.

This is the issue I had:

When running either an /etc/init.d script or directly paster serve --daemon via Fabric (using run() or sudo(), on a remote location that is, more often than not, the process would display: Entering daemon mode but then never run (or die directly).

Here is a description of the problem:

When paster serve enters the part of its code where it starts to daemonize, it will call os.fork(). Then, the next thing it does, is exit the parent process, and let the child go in the background. Its next statement is os.setsid(), which sets the Session ID and Group ID of the process. The problem is that Fabric wraps all of its calls in an /bin/sh -c call, running each command in a separate sub-shell. The moment the parent process (the one who issued the os.fork()) exits, so does the /bin/sh process, because it is finished. But the shell process always sends a SIGHUP to its sub-processes. Some times, the shell *does* kill the child (which is your paster process) before it had time to settle its own signal handlings (through setsid? through other means, I didn't get into it).

If you hook SIGHUP to a no-op (using signal.signal(signal.SIGHUP, signal.SIG_IGN)) just before the call to os.fork() in the paster daemonization procedure, you'll be able to restart your service via Fabric: it won't be killed by the exiting shell. This was annoying to me because it required to patch paster, which I didn't want to do. The other solution, also in Paster, was to add a slight delay to the parent, so that the child had the time to settle its SIGHUP handlers, or was detached correctly. Didn't want to do that either.

I tried to use nohup but this thing sucks, as it requires to redirect everything inside out: I still wanted to have a confirmation of the daemonization proc. success.

So, I found a couple of solutions and settled with one. First you could use screen to launch your daemonization process in the background. I haven't tested that one because I didn't have time to tweak the parameters so that it worked. If you have a working command, send it in the comments I'll add it here. The second one, was simply to add a sleep 1 just after my call to the init.d script, or paster serve --daemon, like that (a fabfile.py snippet):

run("/etc/init.d/myapp-prod restart; sleep 1")

There you go. No patching, simple solution, kind of a hack, just don't forget it!