Alexandre Bourget

geek joy

Welcome to my personal blog

Recent posts

Amazon EC2 vs Google Cloud Compute Engine

June 28, 2012 at 02:45 PM

Here is a quick comparison of the pricing between Amazon's EC2 and Google's newly released Cloud

Definition: GCEU (Google Cloud Engine Unit) ~= EC2 Compute Unit

Provider Name Virtual cores Memory Compute Unit HDD (GB) $ /hour $ /Unit /hour
Google n1-standard-1-d 1 3.75GB 2.75 420 $0.145 $0.053
Amazon m1.medium - Medium Instance 1 3.75GB 2 410 $0.160 $0.080
Google n1-standard-2-d 2 7.5GB 5.5 870 $0.290 $0.053
Amazon m1.large - Large Instance 2 7.5GB 4 850 $0.320 $0.080
Google n1-standard-4-d 4 15GB 11 1770 $0.580 $0.053
Amazon m1.xlarge - Extra Large Instance 4 15GB 8 1690 $0.640 $0.080
Google n1-standard-8-d 8 30GB 22 2 x 1770 $1.160 $0.053
Amazon m2.2xlarge - High-Mem Dbl-X-Large
Memory-equiv, 1.7x less CPU
4 34.2GB 13 850 $0.900 $0.069
Amazon c1.xlarge - High-CPU X-Large
CPU-equiv, 4.3x less mem.
8 7GB 20 1690 $0.660 $0.033
Amazon cc1.4xlarge - Cluster Compute 4x-X-Large
Pseudo-equiv., with tradeoffs
8 23GB 33.5 1690 $1.300 $0.039

Notes:

  • There is no direct equivalent to Google's largest instance. You have to trade off memory or CPU each time. In that sense, Google seems to have more general purpose instances (for now).
  • Also, Amazon will give you less HDD space with the larger instances
  • Only the Linux instances were considered in the comparison, as Google only offers Linux (for now?)
  • I made now comparison of the different Compute Units from each provider. I pretty much assume to be equivalent.
  • The comparison is done using Amazon's regular instances. Google does not seem to provide adaptive pricing according to usage, but they claim you always have full power of your instance, and you don,t notice the other VMs. If this claim is true, then their VMs would be pretty much the equivalent of the high-utilisation VMs of Amazon.
  • The Amazon pricing list that was taken was for the cheapest region (USA East Virginia), so move around the globe and your Amazon bill could be up to ~1.8x higher.
  • This isn't a complete comparison as it doesn't take into account the network and storage fees. I might take a look if there is interest.

Conclusion: for now, Amazon has better fits and prices depending on your purpose for high-end instances, but Google seems to have better prices overall. I'm eager to discover their

Source: Google Cloud specs and pricing, Amazon's specs and pricing

Read and Post Comments

Fabric and Pyramid: automatic deployment issues with paster serve

February 29, 2012 at 04:00 PM

I'm writing a post because I want to keep a memory of the issue I had, and the solution I found.

This is the issue I had:

When running either an /etc/init.d script or directly paster serve --daemon via Fabric (using run() or sudo(), on a remote location that is, more often than not, the process would display: Entering daemon mode but then never run (or die directly).

Here is a description of the problem:

When paster serve enters the part of its code where it starts to daemonize, it will call os.fork(). Then, the next thing it does, is exit the parent process, and let the child go in the background. Its next statement is os.setsid(), which sets the Session ID and Group ID of the process. The problem is that Fabric wraps all of its calls in an /bin/sh -c call, running each command in a separate sub-shell. The moment the parent process (the one who issued the os.fork()) exits, so does the /bin/sh process, because it is finished. But the shell process always sends a SIGHUP to its sub-processes. Some times, the shell *does* kill the child (which is your paster process) before it had time to settle its own signal handlings (through setsid? through other means, I didn't get into it).

If you hook SIGHUP to a no-op (using signal.signal(signal.SIGHUP, signal.SIG_IGN)) just before the call to os.fork() in the paster daemonization procedure, you'll be able to restart your service via Fabric: it won't be killed by the exiting shell. This was annoying to me because it required to patch paster, which I didn't want to do. The other solution, also in Paster, was to add a slight delay to the parent, so that the child had the time to settle its SIGHUP handlers, or was detached correctly. Didn't want to do that either.

I tried to use nohup but this thing sucks, as it requires to redirect everything inside out: I still wanted to have a confirmation of the daemonization proc. success.

So, I found a couple of solutions and settled with one. First you could use screen to launch your daemonization process in the background. I haven't tested that one because I didn't have time to tweak the parameters so that it worked. If you have a working command, send it in the comments I'll add it here. The second one, was simply to add a sleep 1 just after my call to the init.d script, or paster serve --daemon, like that (a fabfile.py snippet):

run("/etc/init.d/myapp-prod restart; sleep 1")

There you go. No patching, simple solution, kind of a hack, just don't forget it!

Read and Post Comments

Introduction to Python in/à Montreal

January 17, 2012 at 05:00 PM

To all Montrealers! There will be an introduction to Python workshop, at UQAM, on January 23rd 2012. It's organized by the Montreal Python User Group and I'll be giving the first workshop. It will be in French though. An English training will be given by a colleague in the following week at McGill.

The cool thing is that Montreal Python is kind of instituting a series of workshops that will happen both in French and in English, in sync with the university sessions. Since Python is not part of the Computer Science programs near Montreal, we'll make sure no one can miss this awesome language.

Head up to the Montreal Python website for more details and registration.

Read and Post Comments

Job in Montreal, for Python and web lovers

May 12, 2011 at 03:40 PM

Savoir-faire Linux

Are you a fan of the web ? Fond of the latest technologies ? You love Python, and maybe you've tried PHP in the past ? You're a FLOSS lover and know Linux personally ? Savoir-faire Linux is the place for you! Being the Free Software reference in Canada, we're a small and friendly S.M.B. that's growing fast.

We have some great work for you, a great environment, a nice office. We're located in Montreal, Canada -- a very nice place if you haven't visited yet.

We are a FLOSS service company that offers expertise on a variety of open source / free software products. We also do some in-house software development and integration of OSS bricks.

We're looking to hire on two fronts: PHP and Python development, both heavily oriented towards the web.

First, we're looking for some good PHP developers to add to our task force. We currently have projects running with Drupal and Symfony.

Second, we're seeking talented Python developers, to do some web oriented development, as well as system-level programming and integration.

Candidates must know the HTML/CSS/JS stack thoroughly. Being in some small and cross-functional teams, candidates will be called to work on all aspects of web apps (design, usability, integration, back-end programming, database and server-side components integration, etc), depending on the interests and capacities of the team members. Candidates must be eager to learn a lot of new stuff (our teams like a constant dose of innovation). We use the Scrum methodology in house and invest a lot in our employees to help them get up to speed with development best practices.

If you're a student, then obviously you don't need to know all these things already, but you should be ready to work hard, on real projects and learn quite a bunch of new things.

Candidates will have opportunities - if desired - to teach what they learn in some of our training classes, to act as consultants at our customers' place, to learn from our support center and infrastructure development team. We are not a huge company so you won't be a number here. Work atmosphere is great, the office is cool (even in the summer :), we sometimes can travel for the job, we've got flexible schedules, a social club, we often have "techno-lunches" (midi-technos) to share our knowledge, and there's plenty of good food around (we're located near the Marché Jean-Talon).

For more details, see: http://www.savoirfairelinux.com

We're waiting for your resume here: rh@savoirfairelinux.com and mention this blog in the body of your message.

UPDATE June 7th: corrected typos

Read and Post Comments

New and hot, part 1: Meta presentation

April 01, 2011 at 09:50 AM

This is part 1 of my March 9th 2011 Confoo presentation. Refer to the Table of Contents for the other parts.

This is a meta-presentation. It's a presentation of the tools and things I've used to build and perform the actual presentation I gave. It is not directly related to Pyramid or Python.

Presentation of my setup and the tools used.

Some people come to me and express wonder on the flow of my live presentation, because it all seems to just work all the time, even without Internet access. Here is what I've done to make things look like they actually work:

Scripts to handle different tasks with keyboard shortcuts

Here is a script I've written that lists a bunch of events with associated chunks of code to be executed when I launch them. The script allows me to hit Ctrl+Spacebar and type in a couple of letters, and then launches the associated piece of code.

They require the Ubuntu xclip and beep packages to be installed. The scripts are available here:

apps.py, gtkdrive.glade, gtkdrive.py, mod_confoo2011.py, presmod.py

This required setting some global hotkeys like this:

$ # Support for compiz (use `ccsm` to activate the "Commands" plugin)
$ gconftool-2 --type string --set /apps/compiz/plugins/commands/command0 "echo 1 > /tmp/btntrigger"
$ gconftool-2 --type string --set /apps/compiz/plugins/commands/run_command0_key "<Super>space"
$ # Support for metacity:
$ gconftool-2 --type string --set /apps/metacity/keybinding_commands/command_1 "echo 1 > /tmp/btntrigger" 
$ gconftool-2 --type string --set /apps/metacity/global_keybindings/run_command_1 "<Super>space"

Afterwards, hitting <Super>space will pop up a little window like this:

Little box

In there, typing one of the commands that were listed when running the presentation will run the associated method in mod_confoo2011.py.

Emacs tweaks

I've added this snippet of code to my ~/.emacs file to adjust the font size, making sure people can see something from a distance and fits a 1024x768 resolution at 80 characters wide (grabbed here):

;; When doing a presentation:
(set-face-attribute 'default nil :height 150)

Also to speed up things when dealing with boilerplate, I'm using yasnippet for Emacs

;; .emacs
(add-to-list 'load-path "~/.emacs.d/plugins/yasnippet-0.6.1c")
(require 'yasnippet)
(yas/initialize)
(yas/load-directory "~/.emacs.d/snippets")

and these yasnippets in my ~/.emacs.d/snippets directory. You'll find in there only the ones I actually used in the presentation, and not the standard ones.

Bash shortcuts

I've added a couple of aliases to speed things up in the presentation. Here is a list of what I've used:

# To use throughout the code to refer to my custom-built FFmpeg. See the post on FFmpeg
export FFMPEG=/home/abourget/build/ffmpeg-0.6.1/bin/ffmpeg

# To make sure I load ipython using the local virtualenv
alias ipython="python `which ipython`"

# PIP download cache, don't download things twice.
export PIP_DOWNLOAD_CACHE=~/.pip/download_cache

Using PIP_DOWNLOAD_CACHE allowed me to install things more quickly. If I had downloaded the package already, it would take it from my download cache. This doesn't prevent *all* Internet access, as pip will still go look on the PyPI if it's the latest version for example.

You could also have set up a local mirror with the packages you needed, and install everything following this method.

Squid caching proxy

At first, I wanted to use Squid caching to fake I had an Internet connection, but in the end, the presentation room did have Internet and I had a cell phone with tethering enabled so I didn't need the proxy. But still, here is how I did it the year before:

$ sudo apt-get install squid3

Tweak the squid3 proxy config:

# Add:
refresh_pattern .               1440    90%     4320 reload-into-ims ignore-no-cache ignore-reload override-expire
# Comment out:
#refresh_pattern -i (/cgi-bin/|\?) 0    0%      0    
# Enable:
offline_mode on

Restart the server, and you should be all set. In any console you wish to use the proxy, run:

$ export http_proxy=http://localhost:3128

Pre-fetched downloads

I've downloaded the HTML5 boilerplate code from http://html5boilerplate.com/.

I also had a copy of socket.io.js from here.

Video and sound recordings

To do video recording of the screen, I've used this ffmpeg command:

~/build/ffmpeg-0.6.1/bin/ffmpeg -y -f x11grab -r 12 -s 1024x768 -i :0.0 -vcodec libx264 -vpre veryfast Presentation.mkv

Sound recording was made with an old SIM-less HTC Dream (USA's G1), running a free Android application named Virtual Recorder, recording in PCM WAVE using the microphone-earbuds thing that came with the phone.

Someone to tape the presentation, using an external camcorder.

I was using a SANYO Xacti camera, which spat some already-encoded H.264/AVC, that was sitting on a tripod.

Final rendering

The final rendering was done with Cinelerra, using ReframeRT to adjust the frame-rate of the Presentation.mkv file, produced by FFmpeg's x11grab. I also needed to adjust the frame-rate for some reasons

I used the audio from the Android device, the video from tx11grab

YouTube publishing

I've used YouTube to publish the video parts, and this Blogofile blog for the tutorials themselves. Here is the YouTube result:

Read and Post Comments

Next Page »