Python Rocks - So what is Stackless Python?

Python may be one of the most widely learned and used languages today, but it was conceived in the late 1980's when if you hadn't got a mainframe, you almost certainly were running your code on a single CPU computer of some sort.

For this reason the original implementation of Python was written with the understanding that it was perfectly sensible to use the same single execution stack that C used - after all Python was written in C. Despite being on version 2.7/3.3 nowadays, the standard Python is still written in C, still uses a single execution stack design and is often known as CPython.

English: CPU Zilog Z8 (Photo credit: Wikipedia)

The execution stack - or call stack - or just stack is like a big spike that you stick messages on in a last on, first off way and it's where the low level machine code subroutines used to stick the current code  address before going away to do some jiggery pokery. When the subroutine was finished it returned by pulling the last address off the stack and then execution continued from there. It was all fairly simple in the days of the Z80 and as I understand it, that's still essentially how a single CPU - or Core - works.

AMD Athlon™ X2 Dual-Core Processor 6400+ in AM2 package (Photo credit: Wikipedia)

The problem is that sometime in the early 2000s, dual-core, and then multi-core chips started to be become increasingly affordable and therefore available. Most of you will be using a multi-core system to read this post. This means that your systems are capable of running more than one process a time - what is called concurrency.

This is a bit of a pain for CPython because it only knows how to use a single stack, i.e. a single core, and that is just a bit of a waste of those other cores which are just itching to make it all run super fast.

So Stackless Python is essentially a redesign of CPython which avoids using the call stack and instead uses something called microthreads to get around the problem. This means four things to you:

• Concurrent programming is possible.
• Concurrency can improve on execution time if done properly.
• You need to learn some new concepts: tasklets and channels.
• You get to use some new stuff: tasklets and channels.

I'll introduce those next time.

There's a very informative interview with the creator of Stackless Python here.

You may like to be ready by reading my post about installing Stackless Python.

Good luck fellow travellers.

Install Stackless Python on Ubuntu

I'm just about to write a couple of posts on Stackless Python and the Nagare Micro Framework which runs on it. So I've been installing Stackless on my Ubuntu 12.04. Here are some nice copy and paste instructions if you want to play along.

First install the required libraries and get stackless itself:

Now install stackless :

After the "make" you'll see some failures as below. Just ignore them.

Now it's time to link your standard (CPython) packages so that they can be used with stackless:

...and edit the paths in the site.py file. At about line 300, edit the file to look like this. It's the second sitepackages.append bit we're adding here:

That should be it! Let's test it:

Aggregating Everything - Map/Reduce and Camel?

If you are used to Map/Reduce you will be used to the idea of breaking tasks down into little chunks and then collecting the partial results together into some final format.

So, recently when I was parsing zillions of rows of data and aggregating related data into partial CSV files  and then aggregating the bits of partial of data to reports I thought - Aha! MapReduce.

For a whole bunch of good design decisions I was using Apache Camel - a neat pipelining tool which with a bit of help from ActiveMQ provides the sort of long running stability that I needed. Camel however does not do Map/Reduce, but it does have the Aggregator Integration pattern, which you can use to so a similar thing.

Image courtesy of Carlos Oliveira

Imagine you empty your jar of loose change on a table. You stack the nickles in stacks of ten coins, the dimes is stacks of ten coins and the quarters in stacks of ten coins. You add up all the 50 cents, $1s and $2.50s and you know how much you have. That's Map/Reduce.

Now, imagine you empty your jar of loose change into one of those coin counting machines in the Mall. Internally all the coins are sorted by falling through a hole which is either nickle, dime or quarter shaped and as they emerge from the other side they are counted*. That's aggregation Camel style.

I did hit a bit of a snag. I couldn't work out how to tell the Aggregator Integration pattern that there were no more files to come... Stop... Woaa-there... Desist!

It turns out that hidden away (in the middle of the docs) the File endpoint rather usefully sets a flag in the headers called CamelBatchComplete which is just what I was looking for:

Good luck fellow travelers.

* I have no idea how a coin counting machine works.

Simple Camel Configuration of a Twitter Endpoint

I was asked just now how my got the Twitter Stream working in my new Camel based project and how I managed the credentials.

The Twitter endpoint works like a dream and this is essentially what my code looks like. All you need is a secrets.properties file in alongside your java file.

Apache Camel - Connection Beans Without Spring

While writing some tests for an Apache Camel project, I just spent rather longer than I'd have liked trying to work out how to configure a connection bean for Mongo without using Spring.

Since I hide my embarrassments in public I thought I'd best share with anyone else with brain freeze.


Don't forget you need the camel-mongodb artifact in your pom.xml file. Good luck fellow travellers.

Python Web Microframeworks - Take Your Pick

You may read my post "Top Python Web Frameworks - Today" in which I took a fresh look at what which Python Web Frameworks were still around and still maintained.

In this post I give a quick overview of about half of those which I have loosely designated as "Microframeworks" - regardless of what the authors have called them. Wikipedia doesn't have a definition for microframework - I just looked - so what I really mean here is anything which let's you get started without having to learn a whole bunch of syntax and convention. Right on sister!

Let's get going:

Now that's done a little code from each -

Bottle (v0.11.13)

Bottle doesn't rely on any other packages at all, which means it's a great framework to use if you want to see all the working parts as they're all in the one file. That being said it can offer client-side sessions and compression and even WebSockets straight out of the box so it's not just a toy by any means.

Flask (v0.9)

Flask is dependent on Werkzeug for all the WSGI stuff and upon Jinja2 as a template library. It comes with client-side sessions, a built in debugger and is totally unicode friendly. I love Flask and use it often as my other posts will testify.

kiss.py (v0.4.9)

The first of the new boys, kiss.py is certainly not package independent! It requires Werkzeug for WSGI, requests for http, Beaker for sessions, Elixir and SQLAlchemy for an ORM (PostgreSQL, MySQL and SQLite), Jinja2 for templates, gevent, pev and greenlet for events as well as compressinja, jsmin, jsonpickle, putils and pyScss which add various other niceties. Almost all well known and trusted libraries.

web.py (v0.37)

Again, web.py doesn't rely on any other packages at all, but to me it's not as useful as flask or kiss.py and not as simple to study as bottle, so I can't see the point, although according to the site, it's well used by others.

wheezy.web (v0.1.307)

pip installs wheezy.web, wheezy.core, wheezy.caching, wheezy.html, wheezy.http, wheezy.routing, wheezy.security, wheezy.validation which to me looks like the developers have taken a sensible approach to the development cycle by splitting everything up into independent code units.

According to the site,  functionality includes routing, model update/validation, authentication/authorization,content caching with dependency, xsrf/resubmission protection, AJAX+JSON, i18n (gettext), middlewares, and more.

In Summary

No I haven't tested them to death and no I haven't even tried out kiss.py and wheezy.web in a real world app, although I will do. I certainly have not done either load or concurrency testing on them. You can do that and I'll read your blog.

Goo luck fellow traveller.

CoffeeScript Love: Backbone.js Tutorials in CoffeeScript

Ahhhh enjoy the aroma, inhale the caffeine. If you're looking for a nice list of Backbone tutorials that use CoffeeScript then look no more - CoffeeScript Love: Backbone.js Tutorials in CoffeeScript.