Python _imaging cannot open shared object file

Okay so I'm using a 64 bit Linux and I was having a lot of trouble getting Calibre ebook tool to convert books into Kindle loving mobi format. I was getting a python error:

ImportError: libjpeg.so.62: cannot open shared object file: No such file or directory

Turns out all you need to do is

sudo apt-get install libjpeg62

You can test this by starting python and doing:

import _imaging

Install Google App Engine on Ubuntu 12.10

Google App Engine  is currently on version 1.7.4 and Ubuntu has recently released Ubuntu 12.10 (Quantal Quetzal). Quantal Quetzal comes with Python 2.7 installed, and App Engine has been providing that version of Python as an option since February 2012. So if you are starting a new App Engine project, it's probably a good time to move to Python 2.7.

I'll explain briefly how you start a new project and there's a nice clean copy of the code at the bottom that you can cut and paste.

Let's get the show on the road. Choose a name for the project and create and switch to a virtual environment:

PROJ=gae_project
mkvirtualenv ${PROJ}
cdvirtualenv

Note that note that "--no-site-packages" and "--distribute" are now the defaults for mkvirtualenv. You don't even need to use "--python=python2.7" on Ubuntu 12.10.

Now we need to know what the latest version of App Engine is, but as of writing it's 1.7.4:

GAE=1.7.4
wget -O /tmp/gae.zip http://googleappengine.googlecode.com/files/google_appengine_${GAE}.zip
unzip /tmp/gae.zip

Now let's create an App Engine app. The app will need a name that has been created in the App Engine Console:

GAE_APP_NAME=dummy
mkdir -p gae_app/static

Now create the app.yaml file:

echo """application: ${GAE_APP_NAME}
version: development
runtime: python27
api_version: 1
threadsafe: true

default_expiration: 7d

handlers:
- url: /static
  static_dir: static
- url: .*
  script: wsgi_app.app
""" > gae_app/app.yaml

And finally the app itself:

echo """import webapp2

class MainPage(webapp2.RequestHandler):
  def get(self):
      self.response.headers['Content-Type'] = 'text/plain'
      self.response.out.write('Please replace me with a decent WSGI App Framework such as Flask')

app = webapp2.WSGIApplication([('/', MainPage)],
                              debug=True)
""" > gae_app/wsgi_app.py

And finally to run the development server:

python ./google_appengine/dev_appserver.py gae_app/

I hope that this has all been of some help to you. Did I miss anything? Please comment below.

Python Rocks - So what is Stackless Python?

Python may be one of the most widely learned and used languages today, but it was conceived in the late 1980's when if you hadn't got a mainframe, you almost certainly were running your code on a single CPU computer of some sort.

For this reason the original implementation of Python was written with the understanding that it was perfectly sensible to use the same single execution stack that C used - after all Python was written in C. Despite being on version 2.7/3.3 nowadays, the standard Python is still written in C, still uses a single execution stack design and is often known as CPython.

English: CPU Zilog Z8 (Photo credit: Wikipedia)

The execution stack - or call stack - or just stack is like a big spike that you stick messages on in a last on, first off way and it's where the low level machine code subroutines used to stick the current code  address before going away to do some jiggery pokery. When the subroutine was finished it returned by pulling the last address off the stack and then execution continued from there. It was all fairly simple in the days of the Z80 and as I understand it, that's still essentially how a single CPU - or Core - works.

AMD Athlon™ X2 Dual-Core Processor 6400+ in AM2 package (Photo credit: Wikipedia)

The problem is that sometime in the early 2000s, dual-core, and then multi-core chips started to be become increasingly affordable and therefore available. Most of you will be using a multi-core system to read this post. This means that your systems are capable of running more than one process a time - what is called concurrency.

This is a bit of a pain for CPython because it only knows how to use a single stack, i.e. a single core, and that is just a bit of a waste of those other cores which are just itching to make it all run super fast.

So Stackless Python is essentially a redesign of CPython which avoids using the call stack and instead uses something called microthreads to get around the problem. This means four things to you:

• Concurrent programming is possible.
• Concurrency can improve on execution time if done properly.
• You need to learn some new concepts: tasklets and channels.
• You get to use some new stuff: tasklets and channels.

I'll introduce those next time.

There's a very informative interview with the creator of Stackless Python here.

You may like to be ready by reading my post about installing Stackless Python.

Good luck fellow travellers.

Install Stackless Python on Ubuntu

I'm just about to write a couple of posts on Stackless Python and the Nagare Micro Framework which runs on it. So I've been installing Stackless on my Ubuntu 12.04. Here are some nice copy and paste instructions if you want to play along.

First install the required libraries and get stackless itself:

Now install stackless :

After the "make" you'll see some failures as below. Just ignore them.

Now it's time to link your standard (CPython) packages so that they can be used with stackless:

...and edit the paths in the site.py file. At about line 300, edit the file to look like this. It's the second sitepackages.append bit we're adding here:

That should be it! Let's test it:

Python Web Microframeworks - Take Your Pick

You may read my post "Top Python Web Frameworks - Today" in which I took a fresh look at what which Python Web Frameworks were still around and still maintained.

In this post I give a quick overview of about half of those which I have loosely designated as "Microframeworks" - regardless of what the authors have called them. Wikipedia doesn't have a definition for microframework - I just looked - so what I really mean here is anything which let's you get started without having to learn a whole bunch of syntax and convention. Right on sister!

Let's get going:

Now that's done a little code from each -

Bottle (v0.11.13)

Bottle doesn't rely on any other packages at all, which means it's a great framework to use if you want to see all the working parts as they're all in the one file. That being said it can offer client-side sessions and compression and even WebSockets straight out of the box so it's not just a toy by any means.

Flask (v0.9)

Flask is dependent on Werkzeug for all the WSGI stuff and upon Jinja2 as a template library. It comes with client-side sessions, a built in debugger and is totally unicode friendly. I love Flask and use it often as my other posts will testify.

kiss.py (v0.4.9)

The first of the new boys, kiss.py is certainly not package independent! It requires Werkzeug for WSGI, requests for http, Beaker for sessions, Elixir and SQLAlchemy for an ORM (PostgreSQL, MySQL and SQLite), Jinja2 for templates, gevent, pev and greenlet for events as well as compressinja, jsmin, jsonpickle, putils and pyScss which add various other niceties. Almost all well known and trusted libraries.

web.py (v0.37)

Again, web.py doesn't rely on any other packages at all, but to me it's not as useful as flask or kiss.py and not as simple to study as bottle, so I can't see the point, although according to the site, it's well used by others.

wheezy.web (v0.1.307)

pip installs wheezy.web, wheezy.core, wheezy.caching, wheezy.html, wheezy.http, wheezy.routing, wheezy.security, wheezy.validation which to me looks like the developers have taken a sensible approach to the development cycle by splitting everything up into independent code units.

According to the site,  functionality includes routing, model update/validation, authentication/authorization,content caching with dependency, xsrf/resubmission protection, AJAX+JSON, i18n (gettext), middlewares, and more.

In Summary

No I haven't tested them to death and no I haven't even tried out kiss.py and wheezy.web in a real world app, although I will do. I certainly have not done either load or concurrency testing on them. You can do that and I'll read your blog.

Goo luck fellow traveller.

Top Python Web Frameworks - Today

Every once in a while it's nice to rediscover old friends. Over the last few years I've had a play with quite a few Python Web Frameworks, but one does tend to get a favourite and stick with it. Same as other areas I suppose - cars, dogs, beers, partners... Still, it's good to have a look at what's new and fresh once in a while (my wife doesn't care, but don't tell my dog.)

example of Python language (Photo credit: Wikipedia)

If you look around the internet, there are probably thirty or so Python Web Frameworks around, but not all of them are actively supported. This means that they are likely to be missing functionality in some of the latest latest areas of interest - OAuth2, HTML5 or mobile support for example. Or maybe they just don't make it easy to stuff that you have become used to over the last few years - AJAX or web services maybe.

In my previous post "My Top Ten Python Web Frameworks" from about 18months ago, I gave you my opinion of what was hot, or not, at that time. Some of those seem to have stalled now - Tipfy, GAE framework, Weblayer; while others have appeared or matured - kiss.py, web.py, wheezy.web.

Below is an alphabetic list of the active (updated this year) which I know about. There may be others out there, so please let me know in the comments at the bottom.

• Bottle now on v0.11
• CubicWeb now on v3.15.5
• Django now on v1.4
• Flask now on v0.9
• kiss.py now on v0.4.9
• Nagare now on v0.4.1
• Pyramid now on v1.3
• TurboGears now on v2.2.0
• web.py now on v0.37
• web2py now on v2.2.1
• wheezy.web now on v0.1

In my next few posts I'll be giving you a run down of what state each of them is in and try to give you some idea of how they fit your requirements.

Good luck fellow traveller.

Nginx & uWSGI Living Together in Your Shed

Let's assume you have just bought the memorable domain name idrinkink.org and you want to host your lovely web app on that old server in the shed. 

I recently set up both Flask-Ahoy! and Django-Ahoy! on a five year old desktop my daughter was chucking out, so I can guarantee you that this configuration can handle enough traffic to get you going, or to demo products to your customers.

Before we start, I'm assuming you are familiar with python web frameworks (Django, Flask, Bottle etc) and with using virtualenv to segregate your environments. I usually use Flask as my framework of choice, so I'm using their "Hello World" app here.

Let's get started:

Now the contents of those files:

Finally copy the config into place and start the uwsgi daemon and the server up:


This of course is just the basics, but it will hopefully give you a starting point from which you can fiddle to your hearts content.

Good luck fellow traveller.

Install Google App Engine on Ubuntu 12.04

If you want to use Google App Engine on Ubuntu there are a couple of things to note. Firstly GAE has recently moved to version 1.6.5 and secondly Ubuntu has recently released Ubuntu 12.04 (Precise Pangolin). Precise Pangolin is a Long Term Support (LTS) version of Ubuntu to come with Python 2.7 installed, and App Engine has been providing it as an option since February 2012. So if you are starting a new App Engine project, it's probably a good time to move to Python 2.7.

I'll explain briefly how you start a new project and there's a nice clean copy of the code at the bottom that you can cut and paste.

Let's get the show on the road. Choose a name for the project and create and switch to a virtual environment:

PROJ=py27_gae_project
mkvirtualenv ${PROJ}
cdvirtualenv

Note that note that "--no-site-packages" and "--distribute" are now the defaults for mkvirtualenv. You don't even need to use "--python=python2.7" on Ubuntu 12.04.

Now we need to know what the latest version of App Engine is, but as of writing it's 1.6.5:

GAE=1.6.5
wget -O /tmp/gae.zip http://googleappengine.googlecode.com/files/google_appengine_${GAE}.zip
unzip /tmp/gae.zip

Now let's create an App Engine app. The app will need a name that has been created in the App Engine Console:

GAE_APP_NAME=dummy
mkdir -p gae_app/static

Now create the app.yaml file:

echo """application: ${GAE_APP_NAME}
version: development
runtime: python27
api_version: 1
threadsafe: true

default_expiration: 7d

handlers:
- url: /static
  static_dir: static
- url: .*
  script: wsgi_app.app
""" > gae_app/app.yaml

And finally the app itself:

echo """import webapp2

class MainPage(webapp2.RequestHandler):
  def get(self):
      self.response.headers['Content-Type'] = 'text/plain'
      self.response.out.write('Please replace me with a decent WSGI App Framework such as Flask')

app = webapp2.WSGIApplication([('/', MainPage)],
                              debug=True)
""" > gae_app/wsgi_app.py

And finally to run the development server:

python ./google_appengine/dev_appserver.py gae_app/

I hope that this has all been of some help to you. Did I miss anything? Please comment below.

Generating Gibberish in Python - Markov Chains

Markov Chains allow us to generate letter sequences which can contain sensible keywords and bulk up your blog. 

I was once taught to speed read. A sham, a party trick, but interesting too. Essentially you just have to practice gliding over pages in a book or whatever and pick out key information (as opposed to just words). You do this by focusing on starts of paragraphs, which tend to be information rich, and capitalized words which tend to be names and useful information and ignore the rest.

What I'm suggesting here is almost the opposite, generating key words which are beefed up by noise in such a way that, I believe, most search engines will not see the join.

Markov Chains have been around for the most of a hundred years and work by splitting a text into groups of letters of a fixed length and record which groups follow that group anywhere in the text. To generate gibberish all you have to do is choose any group of letters and randomly pick one of the other groups that you know can follow that group; loop until you get enough text. If you use short groups of letters (say two) the text contains less information from the original text than if you use longer groups (say six), because with short groups there are more choices available. You should try this yourself to see what I mean, but I find that a group length of about four is a good start.

Now to the code. Firstly I would love to see your improvements, so please let me know or branch it from GitHub. And don't forget to read the extra information at the bottom of this page to understand how it works.

Essentially you just have to practice gliding over pages in a book or whatever and pick out key information (as opposite, generate gibberish all you use short groups of letters of the join. Firstly ins have been around for the most of a hundred years and use longer groups (say six), because obviously there are more choices available. Essentially you just have to do is choose any group of letters and randomly pick one of the other groups that you know can follow that groups of letters of a fixed length and randomly pick one of a fixed length and randomly pick one of a fixed length and randomly pick one of the code. 

Chains have been around for the most of a hundred years and words which are beefed up by noise in such a way that, I believe, most the join. GitHub op until you have to do is choose any group of letter sequences which can contain sensible keywords and capitalized words which groups follow that groups of letters (say two) the text contains less information from the original text that you know can follow that groups of letters (say two) the text contains less information from the original text. 

What up by noise in such a way that, I believe, most the other groups than if you get enough text than if you use short group anywhere information rich, and capitalized words which tent to the opposed to just words which tent to be names and work by splitting a text into groups of paragraphs, which tend to be information from the original text than if you use short group anywhere in the most of a hundred years and words which are been around for the most of a hundred years and work by splitting a text into group; loop until you have to do is choose any group of letter sequences which can contain sensible keywords and bulk up your blog. Chains have beefed up by noise in such a way that, I believe, most the opposed to just words which groups (say six), because obviously there are more choices available.

Get Real Data from the Semantic Web - Finding Resources

In my last article, I briefly explained how to get data from a resource using python and SPARQL. This article explains how to find the resource in the first place.
Have you ever been taught how to knit? I you have, then you'll know that you are not usually taught how to cast on (or start off) on your first lesson. That's because it much easier to learn how to knit than it is to cast on.

So it is with the Semantic Web. Once you have a resource URL, it's reasonably easy to extract information linked to that resource, but finding the starting resource is a bit trickier.
So let's just recap how we might get the abstract description for London from DBpedia.

If we know the URL then that's pretty straight forward:
(If you want to follow this tutorial, then you had better copy the sparql.py file from there.)

RDF types for the DBpedia entry for London

If you don't however, then you'll have to search for it. According to the dbpedia entry, London is many things, including a owl:Thing, there are a lot of Things out there, probably enough to make even the DBpdia  endpoint time out, so let's choose something more restrictive such as yago:Locations but not too restrictive, for example yago:BritishCapitals.


Just to be a smart ass as I finish off, you can get both at the same time by doing this, but don't forget that doing this will stress the SPARQL endpoint more than is probably necessary. Be kind.

Get Real Data from the Semantic Web

Semantic Web this, Semantic Web that, what actual use is the Semantic Web in the real world? I mean how can you actually use it?

If you haven't heard the term "Semantic Web" over the last couple of years then you must have been in... well somewhere without this interweb they're all talking about.

Basically, by using metadata (see RDF), disparate bits of data floating around the web can be joined up. In otherwords they stop being disparate. Better than that, theoretically you can query the connections between the data and get lots of lovely information back. This last bit is done via SPARQL, and yes, the QL does stand for Query Language.

I say theoretically because in reality it's a bit of a pain. I may be an intelligent agent capable of finding linked bits of data through the web, but how exactly would you do that in python.

It is possible to use rdflib to find information, but it's very long winded. It's much easier to use SPARQLWrapper andin fact in the simple example below, I've used a SPARQLWrapperWrapper to make asking for lots of similarly sourced data, in this case DBPedia, even easier.

To use this try importing the DBpediaEndpoint and feeding it some SPARQL:

Your homework is - How do you identify the resource_uri in the first place?

That's for another evening.

Google App Engine Python2.5 Development in Ubuntu 11.10

How to set up a development environment for Google App Engine, Python2.5 in Ubuntu 11.10.
Firstly, sorry if the code below is badly formatted, but there's a clearer copy of the code at the bottom.

Python2.5 isn't in the Ubuntu 11.10 sources by default so in a console:

sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update

and install python2.5 and some other stuff:

sudo apt-get install python2.5 python-virtualenv virtualenvwrapper python-pip

Next for project 'oinkyoinker':

export CURRENT=oinkyoinker
mkvirtualenv --no-site-packages --distribute --python python2.5 ${CURRENT}
workon ${CURRENT}
cdvirtualenv
pip install fabric yolk ipython readline

Now download GAE and fix the path:

wget -O /tmp/gae.zip http://googleappengine.googlecode.com/files/google_appengine_1.5.5.zip
unzip /tmp/gae.zip
echo "../../../google_appengine" > lib/python2.5/site-packages/gae.pth

and create a simple file server app:

mkdir -p application/static

echo """application: oinkyoinker
version: 1
runtime: python
api_version: 1
default_expiration: "7d"
handlers:
- url: /
  static_dir: static
""" > application/app.yaml

and run it:

./google_appengine/dev_appserver.py application/

Hope this helped you too.

MapReduce in Python

Over the last while I've been using NodeJS, MongoDB and Google's App Engine. Everything seems to have MapReduce functionality and frankly I was missing it when I went back to do some work in plain old Python. This is a nice short example of how it can be done.

The basic format that you need is something like this:

Now, a simple working example using the usual MapReduce example:

The output will look something like this:

• I got most of this from the following blog: http://goo.gl/nW8iA .
• Docs are here: http://docs.python.org/library/multiprocessing.html

My Top Ten Python Web Frameworks

An updated version of this article can be found here.

I have recently revisited all of the Python Web Frameworks that I know of and re-evaluated them for use in a new project.

I did this first two or three years years ago and so I imagine I will be biased by the decisions that I made then, so I think it's only fair to say that I started using Django because of it's fantastic documentation and I suppose the helper scripts which help the beginner along. However, all that Convention over configuration stuff can be too much for me sometimes.

I then moved onto Tipfy because it slotted in so easily to Google App Engine which I was using a lot at the time  and the Django framework itself was starting to be more of a hindrance than a help.

One day however I stumbled upon Bottle which was a revelation. Being a micro framework it's explicit. That is if you want to use something, you make up your mind to do so; it's not some cleverness built into the framework over which you have no obvious control. If you want to know how it works, then read the code!

Finally I came to Flask, another micro framework, because I like the way it is based on Werkzeug and other well trusted libraries which gives me a some peace of mind that it is supportable, and secondly because it is also explicit like Bottle.

All that being said, here's my top ten:

1. Flask - small, fast, easy to learn and built on reassuringly supported libraries.
2. Bottle - small, fast and a great tool for learning.
3. Tipfy - built for GAE, easy to pick up and well supported.
4. Django - contains most of what you will ever need to build a website, well documented, but starting to bulge at the wasteline.
5. Pyramid - the successor to Pylons, this one has great docs and great ideas. It is a framework, but you can unpick the bits you don't like.
6. CubicWeb - like Django, Convention over configuration, but nice in some ways. 
7. GAE framework - Simple, built for GAE, and well documented.
8. Web2py - I hate it, but love it. All that framework kinda helps, kinda hinders. Give it a little time and you can definitely pull a website together, it depends on your personality whether or not you go mad first though.
9. Nagare - I admit I haven't built a full system with this one, but I have a feeling I will be soon. I'd rather it was a wee bit more mature though, but the libraries/concepts it's built on is all good stuff.
10. Weblayer - Another micro framework. Some good ideas here, but could do with a little love, If you are a developer looking for a worthwhile project, you should consider lending a hand.

[Please note I am currently updating this article here]

For more information, please read:

• http://wiki.python.org/moin/WebFrameworks
• http://en.wikipedia.org/wiki/Comparison_of_Web_application_frameworks

Adding Shell Access to Our Flask-Script Example

The starting example of how you can use flask-script was given in my earlier article;
A Simple Flask-Script Example.

Next thing we are going to add is the ability to run a shell which that shares your applications environment. You need to decide what you ant to import and put it in the shell_context function. Call it whatever you want, but it has to return a dict with your context items inside. To keep it simple I just import the app itself. The lines you need to change are:

from flaskext.script import Manager, Server, Shell

and

def shell_context():
    return dict(app=app)            
            
if __name__ == "__main__":
    manager.add_command('dev', DevServer())
    manager.add_command('test', Test())
    manager.add_command('zen', ZenTest())
    manager.add_command('shell', Shell(make_context=shell_context))
    manager.run()

now test it:

python manager.py shell

>>> app.debug
True
>>> app.logger
<flask.logging.DebugLogger instance at 0x2ee3518>

and so on.

There's one final installment to follow - Let Your Imagination Run Riot On Our Flask-Script Example


The full listing for what we did here is below:

from flaskext.script import Manager, Server, Shell
from flaskext.zen import Test, ZenTest
import application


app = application.create_app()

manager = Manager(app)

class DevServer(Server):
    
    def handle(self, app, host, port, use_debugger, use_reloader):
        try:
            from flaskext.lesscss import lesscss
            lesscss(app)
        except: pass

        app.run(host=host,
            port=port,
            debug=use_debugger,
            use_debugger=use_debugger,
            use_reloader=use_reloader,
            **self.server_options)
    
def shell_context():
    return dict(app=app)            
            
if __name__ == "__main__":
    manager.add_command('dev', DevServer())
    manager.add_command('test', Test())
    manager.add_command('zen', ZenTest())
    manager.add_command('shell', Shell(make_context=shell_context))
    manager.run()

A Simple Flask-Script Example

Here is a simple flask-script example that you can use to start your application in development mode, or test modes.

I know that you can do that without flask-script, but the point of this script is that you can do so much more from this starting point. Notice for example that I'm trying to load flask-lesscss and execute it (which won't work, but won't fail if you don't have lesscss installed.)

To get the application (in the 'application' module) running you just need to do one of the following:

python manager.py dev
python manager.py test
python manager.py zen

I show how to add shell access in the second article in this series. Meanwhile why not let us see what you have in your manager script?

from flaskext.script import Manager, Server
from flaskext.zen import Test, ZenTest
import application


app = application.create_app()

manager = Manager(app)

class DevServer(Server):
    
    def handle(self, app, host, port, use_debugger, use_reloader):
        try:
            from flaskext.lesscss import lesscss
            lesscss(app)
        except: pass

        app.run(host=host,
            port=port,
            debug=use_debugger,
            use_debugger=use_debugger,
            use_reloader=use_reloader,
            **self.server_options)
            
            
if __name__ == "__main__":
    manager.add_command("dev", DevServer())
    manager.add_command('test', Test())
    manager.add_command('zen', ZenTest())
    manager.run()
    

Install Git, Pip and VirtualEnv and VirtualEnvWrapper on Ubuntu in a Minute

From a terminal, type:

sudo apt-get install git python-setuptools
sudo easy_install pip
sudo pip install virtualenv virtualenvwrapper

Now, you may want to set some defaults in your ~/.bashrc. My relevant entries look like this:

export WORKON_HOME=$HOME/Projects
export VIRTUALENVWRAPPER_HOOK_DIR=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_LOG_DIR=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--no-site-packages --distribute'
source /usr/local/bin/virtualenvwrapper.sh

git config --global user.name "Terse Col"
git config --global user.email myemail@gmail.com
git config --global http.sslVerify false

Now away you go:

mkdir $HOME/.virtualenvs
source $HOME/.bashrc
mkvirtualenv mynewproject 
cdvirtualenv