Terse Words

Generating Gibberish in Python - Markov Chains

2012-02-27T15:10:00.000-08:00

Markov Chains allow us to generate letter sequences which can contain sensible keywords and bulk up your blog.

I was once taught to speed read. A sham, a party trick, but interesting too. Essentially you just have to practice gliding over pages in a book or whatever and pick out key information (as opposed to just words). You do this by focusing on starts of paragraphs, which tend to be information rich, and capitalized words which tent to be names and useful information and ignore the rest.

What I'm suggesting here is almost the opposite, generating key words which are beefed up by noise in such a way that, I believe, most search engines will not see the join.

Markov Chains have been around for the most of a hundred years and work by splitting a text into groups of letters of a fixed length and record which groups follow that group anywhere in the text. To generate gibberish all you have to do is choose any group of letters and randomly pick one of the other groups that you know can follow that group; loop until you get enough text. If you use short groups of letters (say two) the text contains less information from the original text than if you use longer groups (say six), because with short groups there are more choices available. You should try this yourself to see what I mean, but I find that a group length of about four is a good start.

Now to the code. Firstly I would love to see your improvements, so please let me know or branch it from GitHub. And don't forget to read the extra information at the bottom of this page to understand how it works.

Essentially you just have to practice gliding over pages in a book or whatever and pick out key information (as opposite, generate gibberish all you use short groups of letters of the join. Firstly ins have been around for the most of a hundred years and use longer groups (say six), because obviously there are more choices available. Essentially you just have to do is choose any group of letters and randomly pick one of the other groups that you know can follow that groups of letters of a fixed length and randomly pick one of a fixed length and randomly pick one of a fixed length and randomly pick one of the code.

Chains have been around for the most of a hundred years and words which are beefed up by noise in such a way that, I believe, most the join. GitHub op until you have to do is choose any group of letter sequences which can contain sensible keywords and capitalized words which groups follow that groups of letters (say two) the text contains less information from the original text that you know can follow that groups of letters (say two) the text contains less information from the original text.

What up by noise in such a way that, I believe, most the other groups than if you get enough text than if you use short group anywhere information rich, and capitalized words which tent to the opposed to just words which tent to be names and work by splitting a text into groups of paragraphs, which tend to be information from the original text than if you use short group anywhere in the most of a hundred years and words which are been around for the most of a hundred years and work by splitting a text into group; loop until you have to do is choose any group of letter sequences which can contain sensible keywords and bulk up your blog. Chains have beefed up by noise in such a way that, I believe, most the opposed to just words which groups (say six), because obviously there are more choices available.

Get Real Data from the Semantic Web - Finding Resources

2012-01-22T15:14:00.000-08:00

In my last article, I briefly explained how to get data from a resource using python and SPARQL. This article explains how to find the resource in the first place.
Have you ever been taught how to knit? I you have, then you'll know that you are not usually taught how to cast on (or start off) on your first lesson. That's because it much easier to learn how to knit than it is to cast on.

So it is with the Semantic Web. Once you have a resource URL, it's reasonably easy to extract information linked to that resource, but finding the starting resource is a bit trickier.
So let's just recap how we might get the abstract description for London from DBpedia.

If we know the URL then that's pretty straight forward:
(If you want to follow this tutorial, then you had better copy the sparql.py file from there.)

RDF types for the DBpedia entry for London

If you don't however, then you'll have to search for it. According to the dbpedia entry, London is many things, including a owl:Thing, there are a lot of Things out there, probably enough to make even the DBpdia endpoint time out, so let's choose something more restrictive such as yago:Locations but not too restrictive, for example yago:BritishCapitals.

Just to be a smart ass as I finish off, you can get both at the same time by doing this, but don't forget that doing this will stress the SPARQL endpoint more than is probably necessary. Be kind.

Get Real Data from the Semantic Web

2012-01-19T13:53:00.001-08:00

Semantic Web this, Semantic Web that, what actual use is the Semantic Web in the real world? I mean how can you actually use it?

If you haven't heard the term "Semantic Web" over the last couple of years then you must have been in... well somewhere without this interweb they're all talking about.

Basically, by using metadata (see RDF), disparate bits of data floating around the web can be joined up. In otherwords they stop being disparate. Better than that, theoretically you can query the connections between the data and get lots of lovely information back. This last bit is done via SPARQL, and yes, the QL does stand for Query Language.

I say theoretically because in reality it's a bit of a pain. I may be an intelligent agent capable of finding linked bits of data through the web, but how exactly would you do that in python.

It is possible to use rdflib to find information, but it's very long winded. It's much easier to use SPARQLWrapper andin fact in the simple example below, I've used a SPARQLWrapperWrapper to make asking for lots of similarly sourced data, in this case DBPedia, even easier.

To use this try importing the DBpediaEndpoint and feeding it some SPARQL:

Your homework is - How do you identify the resource_uri in the first place?

That's for another evening.

Github: Who needs it?

2012-01-17T14:13:00.001-08:00

Do you ever think that you just don't want all your code on Github? I mean it's only a quick hack right?

Truth is, once you start using git you probably use it automatically for all your code, but you don't always want all your code floating around the net. What about those hard-coded email addresses and API tokens, or those references to your private net servers?

The answer is probably so simple that you have just overlooked it. You don't need to set up a local git server or hire one from Amazon. All you need to do is use DropBox or Ubuntu One as your remote origin repository.

Here's how, using Ubuntu One on Ubuntu:

Write a short shell script something like this and save it on your path as repo.sh.

Now when you want to create a new repository all you have to do is:

If you use Python and virtualenv you may be interested in the slightly extended script at http://pythonic-apis.blogspot.com/2012/01/using-ubuntu-one-as-git-repository.html.

Twitter Streaming API - Almost Useful

2011-12-15T08:20:00.000-08:00

Twitter's Streaming API is a splendid idea. It gives developer's access to a good splodge of Tweets and let's us filter them in different ways.

However it has a few flaws.

For Twitter, the benefit of a Streaming API is probably one of scalability. Instead of us using the old REST API to ask for specific data and causing tens of thousands of data look ups, all they do is give us the end of their own data stream once it has been used in house and is now halfway across the back garden. All they have to do is allow us to filter the stream a bit to make it a bit more relevant to our needs and put an absolute cap on the throughput (about 1% for most of us.)

This looks good. 1% is enough for most development needs and streams down your connection like a low bandwidth radio station. I don't really know what the bandwidth or download is, but it's not much. Once we've developed our new and wonderful website, then we can ask, or possibly pay, Twitter to turn up the pressure a bit.

So, now let's look at the filters.

There are several ways that the stream can be filtered

follow - filter by userid
track - filter by keyword
location - filter by geographic location
retweets - just the retweets ma'am
links - only tweets containing a link
random - I think they just mean unfiltered

On the face of it this is pretty useful. You could filter the stream by adding the location of your home town so that you can get the jist of what is important to your townsfolk. You could filter the stream by the keyword 'Elvis' to see who has spotted Elvis lately and plot the results on a map.

My own first idea was inspired by the M5 motorway accident just a few miles from where I live and astounded that even in this day and age, the scale of the incident was only uncovered somewhat slowly. Surely what the quantity and content of the tweets from the people who were NOT in the incident itself, would help scale the incident? So what I wanted to do was:

Listen to what people Tweet at known traffic jam locations.
Identify some fingerprint of common words, maybe "traffic, jam, standstill, miles" or whatever.
Look for clusters of these words near to motorways.
Plot the clusters based on the location of the phones that made the tweets.

So I get an updating list of current traffic incidents from the Traffic England RSS feed and start listening within 5 kms of the stated location using a bounding box centred on the location. The first thing I notice is that some of the tweets are in the bounding box, some are around it and some, quite a few are far away, sometimes 200km away or more!

I don't know how Twitter do the filtering, but evidently it's based on something fairly broad brushed. I can live with that maybe, all I have to do is check the Tweets geo location, which is added if you tweet by most modern phones. I was expecting most of the useful tweets to be from a mobile anyway, so that would work if I can just get used to maybe 2% of the tweets actually being in the bounding box. 2% 0f 1% is after all only 0.02% of all Tweets or 1 Tweet in 5000.

So what happens if I assume that the word "traffic" will occur in the most useful Tweets? This is either bad science or common sense data filtering depending on how you look at it.

Alas, it appears that the Streaming API does not allow you to filter by location AND keyword! All you can do is do an OR filter, so I can filter the stream to include certain areas OR certain keywords, but not certain keywords within a certain location.

To me this just renders the API all but useless, but no doubt you lot are much smarter than I and will dazzle me with your great ideas.

Please let me know.

Google App Engine Python2.5 Development in Ubuntu 11.10

2011-11-06T15:07:00.000-08:00

How to set up a development environment for Google App Engine, Python2.5 in Ubuntu 11.10.
Firstly, sorry if the code below is badly formatted, but there's a clearer copy of the code at the bottom.

Python2.5 isn't in the Ubuntu 11.10 sources by default so in a console:

sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update

and install python2.5 and some other stuff:

sudo apt-get install python2.5 python-virtualenv virtualenvwrapper python-pip

Next for project 'oinkyoinker':

export CURRENT=oinkyoinker
mkvirtualenv --no-site-packages --distribute --python python2.5 ${CURRENT}
workon ${CURRENT}
cdvirtualenv
pip install fabric yolk ipython readline

Now download GAE and fix the path:

wget -O /tmp/gae.zip http://googleappengine.googlecode.com/files/google_appengine_1.5.5.zip
unzip /tmp/gae.zip
echo "../../../google_appengine" > lib/python2.5/site-packages/gae.pth

and create a simple file server app:

mkdir -p application/static

echo """application: oinkyoinker
version: 1
runtime: python
api_version: 1
default_expiration: "7d"
handlers:
- url: /
  static_dir: static
""" > application/app.yaml

and run it:

./google_appengine/dev_appserver.py application/

Hope this helped you too.

MapReduce in Python

2011-11-06T13:14:00.000-08:00

Over the last while I've been using NodeJS, MongoDB and Google's App Engine. Everything seems to have MapReduce functionality and frankly I was missing it when I went back to do some work in plain old Python. This is a nice short example of how it can be done.

The basic format that you need is something like this:

Now, a simple working example using the usual MapReduce example:

The output will look something like this:

I got most of this from the following blog: http://goo.gl/nW8iA .
Docs are here: http://docs.python.org/library/multiprocessing.html

MapReduce with Mongoose and CoffeeScript

2011-10-19T06:32:00.000-07:00

After searching the InterWeb for a decent MapReduce example coded in CoffeeScript I came up blank and decided to write my own. This one uses Mongoose too - well why use anything else?

I haven't written a whole lot of explanation in the text, but commented the code quite heavily so you can copy and paste and still have it with you.

Load Mongoose and connect to your database.Describe your schema and model and then load up some dummy data.

Now for the MapReduce stuff. If you still don't get MapReduce then you will once you read this Star Trek based article.

Define a Map function.

Define a Reduce function.

Get some output

The results will now be in a collection called "results" which you can look at with JMongoBrowser or whatever.

I hope this is of some use to you. If there are any corrections or improvements then please post something below.

Six Ways to Have a Lazy Day at the Office

2011-10-18T05:23:00.000-07:00

Sometimes you just need a lazy day at the Office. Now I'm not saying that I am lazy, but I have built up a certain amount of useful expertise in this area. Sometimes it's tiredness, sometimes I'm hungover, but mostly I'm just still drunk.

Either way I hope these are of use to you.

Leave a Post-it saying that you have gone for a meeting with your new boss. If your current boss asks where you were just say you 've been asked not to say until "things are sorted out". They'll be too worried about their own job to worry about what you've been up to.
Sit in a café all day sending the occasional email from your smart phone letting people know you are meeting a client.
Sit in the smoker's shed smoking one of those electronic cigarettes. Occasionally switch sheds; smokers are people of habit and always use the same shed.
When you need to have a lie down, then just go ahead and lie on the floor complaining of "the old slipped disk". Don't snore.
Book a meeting room and send an invite to a manager that you know isn't in today. Take some paperwork along and if asked just say that you thought you'd use the time while waiting to catch up on some documents.
Before you disappear from your desk tell your neighbours that your having an affair with the bosses partner. They will do all the lying for you.

Ten Annoying Things About IT Contractors

2011-10-17T12:14:00.000-07:00

OK, so I'm bitter and twisted. I've never been brave enough to leave my long term job and take the risk. Even bumbling along like I do, you eventually end up working alongside; hiring, and managing IT contractors and you spot some peculiarities. Strange scary (or at least annoying) peculiarities.

Suits

Nobody needs a suit that sharp to meet our customers and certainly not to go to internal meetings. It's just not fair to show up my Hawaii shirt. It deserves respect. You seem to forget that once it still fitted and had all the buttons.
Your code is shit.
I'm signing your damn hours so please stop calling me a "management overhead".
If you want to read a morning publication, read Dilbert like the rest of us, not the broad sheets and not that one in particular. Even more than that, reading the Financial Times/Wall Street Journal is just plain wrong.
You don't need to park your BMW next to my Rover. I'll have you know that Rover were the last great British car manufacturer and you can still find parts. Mostly in scrapyards admittedly, but all cheap.
It's just rude to send your CV to another company using our email system. I don't care if it is lunch-time.
Stop bringing your own laptop. Yes, it's nice and slim, but the company laptops improve your muscle tone and can store loads of stuff if you use a USB stick.
On doughnut Wednesday you just can't bring fruit and carrot sticks unless you want to be ridiculed. Gym-boy!
Dress down Friday doesn't mean wear a slightly cheaper suit, it means you don't need to wear pants at all.
Your code is still shit, but don't worry we'll fix it and write the documentation too. Enjoy your new job.

Comment below as you will.

Staying Up Late - CoffeeScript and Zappa

2011-10-17T07:17:00.000-07:00

Over the last couple of years NodeJS has become the darling of "coding for pleasure" crowd. All that cross fertilization from the client side buffs into the server side and the server side coders ranting about the whole event-driveness of NodeJS.

Yes, it's the next big thing. Well, it's probably not, but it's certainly "A" next big thing or maybe we'll all get bored dream up something else with a catchy name... "perl for parallelism" or "Friendly Java (without the memory leaks)", or what about "C+=3" or "Oil for Android", "Lube for Palm OS". Well you know what I mean.

We see something; we explore it and once it starts getting a bit boring we convince ourselves and then our customers that we need to use some new technology or other. It's all true of course, or partly true, or lies.

Anyway, I'm not here to rant about that, after all I as guilty as anyone else.

A Messy Affair

What I want to rant about is Zappa which is my own personal cure for technical disfunction at the moment. Well that sounds real bad, but I've been doing this job for years and it's only these little affairs over the years that keep me coming home; "Assembler", "C", "C++", "Ruby", "Python", how I remember their sweet names, but sooner or later they all start to look like their mother and have to go.

Zappa is a young virgin of a technology, only now reaching a pubescent version 0.3. Essentially it's a lightweight framework built on top of known technologies such like NodeJS, CoffeeScript, Express and Socket.IO in much the same way as Python's Flask is built on top of Werkzeug, Jinja 2 and so on.

It's that Socket.IO bit that makes this one so sexy (I'm overdoing the procreation metaphors aren't I?). Socket.IO provides cross-browser WebSocket goodness which means you can do stuff like stop using the Model-View-Controller paradigm and start thinking outside the cubical with your "Model-View-Controller-View-Controller-Controller-View Paradigm" or whatever. Mind you, that's when you'll realise that MVC wasn't about limiting your options, it was about simplifying them.

Anyway, back to Zappa itself. Go and have a look, there are great examples that will get your inventive juices flowing (stop it!) and inspire you to spend a little time alone locked it the bathroom... I mean office. The community is starting to grow and the mailing list is big enough to be useful, but not so big as to be faceless and scarey.

Have fun but take precautions.

Back of an Envelope Node.js Framework - Discuss

2011-07-13T08:09:00.000-07:00

I had a quiet afternoon today and while "keeping up with recent developments in my field" or as you might call it "wasting my afternoon googling" I got to thinking what I'd like a Node.js MVC framework should or could contain.

How design shouldn't be done

Firstly, I've been playing with Node.js for about three weeks now and I'm impressed, but I can't seem to find a framework that I'm happy with. Backbone looked good, but I wasted three days on it and it was too much like hard work. Similarly Spine which tried to be less complicated, but eventually failed. Finally I really wanted Brunch to work but it was a release behind backbone (which is a major component of it) and I didn't want to get lost in the mismatching documentation (the Backbone release was pretty major).

I have spent a whole lot of time over the last couple of years using python frameworks, especially Flask and I like their approach of starting with a good foundation (in their case Werkzeug) and adding the bits that are still needed. Could that work with Node? I mean writing a basic HTTP server looks pretty straight forward and there are tons of modules to add the other functionality.

So with this possibly naive starting position I grabbed my pen and scribble down some stuff I thought I'd need. Now bear in mind that I haven't much used any of these modules yet, and I pinched several of them from the bits that go to make up Brunch.

I'd appreciate any feedback, suggestions and so on that those more learned than I might care to offer. Few of you will be less learned I imagine.

Here are the transcribed notes with some links:

Server

Simple Node.js
Express

Routing

choose a module
Express?

Data Store

Models

JS Object/JSON?
ORM?

Views
-CSS

lesscss ?
Stylus ?

-Templating

mustache(js) ?
Eco (coffeescript)

Controllers

Plain Old JS
CoffeeScript?
jQuery
Underscore.js
Quip (responses) ?

Middleware

how? Express

Other Stuff

If you spot any typos, inaccuracies or just plain lies then please let me know.

Binding Updates to Model Changes in Backbone/CoffeeScript

2011-06-27T14:37:00.000-07:00

My previous post showed a View that looked like this:

But what happens if we have a more complicated app and something else changes the Model?

Well as I said I'm just learning myself here, co the correct way that should be handled is to bind Model changes to the Views render function which looks like this:

Note the bind instruction which means that every time the counter changes, it calls the render function. You might also notice the fat arrow now on the render which make "this" still mean the View rather than the caller (which would be the Model otherwise.)

The explanation for all this is at http://jashkenas.github.com/coffee-script under "Function Binding".

Simplest Ever Backbone.js/CoffeeScript Example

2011-06-26T13:01:00.000-07:00

I'm teaching myself to use Backbone because after having done quite a lot of JavaScript in work lately, it has become obvious to me that I need a framework.

I'm also learning how to use CoffeeScript, which is close enough to python/ruby for me to understand.

First thing that I noticed is that most of the examples for Backbone out there are in pure JavaScript, so I thought I'd do my bit by providing something in CoffeeScript.

To that end, here is a very simple app which displays a number. You can increase or decrease the count by clicking buttons.

The app.coffee code:

and the html to go with that:

In detail

At the top, we set up global hashes to store our objects. In a bigger app this would make referencing bits and pieces much easier, although we don't really need it in this example.

Next the controller. This one is almost empty except it has two routes both taking you to the home control which is also part of the controller. The route with no reference (the empty string) and the one for #home tagged on the end of your URL (by way of an example) both call the home function in this case.

The home control just tells the view to render itself.

Next the Counter model which has one attribute called count which defaults to zero. In reality I think you would want to synchronize your models with a database, but we'll do that in another post soon.

And now for the View. The View is obviously what you are seeing in the browser. In our trivial example we only have one view, but a real app would probably have several. A view lumps together a unit of the app, sometimes nowadays we call these widgets, or blocks or display regions or whatever. You get the idea.

In the initialize function, we just connect the view to the model that backs it. The events connect the button to the functions that they call. The render function draws the result into the #count div; the @ symbol at the end is shorthand for this. In CoffeeScript (like Ruby) the last line of a function gets returned even if you don't specifically say "return". So in other words it means "return this". This is handy since it lets us chain together functions.

The inc and dec functions increase and decrease the count in the model and then update the display. Please see the update here.

Finally the equivalent of the main() function which instaniates the bits and bobs and then starts the history. I've no idea why they called it history, but basically it starts up all the listeners.

That's all for now. Next time I'll show you how to connect all this to your database using python and Flask.

If you see anything wrong with what I've written here or if you spot spelling or other mistakes then please let me know.

Set Up Your Own Git Repository Hosting Service in Three Minutes

2011-06-25T12:15:00.000-07:00

You might have seen my earlier post "A Review of Free Code Repositories".

Well I forgot about hosting your own. Really handy for hobbyist development.I have one of those free developer servers from Amazon out there somewhere. I don't use it much except to demo the occasional site and so on.

I decided that it was a good idea to set it up as a git repository and i show you here how to do this.If you are using something Ubuntu-ish remotely then this should take about three minutes.

A Word of Explanation
This is the simple setup, it avoids creating 'git' user on the remote server, but you DO need to use a seperate key for using the git service and I show you that first.

On your local machine:


# echo "Host gitserver
  Hostname ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com
  User git-col
  PreferredAuthentications publickey
  IdentityFile ~/keys/git-col.pub
" >> ~/.ssh/config

# mkdir ~/keys

# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/col/.ssh/id_rsa): git-col.pub # do NOT accept the default

# cp git-col* ~/keys
# scp git-col.pub ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:
# ssh 
ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com

On your remote host you are now logged in as 'ubuntu':

# sudo apt-get install git
# git clone git://github.com/sitaramc/gitolite
# ./gitolite/src/gl-system-install 
# gl-setup -q ~/git-col.pub 
# exit

On your local machine:

# git config --global user.name "Your Name"
# git config --global user.email your@email.address
# git clone gitserver:gitolite-admin
# exit

That's it really.To create a new repository called dummy and initialize it, you can do something like this.

On your local machine:

# set the new repository name
export repo=dummy

# update the gitolite conf file with your new repository
cd ~/gitolite-admin
echo "        repo    ${repo}" >> conf/gitolite.conf
echo "                RW+     =   @all" >> conf/gitolite.conf

# push the config to your server
git commit -m "added new repository ${repo}" conf/gitolite.conf
git push 

# make sure you cd to an existing directory!
mkdir ~/Projects
cd ~/Projects

# pull back the new empty repository
git clone gitserver:${repo}

# go to repository and make any change
cd ${repo}
echo "*.pyc" >> .git/info/exclude # I use python
touch README

# do your first push back to master
git add .
git commit -am "repository ${repo} pushed to origin on gitserver"
git push origin master

You'll be using that last code section again, so you might want to make it into a script.

Never type anything you don't understand.

If I've made any mistakes then please let me know.

Flask-Coffee - Fill your flask with coffee

2011-06-21T15:42:00.000-07:00

Flask-Coffee is a Flask extension that compiles .coffee CoffeeScript files into .js JavaScript files for you if they have changed before your app renders.

This is a nice idea in development and not so nice an idea in production, so please keep that in mind.

Prerequisites

You will need to have to have Node.js installed with NPM and coffee-script.

You can follow the instructions in my previous article to install Node.js and then it's just

npm -g install coffee-script

Installation

Install Flask-Coffee with pip:

pip install flask-coffee

Usage

from flaskext.coffee import coffee

coffee(app)

This will watch your app’s static media directory and automatically render .coffee files into .js files in the same (sub)directory.

The best way to incorporate Flask-Coffee into your development is as described for flask-lesscss in my earlier flask-script tutorial.

Contribute

If you want to contribute email me at the email at http://pypi.python.org/pypi/Flask-Coffee after checking out the code at http://bettercodes.org/projects/flask-coffee.

CoffeeScript Development in 3 mintutes

2011-06-20T04:58:00.000-07:00


Any obvious reference to coffee needs a picture

The idea of CoffeeScript just sounds great me. I like the idea of compiling decent JavaScript from code you can actually read. But what do you do if you have no Node.js available to compile it?

Well the answer is to compile to JS at runtime. However, I had an awful time getting the right setup to get going. If you are interested in playing with CoffeeScript where Node.js is not available (I'm using it in work to show what it's capable of) then just follow these instructions.

Create a directory to work in and inside that create an index.html file thus:

<!DOCTYPE html>
<html>
<head>
    <title>CoffeeScript Demo</title>
    <script data-main="scripts/requirements" 
               src="scripts/require.js"></script>
</head>
<body>
    <h1>CoffeeScript Demo</h1>
</body>
</html>

and a directory called 'scripts' into which you put the following.

JQuery
Require
CoffeeScript Plugin For Require
A file called requirements.js containing:

require([
    'scripts/jquery-1.6.1.min.js',
    'cs', 
    'cs!app',
]);

And a file called app.coffee containing:

square = (x) -> x * x
alert square 42

That's it!

Now load the index.html page into your browser and the CoffeeScript code app.coffee should execute.

Of course you will now have to go away and read the docs so that you know why that worked, but then so will I.

Flask Extensions For Authorization with Examples

2011-06-15T06:30:00.000-07:00

In my opinion, there is one serious omission to Flask at the moment and that is that there is no authoritative method for authorizing users.

In a way this goes with the general ethos of micro frameworks - it isn't being forced down your throat unless you want it.

However in this case I think that at least some structured 'recipes' are required to provide peer guidance, best practice or whatever you want to call it. Too often the advice is "roll your own", but I'm not sure I'd be comfortable recommending that to the inexperienced.

So where are we?

Flask-Principal and Flask-Login

At the moment there is only one approved auth extension called Flask-Principal which takes care of authorizing different roles of user. It does not dictate what those roles should be and it does not force you to use any particular method to authenticate those users.

A second extension called Flask-Login is currently awaiting approval, which I suspect it will soon get. This extension also takes care of authorizing users, but only understands one type of user role, so in other words you are either a user or not. Like Flask-Principal it does not force you to use any particular method to authenticate those users.

Example Code

The code below is sort of a basic outline that we will fill in using proper code in a moment.

The User model is very silly. If you create a User by passing an id of '1' you get a User with a username of 'user1' and a password of 'user1_secret'. To help with the examples we then create users one to twenty (user1 to user20). Obviously you would normally have them stored in a database or something, but we're keeping it simple.

At this stage there is nothing protecting access to the 'home' resource and 'login' and 'logout' don't work yet either.

from flask import Flask, Response, redirect, \
    url_for, request, session, abort

app = Flask(__name__)

# config
app.config.update(
    DEBUG = True,
    SECRET_KEY = 'secret_xxx'
)

# middlewares etc go here
      
# silly user model
class User(object):

    def __init__(self, id):
        self.id = id
        self.name = "user" + str(id)
        self.password = self.name + "_secret"        
    def __repr__(self):
        return "%d/%s/%s" % (self.id, self.name, self.password)


# create some users with ids 1 to 20       
users = [User(id) for id in range(1, 21)]


# we'd like to protect this resource
@app.route('/')
def home():
    return Response("Hello World!")


# somewhere to login    
@app.route('/login', methods=['GET', 'POST'])
def login():
    pass # this does not work yet

# somewhere to logout
@app.route("/logout")
def logout():
    pass # this does not work yet
    
        
# handle login failed
@app.errorhandler(401)
def page_not_found(e):
    return Response('Login failed')


@app.errorhandler(403)
def page_not_found(e):
    return Response('Unauthorized')


if __name__ == "__main__":
    app.run()

Download the code!

Flask-Principal Example
Now that we know how that works, or doesn't, let's look first at how flask principal might set about protecting the home resource.

As I said earlier, Flask-Principal doesn't tell you what roles you need for your app. In this example I'm only using one called 'normal' which we need to tell the extension about before we tell Flask to initialize the extension:

# flask-principal
principals = Principal()
normal_role = RoleNeed('normal')
normal_permission = Permission(normal_role)
principals._init_app(app)

After that we can protect the 'home' resource thus:

@normal_permission.require(http_exception=403)

If your request's session doesn't have all the stuff it's asked for (ie. has not logged in) then Flask-Principal gives it a HTTP 403 (Forbidden) response which redirects it to:

@app.errorhandler(403)
def page_not_found(e):
    session['redirected_from'] = request.url
    return redirect(url_for('login'))

Which, as you can see stores the URL that you wanted in the first place ('home' in this case) and then redirects you to the login page. Since by default this request is a GET, you are shown the login form and not the login process which is used for POSTs.

Enter a valid username/password combination, e.g. 'user7' and 'user7_secret' and you get logged in and sent back to the 'home' page. The login sends a signal to the extension to tell it who has logged in and congratulations you get to see the 'Hello World' message!

If you enter an invalid username/password combination you get served an HTTP 401 (Unauthorized) and thus you end up in the 401 errorhandler seeing 'Login Failed'

If you go to 'logout' you the code will destroy any of the relevent session information and tell you you are 'Logged Out':

@app.route("/logout")
def logout():
    for key in ['identity.name', 'identity.auth_type', 'redirected_from']:
        try:
            del session[key]
        except:
            pass
    return Response('Logged out
')

Try out the code!

Flask-Login Example
With Flask-Login you need to tell Flask to use the extension and this time tell the extension where the login page is (in this case "login"):

# flask-login
login_manager = LoginManager()
login_manager.setup_app(app)
login_manager.login_view = "login"

Secondly you have to use a mixin to create your User class. The mixin just makes some standard functions available for the User.

class User(UserMixin):

Thirdly, you need to set up a user_loader that will load your user based on the user id:

# callback to relaad the user object        
@login_manager.user_loader
def load_user(userid):
    return User(userid)

This time when you go to 'home' and you haven't yet logged in, you don't get a HTTP 403 (Forbidden), but instead get sent to the resource you defined earlier; 'login'.

Login successfully and this time the login retrieves the actual User object and tells the extension using the login_user function. Again, congratulations!

Try out the code!

You can work out for yourself how to store the User object in a database or datastore and how to check the passwords. There are a good ways to do both, but this is not an article about that. Should we have one in a future article?

My Top Ten Python Web Frameworks

2011-06-14T04:04:00.000-07:00

I have recently revisited all of the Python Web Frameworks that I know of and re-evaluated them for use in a new project.

I did this first two or three years years ago and so I imagine I will be biased by the decisions that I made then, so I think it's only fair to say that I started using Django because of it's fantastic documentation and I suppose the helper scripts which help the beginner along. However, all that Convention over configuration stuff can be too much for me sometimes.

I then moved onto Tipfy because it slotted in so easily to Google App Engine which I was using a lot at the time and the Django framework itself was starting to be more of a hindrance than a help.

One day however I stumbled upon Bottle which was a revelation. Being a micro framework it's explicit. That is if you want to use something, you make up your mind to do so; it's not some cleverness built into the framework over which you have no obvious control. If you want to know how it works, then read the code!

Finally I came to Flask, another micro framework, because I like the way it is based on Werkzeug and other well trusted libraries which gives me a some peace of mind that it is supportable, and secondly because it is also explicit like Bottle.

All that being said, here's my top ten:

Flask - small, fast, easy to learn and built on reassuringly supported libraries.
Bottle - small, fast and a great tool for learning.
Tipfy - built for GAE, easy to pick up and well supported.
Django - contains most of what you will ever need to build a website, well documented, but starting to bulge at the wasteline.
Pyramid - the successor to Pylons, this one has great docs and great ideas. It is a framework, but you can unpick the bits you don't like.
CubicWeb - like Django, Convention over configuration, but nice in some ways.
GAE framework - Simple, built for GAE, and well documented.
Web2py - I hate it, but love it. All that framework kinda helps, kinda hinders. Give it a little time and you can definitely pull a website together, it depends on your personality whether or not you go mad first though.
Nagare - I admit I haven't built a full system with this one, but I have a feeling I will be soon. I'd rather it was a wee bit more mature though, but the libraries/concepts it's built on is all good stuff.
Weblayer - Another micro framework. Some good ideas here, but could do with a little love, If you are a developer looking for a worthwhile project, you should consider lending a hand.

For more information, please read:

Let Your Imagination Run Riot On Our Flask-Script Example

2011-06-09T06:30:00.000-07:00

In my last article in this series, "Adding Shell Access to Our Flask-Script Example" I showed you how you could indeed get shell access from our script.

You see the whole point of Flask-Script is that it gives you scripted access not to the application, but to the environment in which the app runs. You don't actually need to be running the app, or importantly, to interfere with the app to do some task or other.

An Example
For example, lets say you want to load some data into the database model.

There are three ways you could do this:

Write a script that understands your app's data model and get it to insert the data. You may be able to import the model module straight from your app, but you may not, due to dependencies etc.
Run the code straight from your app, possibly using an admin screen somewhere. This means designing and implementing the screens - including the obvious security requirements.
Use Flask-Script like this:

from application.model import Articles
import csv


@manager.command
def ingest_csv():
    '''read data from a csv file and ingest it into the database'''
    reader = csv.reader(open('articles.csv'))
    for row in reader:
        a = Article()
        a.init_from_csv_row(row)

and then;

python manager.py ingest_csv

Simpler I think you'll agree?

Scheduling Task
There is one other huge advantage to using Flask-Script; you can run them from crontabs.

*/10 * * * * /some/path/bin/python /some/path/manager.py ingest_csv >>/tmp/ingest.log 2>&1

Other Ideas
Here are some other examples of things you might want to dovia script:

Purge old data from your database.
Backup data from your database.
Export data from your database.
Do anything else you want from your database!
Flush your caches.
Mail out reports.
Other stuff I haven't thought of. Let me know!

Okay, enough of all this, go and enjoy!

Install Postgres/PostGIS on Ubuntu 11.04 (Natty) or Mint 11

2011-06-08T12:37:00.000-07:00

One of my most visited entries on my old blog was "Install Postgres/PostGIS on Ubuntu Maverick/10.10". So I was keen to update what had obviously been quite a useful article.

After all it was only a couple of dozen lines of code.

Turns out that it's one of those things I know that will never be used again; like how to hotwire a 1972 Ford Cortina, or how to get into the Tullyglass Hotel Disco for free.

Ah, for a misspent youth!

What's that? You want to know?

sudo apt-get install postgresql-8.4-postgis

Adding Shell Access to Our Flask-Script Example

2011-06-07T15:09:00.000-07:00

The starting example of how you can use flask-script was given in my earlier article;
A Simple Flask-Script Example.

Next thing we are going to add is the ability to run a shell which that shares your applications environment. You need to decide what you ant to import and put it in the shell_context function. Call it whatever you want, but it has to return a dict with your context items inside. To keep it simple I just import the app itself. The lines you need to change are:

from flaskext.script import Manager, Server, Shell

and

def shell_context():
    return dict(app=app)            
            
if __name__ == "__main__":
    manager.add_command('dev', DevServer())
    manager.add_command('test', Test())
    manager.add_command('zen', ZenTest())
    manager.add_command('shell', Shell(make_context=shell_context))
    manager.run()

now test it:

python manager.py shell

>>> app.debug
True
>>> app.logger
<flask.logging.DebugLogger instance at 0x2ee3518>

and so on.

There's one final installment to follow - Let Your Imagination Run Riot On Our Flask-Script Example

The full listing for what we did here is below:

from flaskext.script import Manager, Server, Shell
from flaskext.zen import Test, ZenTest
import application


app = application.create_app()

manager = Manager(app)

class DevServer(Server):
    
    def handle(self, app, host, port, use_debugger, use_reloader):
        try:
            from flaskext.lesscss import lesscss
            lesscss(app)
        except: pass

        app.run(host=host,
            port=port,
            debug=use_debugger,
            use_debugger=use_debugger,
            use_reloader=use_reloader,
            **self.server_options)
    
def shell_context():
    return dict(app=app)            
            
if __name__ == "__main__":
    manager.add_command('dev', DevServer())
    manager.add_command('test', Test())
    manager.add_command('zen', ZenTest())
    manager.add_command('shell', Shell(make_context=shell_context))
    manager.run()

Installing Node.js on Ubuntu 11.04 and Mint 11

2011-06-07T09:25:00.000-07:00

Installing Node.js is simple and let's you get play with all the Javascript goodness that it offers.

Another reason to install it is to use Flask-CSS (and hence lesscss) which I mentioned in my last article, so I thought I'd better cover that now.

Let's install it in our home directory:

cd ~
rm -fr node # in case this is a fresh install

sudo apt-get install -y g++ curl libssl-dev apache2-utils git-core
wget http://nodejs.org/dist/node-v0.4.9.tar.gz
tar xfz http://nodejs.org/dist/node-v0.4.9.tar.gz
cd node-v0.4.9 && ./configure 
# or the bleeding edge version
# git clone git://github.com/joyent/node.git 
# cd node && ./configure 
make 
sudo make install

and finally you need to add the following to your ~/.bashrc file:

export PATH="$HOME/node/bin:$PATH"

Finally install NPM (Node Package Manager):

source ~/.bashrc
curl http://npmjs.org/install.sh | sudo sh

Simples!

Thanks to:

http://thinkdevcode.wordpress.com/2011/05/14/installing-node-js-on-ubuntu-11-04-natty-narwhal

A Simple Flask-Script Example

2011-06-07T06:01:00.000-07:00

Here is a simple flask-script example that you can use to start your application in development mode, or test modes.

I know that you can do that without flask-script, but the point of this script is that you can do so much more from this starting point. Notice for example that I'm trying to load flask-lesscss and execute it (which won't work, but won't fail if you don't have lesscss installed.)

To get the application (in the 'application' module) running you just need to do one of the following:

python manager.py dev
python manager.py test
python manager.py zen

I show how to add shell access in the second article in this series. Meanwhile why not let us see what you have in your manager script?

from flaskext.script import Manager, Server
from flaskext.zen import Test, ZenTest
import application


app = application.create_app()

manager = Manager(app)

class DevServer(Server):
    
    def handle(self, app, host, port, use_debugger, use_reloader):
        try:
            from flaskext.lesscss import lesscss
            lesscss(app)
        except: pass

        app.run(host=host,
            port=port,
            debug=use_debugger,
            use_debugger=use_debugger,
            use_reloader=use_reloader,
            **self.server_options)
            
            
if __name__ == "__main__":
    manager.add_command("dev", DevServer())
    manager.add_command('test', Test())
    manager.add_command('zen', ZenTest())
    manager.run()

Install Git, Pip and VirtualEnv and VirtualEnvWrapper on Ubuntu in a Minute

2011-06-06T22:46:00.000-07:00

From a terminal, type:

sudo apt-get install git python-setuptools
sudo easy_install pip
sudo pip install virtualenv virtualenvwrapper

Now, you may want to set some defaults in your ~/.bashrc. My relevant entries look like this:

export WORKON_HOME=$HOME/Projects
export VIRTUALENVWRAPPER_HOOK_DIR=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_LOG_DIR=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--no-site-packages --distribute'
source /usr/local/bin/virtualenvwrapper.sh

git config --global user.name "Terse Col"
git config --global user.email myemail@gmail.com
git config --global http.sslVerify false

Now away you go:

mkdir $HOME/.virtualenvs
source $HOME/.bashrc
mkvirtualenv mynewproject 
cdvirtualenv

Use Twitter and Facebook with Ubuntu 11.04 (Natty) or Mint 11

2011-06-06T15:58:00.000-07:00

Ubuntu's preferred Internet Messenger is not Pidgin which with a little love can happily act as your Twitter and Facebook messenger/client.

You want to install the pidgin-microblog package which I think used to be called purple-pidgin:

sudo apt-get install pidgin-microblog

Strictly, Facebook doesn't need this, but it makes the experience nicer.

Now you want to start up Pidgin and do "Accounts/Manage Accounts" (or CTRL+a).

Add Twitter

"Add" (or ALT+a) then the protocol should be "TwitterIM"

and then "add and that's it"

Add Facebook

Facebook is almost the same, EXCEPT that you may or not actually know your Facebook username if you, like me have always used your email address and password.

You can find it at Facebook, in the top right hand corner there's drop down called "Account" then "Account Settings/Settings/Username" which will show you what it is.

Then back in Pidgin do "Accounts/Manage Accounts" (or CTRL+a) "Add" (or ALT+a) then the protocol should be "Facebook (XMPP)" and enter your details:

Note the Resource = Pidgin entry.

Have fun and please comment on any problems you come across.

A Review of Free Code Repositories

2011-06-01T03:56:00.000-07:00

In this article, I’m going to assume you are part of a *One Sized Team*, because there are loads of alternatives and I’m trying to keep the word count at a reasonable level. If you don’t fit that outline then you can still read on because most of the available offerings also have options that might well be right for you, but at a small monthly cost. I cover these in my article on *Code Repositories for Fun Sized Teams*.

Where to store your code?

So you've got some code and you need somewhere to put it so that it won't get lost when someone steals your laptop.

Well there are options:

Copy it to a USB drive
Upload the folder to Google Docs
Email it to yourself
Paste it into Pastebin

Well I know that I have done of all of these things at times and that’s just fine for snippets of code. With bigger bits of code you’ll need something a bit cleverer. Something that let’s you keep track of changes. A Code Repository.

Assembla
Assembla offers one free private Git or Subversion repository (which they call a space) although you have to find it because they'd rather you took a paid for option. You can find the free offerings at https://www.assembla.com/free. You are allowed as many public spaces as you want. The Git implementation uses Git over ssh.

Also be aware that they automatically throw in the free trial additions which is fair enough, but you can drop these extras immediately if want and just get used to the free set of tools which include you're storage space and a ticketing system.

They also provide nice clear instructions on how to get going with setting up your repository, although putting your stuff under 'start' in their menu system confused me because I expected it to be under 'Home' or 'Your spaces' or something similar.

Berlios
Berlios offer Git and Subversion hosting (and other tools) for proper grown up Open Source projects. Too much hassle for me, which is probably their plan. Looks much like SourceForge. Have a read at the Wikipedia entry.

Bettercodes
Bettercodes is rather cool for a number of reasons. Firstly, they allow you multiple projects each with it's own Subversion or Git (Smart HTTP) repository. You can either make your code public, private of hidden and also invite others to work on your projects. You also get milestones and tasks and flat file storage. One word of caution; this site is in beta and although the repository stuff worked fine, there are definitely some teething problems. There's no word yet of a pricing plan, so get on there and give them some feedback.

Bitbucket
Bitbucket offers multiple, free, private project repositories with wikis and issue tracking. It's easy to use and reassuringly swish.

My only issue is that it only offers Mercurial repositories and I much prefer Git. That being said, let me just repeat. Bitbcket offers multiple, free, private project repositories with wikis and issue tracking."

Codeplex
Codeplex is Microsoft's version of proper grown up Open Source projects. Free for Open Source stuff and uses Mercurial or Subversion. Again, like Berlios,too much hassle for me.

Fedora Hosted
"Fedora Hosted is a project sponsored by the Fedora Project to allow upstream developers to host their code and collaborate online." See Berlios, Codeplex and grown ups. Not for me at this stage.

GitHub
GitHub is the most famous Git repository out there. Free plans are all public so other people will be able to see your code, but the interface, instructions etc are lovely and there's a nice repository explorer. Only Git (over ssh) is available.

Gitorious
Again, like GitHub, Gitorious is an attractive and easy to use site. Repositories are required to be under an open source licence (not just public) which may or may not suit your purposes. Only Git (over ssh) is available.

GNU Savannah
Savannah is another site for grown ups. This time focusing on free software projects (hence GNU). The rules here are particularly strictly enforced and you aren't even allowed to utilize non-free formats in your code. They provide repositories in CVS, GNU arch, Subversion, Git, Mercurial, Bazaar which is quite astonishing and there are loads of extra tools such as mailing lists and bug tracking etc. Looks like SourceForge. Don't get me wrong, this is a mighty site, but it doesn't suit everybody's requirements.

Is your's an Open Source Project?

Google Code
The sheer number of Google tools that integrate into this site must make it tempting for anyone. It's really useful, but as obvious as the advantages are, I'm going to mention a couple negative points here, but please take them in the context of an excellent site.

Firstly it only offers Mercurial and Subversion code repositories. These are perfectly good, but I still prefer Git.

Secondly projects have to be Open Source. This is not so different from lots of other sites mentioned here, but since Google offers almost all it's tools freely, including Gigs of space for email, sites, docs etc, I don't really understand why they don't offer a few megabytes for private code repositories. Any Google guys out there?

JavaForge
JavaForge looks excellent and the number of tools is impressive (" Web hosting, Document Management, Wiki, Forum, Online chat, Issue tracking integrated with optional Git, Mercurial or Subversion revision control"), but in short I don't do Java as a hobby and this is definitely a Java site. The Open Source rule applies. Java for grown ups.

LaunchPad
I haven't used LaunchPad, but it looks really rather useful. Firstly it's only for Open Source projects (for which it is free), and secondly it uses Bazaar as it's version control system. If you can work with those then I think you should have a good hard look at this offering as it offers stuff like Bug Tracking, Mailing Lists, Teams, FAQs, Code Review and more.

ProjectKenai
ProjectKenai is now owned by Oracle and is being closed/shifted/shut down.

SourceForge
The daddy of them all, SourceForge has some great statistics. I looked just now and it claims to be hosting 294,229 projects and had 3.8 million downloads today. If you don't mind people downloading your code for free or with an Open Source license, you can host it here. It offers all the version control systems already mentioned (and good old CVS) and gives you room for WIKI, files and other niceties.

Surely there's some kudos in just having a SourceForge Project. You could print your own t-shirts!

Anyway, don't forget about SourceForge, there's a reason the other grown up sites tend to emulate it.

Finally

Have I missed anything? Please ley me know.

There's some other information on wikipedia:

http://en.wikipedia.org/wiki/Comparison_of_free_software_hosting_facilities

What a Smart Git

2011-05-31T05:29:00.000-07:00

This article is a primer on Git protocols so that you get to grips with what specific Git repository providers are offering. I'll cover Code Repositories in a follow up article.

Traditionally Git lets you communicate with the Git Server (Repository) via four protocols:

Local (file system)
Secure Shell (SSH)
Git
Dumb HTTP

Local is probably little used because after all what's the point of having a repository for your code on your hard disk if you're hard disk crashes, or your laptop gets stolen.

SSH and Git protocols worked just fine because they could talk to the server and get things done efficiently.

Dumb HTTP was a bit of a bodge because it could not be used to talk to the server and things were very inefficient (see the references below).

So, What's The Problem?

Firewalls, especially in the workplace, just hate it when people try to use anything except the standard ports. You know, HTTP, HTTPS, SMTP and after that they get sulky. Maybe, you'll be allowed to use FTP, but certainly not port 9418 (Git) or 22 (SSH).

Smart HTTP

The Smart HTTP protocol is really still just HTTP, but it's the server side that got smarter. Since Git version 1.6.6, Git servers can communicate with the client in a much smarter way using HTTP which makes it just as efficient as SSH or Git protocols AND allowing us to use Git anywhere we can browse the Internet; so long as the repository provider has things set up to use Smart HTTP.

If they aren't now, they soon will be.

References: