Twitter Streaming API - Almost Useful

Twitter's Streaming API is a splendid idea. It gives developer's access to a good splodge of Tweets and let's us filter them in different ways. 

However it has a few flaws.

For Twitter, the benefit of a Streaming API is probably one of scalability. Instead of us using the old REST API to ask for specific data and causing tens of thousands of data look ups, all they do is give us the end of their own data stream once it has been used in house and is now halfway across the back garden. All they have to do is allow us to filter the stream a bit to make it a bit more relevant to our needs and put an absolute cap on the throughput (about 1% for most of us.)

This looks good. 1% is enough for most development needs and streams down your connection like a low bandwidth radio station. I don't really know what the bandwidth or download is, but it's not much. Once we've developed our new and wonderful website, then we can ask, or possibly pay, Twitter to turn up the pressure a bit.

So, now let's look at the filters.

There are several ways that the stream can be filtered

• follow - filter by userid
• track - filter by keyword
• location - filter by geographic location
• retweets - just the retweets ma'am
• links - only tweets containing a link
• random - I think they just mean unfiltered

On the face of it this is pretty useful. You could filter the stream by adding the location of your home town so that you can get the jist of what is important to your townsfolk. You could filter the stream by the keyword 'Elvis' to see who has spotted Elvis lately and plot the results on a map.

My own first idea was inspired by the M5 motorway accident just a few miles from where I live and astounded that even in this day and age, the scale of the incident was only uncovered somewhat slowly. Surely what the quantity and content of the tweets from the people who were NOT in the incident itself, would help scale the incident? So what I wanted to do was:

1. Listen to what people Tweet at known traffic jam locations.
2. Identify some fingerprint of common words, maybe "traffic, jam, standstill, miles" or whatever.
3. Look for clusters of these words near to motorways.
4. Plot the clusters based on the location of the phones that made the tweets.

So I get an updating list of current traffic incidents from the Traffic England RSS feed and start listening within 5 kms of the stated location using a bounding box centred on the location. The first thing I notice is that some of the tweets are in the bounding box, some are around it and some, quite a few are far away, sometimes 200km away or more!

I don't know how Twitter do the filtering, but evidently it's based on something fairly broad brushed. I can live with that maybe, all I have to do is check the Tweets geo location, which is added if you tweet by most modern phones. I was expecting most of the useful tweets to be from a mobile anyway, so that would work if I can just get used to maybe 2% of the tweets actually being in the bounding box. 2% 0f 1% is after all only 0.02% of all Tweets or 1 Tweet in 5000.

So what happens if I assume that the word "traffic" will occur in the most useful Tweets? This is either bad science or common sense data filtering depending on how you look at it.


Alas, it appears that the Streaming API does not allow you to filter by location AND keyword! All you can do is do an OR filter, so I can filter the stream to include certain areas OR certain keywords, but not certain keywords within a certain location.

To me this just renders the API all but useless, but no doubt you lot are much smarter than I and will dazzle me with your great ideas.

Please let me know.

Google App Engine Python2.5 Development in Ubuntu 11.10

How to set up a development environment for Google App Engine, Python2.5 in Ubuntu 11.10.
Firstly, sorry if the code below is badly formatted, but there's a clearer copy of the code at the bottom.

Python2.5 isn't in the Ubuntu 11.10 sources by default so in a console:

sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update

and install python2.5 and some other stuff:

sudo apt-get install python2.5 python-virtualenv virtualenvwrapper python-pip

Next for project 'oinkyoinker':

export CURRENT=oinkyoinker
mkvirtualenv --no-site-packages --distribute --python python2.5 ${CURRENT}
workon ${CURRENT}
cdvirtualenv
pip install fabric yolk ipython readline

Now download GAE and fix the path:

wget -O /tmp/gae.zip http://googleappengine.googlecode.com/files/google_appengine_1.5.5.zip
unzip /tmp/gae.zip
echo "../../../google_appengine" > lib/python2.5/site-packages/gae.pth

and create a simple file server app:

mkdir -p application/static

echo """application: oinkyoinker
version: 1
runtime: python
api_version: 1
default_expiration: "7d"
handlers:
- url: /
  static_dir: static
""" > application/app.yaml

and run it:

./google_appengine/dev_appserver.py application/

Hope this helped you too.

MapReduce in Python

Over the last while I've been using NodeJS, MongoDB and Google's App Engine. Everything seems to have MapReduce functionality and frankly I was missing it when I went back to do some work in plain old Python. This is a nice short example of how it can be done.

The basic format that you need is something like this:

Now, a simple working example using the usual MapReduce example:

The output will look something like this:

• I got most of this from the following blog: http://goo.gl/nW8iA .
• Docs are here: http://docs.python.org/library/multiprocessing.html

MapReduce with Mongoose and CoffeeScript

After searching the InterWeb for a decent MapReduce example coded in CoffeeScript I came up blank and decided to write my own. This one uses Mongoose too - well why use anything else?

I haven't written a whole lot of explanation in the text, but commented the code quite heavily so you can copy and paste and still have it with you.


Load Mongoose and connect to your database.Describe your schema and model and then load up some dummy data.

Now for the MapReduce stuff. If you still don't get MapReduce then you will once you read this Star Trek based article.

Define a Map function.

Define a Reduce function.

Get some output

The results will now be in a collection called "results" which you can look at with JMongoBrowser or whatever.

I hope this is of some use to you. If there are any corrections or improvements then please post something below.

Staying Up Late - CoffeeScript and Zappa

Over the last couple of years NodeJS has become the darling of "coding for pleasure" crowd. All that cross fertilization from the client side buffs into the server side and the server side coders ranting about the whole event-driveness of NodeJS.

Yes, it's the next big thing. Well, it's probably not, but it's certainly "A" next big thing or maybe we'll all get bored dream up something else with a catchy name... "perl for parallelism" or "Friendly Java (without the memory leaks)", or what about "C+=3" or "Oil for Android", "Lube for Palm OS". Well you know what I mean.

We see something; we explore it and once it starts getting a bit boring we convince ourselves and then our customers that we need to use some new technology or other. It's all true of course, or partly true, or lies.

Anyway, I'm not here to rant about that, after all I as guilty as anyone else.

A Messy Affair

What I want to rant about is Zappa which is my own personal cure for technical disfunction at the moment. Well that sounds real bad, but I've been doing this job for years and it's only these little affairs over the years that keep me coming home; "Assembler", "C", "C++", "Ruby", "Python", how I remember their sweet names, but sooner or later they all start to look like their mother and have to go.

Zappa is a young virgin of a technology, only now reaching a pubescent version 0.3. Essentially it's a lightweight framework built on top of known technologies such like NodeJS, CoffeeScript, Express and Socket.IO in much the same way as Python's Flask is built on top of Werkzeug, Jinja 2 and so on.

It's that Socket.IO bit that makes this one so sexy (I'm overdoing the procreation metaphors aren't I?). Socket.IO provides cross-browser WebSocket goodness which means you can do stuff like stop using the Model-View-Controller paradigm and start thinking outside the cubical with your "Model-View-Controller-View-Controller-Controller-View Paradigm" or whatever. Mind you, that's when you'll realise that MVC wasn't about limiting your options, it was about simplifying them.

Anyway, back to Zappa itself. Go and have a look, there are great examples that will get your inventive juices flowing (stop it!) and inspire you to spend a little time alone locked it the bathroom... I mean office. The community is starting to grow and the mailing list is big enough to be useful, but not so big as to be faceless and scarey.

Have fun but take precautions.

Back of an Envelope Node.js Framework - Discuss

I had a quiet afternoon today and while "keeping up with recent developments in my field" or as you might call it "wasting my afternoon googling" I got to thinking what I'd like a Node.js MVC framework should or could contain.

How design shouldn't be done

Firstly, I've been playing with Node.js for about three weeks now and I'm impressed, but I can't seem to find a framework that I'm happy with. Backbone looked good, but I wasted three days on it and it was too much like hard work. Similarly Spine which tried to be less complicated, but eventually failed. Finally I really wanted Brunch to work but it was a release behind backbone (which is a major component of it) and I didn't want to get lost in the mismatching documentation (the Backbone release was pretty major).

I have spent a whole lot of time over the last couple of years using python frameworks, especially Flask and I like their approach of starting with a good foundation (in their case Werkzeug) and adding the bits that are still needed. Could that work with Node? I mean writing a basic HTTP server looks pretty straight forward and there are tons of modules to add the other functionality.

So with this possibly naive starting position I grabbed my pen and scribble down some stuff I thought I'd need. Now bear in mind that I haven't much used any of these modules yet, and I pinched several of them from the bits that go to make up Brunch.


I'd appreciate any feedback, suggestions and so on that those more learned than I might care to offer. Few of you will be less learned I imagine.


Here are the transcribed notes with some links:

Server

• Simple Node.js
• Express

Routing

• choose a module
• Express?

Data Store

• Mongo?
• CouchDB?
• Barricane-db

Models

• JS Object/JSON?
• ORM?

Views
    -CSS

• lesscss ?
• Stylus ?

    -Templating

• mustache(js) ?
• Eco (coffeescript)

Controllers

• Plain Old JS
• CoffeeScript?
• jQuery
• Underscore.js
• Quip (responses) ?

Middleware

• how? Express

Other Stuff

• node-boilerplate?
• Stitch (glue)
• Cake

If you spot any typos, inaccuracies or just plain lies then please let me know.

Binding Updates to Model Changes in Backbone/CoffeeScript

My previous post showed a View that looked like this:



But what happens if we have a more complicated app and something else changes the Model?

Well as I said I'm just learning myself here, co the correct way that should be handled is to bind Model changes to the Views render function which looks like this:



Note the bind instruction which means that every time the counter changes, it calls the render function. You might also notice the fat arrow now on the render which make "this" still mean the View rather than the caller (which would be the Model otherwise.)

The explanation for all this is at http://jashkenas.github.com/coffee-script under "Function Binding".

Simplest Ever Backbone.js/CoffeeScript Example

I'm teaching myself to use Backbone because after having done quite a lot of JavaScript in work lately, it has become obvious to me that I need a framework.

I'm also learning how to use CoffeeScript, which is close enough to python/ruby for me to understand.

First thing that I noticed is that most of the examples for Backbone out there are in pure JavaScript, so I thought I'd do my bit by providing something in CoffeeScript.

To that end, here is a very simple app which displays a number. You can increase or decrease the count by clicking buttons.

The app.coffee code:



and the html to go with that:



In detail

At the top, we set up global hashes to store our objects. In a bigger app this would make referencing bits and pieces much easier, although we don't really need it in this example.



Next the controller. This one is almost empty except it has two routes both taking you to the home control which is also part of the controller. The route with no reference (the empty string) and the one for #home tagged on the end of your URL (by way of an example) both call the home function in this case.

The home control just tells the view to render itself.



Next the Counter model which has one attribute called count which defaults to zero. In reality I think you would want to synchronize your models with a database, but we'll do that in another post soon.



And now for the View. The View is obviously what you are seeing in the browser. In our trivial example we only have one view, but a real app would probably have several. A view lumps together a unit of the app, sometimes nowadays we call these widgets, or blocks or display regions or whatever. You get the idea.

In the initialize function, we just connect the view to the model that backs it. The events connect the button to the functions that they call. The render function draws the result into the #count div; the @ symbol at the end is shorthand for this. In CoffeeScript (like Ruby) the last line of a function gets returned even if you don't specifically say "return". So in other words it means "return this". This is handy since it lets us chain together functions.

The inc and dec functions increase and decrease the count in the model and then update the display. Please see the update here.



Finally the equivalent of the main() function which instaniates the bits and bobs and then starts the history. I've no idea why they called it history, but basically it starts up all the listeners.



That's all for now. Next time I'll show you how to connect all this to your database using python and Flask.

If you see anything wrong with what I've written here or if you spot spelling or other mistakes then please let me know.

Set Up Your Own Git Repository Hosting Service in Three Minutes

You might have seen my earlier post "A Review of Free Code Repositories".

Well I forgot about hosting your own. Really handy for hobbyist development. I have one of those free developer servers from Amazon out there somewhere. I don't use it much except to demo the occasional site and so on.

I decided that it was a good idea to set it up as a git repository and i show you here how to do this. If you are using something Ubuntu-ish remotely then this should take about three minutes.

A Word of Explanation
This is the simple setup, it avoids creating 'git' user on the remote server, but you DO need to use a seperate key for using the git service and I show you that first.

On your local machine:

# echo "Host gitserver
  Hostname ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com
  User git-col
  PreferredAuthentications publickey
  IdentityFile ~/keys/git-col.pub
" >> ~/.ssh/config

# mkdir ~/keys

# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/col/.ssh/id_rsa): git-col.pub # do NOT accept the default

# cp git-col* ~/keys
# scp git-col.pub ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:
# ssh 
ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com

On your remote host you are now logged in as 'ubuntu':

# sudo apt-get install git
# git clone git://github.com/sitaramc/gitolite
# ./gitolite/src/gl-system-install 
# gl-setup -q ~/git-col.pub 
# exit

On your local machine:

# git config --global user.name "Your Name"
# git config --global user.email your@email.address
# git clone gitserver:gitolite-admin
# exit

That's it really. To create a new repository called dummy and initialize it, you can do something like this.

On your local machine:

# set the new repository name
export repo=dummy

# update the gitolite conf file with your new repository
cd ~/gitolite-admin
echo "        repo    ${repo}" >> conf/gitolite.conf
echo "                RW+     =   @all" >> conf/gitolite.conf

# push the config to your server
git commit -m "added new repository ${repo}" conf/gitolite.conf
git push 

# make sure you cd to an existing directory!
mkdir ~/Projects
cd ~/Projects

# pull back the new empty repository
git clone gitserver:${repo}

# go to repository and make any change
cd ${repo}
echo "*.pyc" >> .git/info/exclude # I use python
touch README

# do your first push back to master
git add .
git commit -am "repository ${repo} pushed to origin on gitserver"
git push origin master

You'll be using that last code section again, so you might want to make it into a script.

Never type anything you don't understand. 

If I've made any mistakes then please let me know.

Flask-Coffee - Fill your flask with coffee

Flask-Coffee is a Flask extension that compiles .coffee CoffeeScript files into .js JavaScript files for you if they have changed before your app renders.

This is a nice idea in development and not so nice an idea in production, so please keep that in mind.

Prerequisites

You will need to have to have Node.js installed with NPM and coffee-script.

You can follow the instructions in my previous article to install Node.js and then it's just

npm -g install coffee-script

Installation

Install Flask-Coffee with pip:

pip install flask-coffee

Usage

from flaskext.coffee import coffee

coffee(app) 

This will watch your app’s static media directory and automatically render .coffee files into .js files in the same (sub)directory.

The best way to incorporate Flask-Coffee into your development is as described for flask-lesscss in my earlier flask-script tutorial.

Contribute

If you want to contribute email me at the email at http://pypi.python.org/pypi/Flask-Coffee after checking out the code at http://bettercodes.org/projects/flask-coffee.

CoffeeScript Development in 3 mintutes

Any obvious reference to coffee needs a picture

The idea of CoffeeScript just sounds great me. I like the idea of compiling decent JavaScript from code you can actually read. But what do you do if you have no Node.js available to compile it?

Well the answer is to compile to JS at runtime. However, I had an awful time getting the right setup to get going. If you are interested in playing with CoffeeScript where Node.js is not available (I'm using it in work to show what it's capable of) then just follow these instructions.

Create a directory to work in and inside that create an index.html file thus:

<!DOCTYPE html>
<html>
<head>
    <title>CoffeeScript Demo</title>
    <script data-main="scripts/requirements" 
               src="scripts/require.js"></script>
</head>
<body>
    <h1>CoffeeScript Demo</h1>
</body>
</html>

and a directory called 'scripts' into which you put the following.

• JQuery
• Require
• CoffeeScript Plugin For Require
• A file called requirements.js containing:

require([
    'scripts/jquery-1.6.1.min.js',
    'cs', 
    'cs!app',
]);

• And a file called app.coffee containing:

square = (x) -> x * x
alert square 42

That's it!

Now load the index.html page into your browser and the CoffeeScript code app.coffee should execute.

Of course you will now have to go away and read the docs so that you know why that worked, but then so will I.

Flask Extensions For Authorization with Examples

In my opinion, there is one serious omission to Flask at the moment and that is that there is no authoritative method for authorizing users. 

In a way this goes with the general ethos of micro frameworks - it isn't being forced down your throat unless you want it.

However in this case I think that at least some structured 'recipes' are required to provide peer guidance, best practice or whatever you want to call it. Too often the advice is "roll your own", but I'm not sure I'd be comfortable recommending that to the inexperienced.

So where are we?

Flask-Principal and Flask-Login

At the moment there is only one approved auth extension called Flask-Principal which takes care of authorizing different roles of user. It does not dictate what those roles should be and it does not force you to use any particular method to authenticate those users.

A second extension called Flask-Login is currently awaiting approval, which I suspect it will soon get. This extension also takes care of authorizing users, but only understands one type of user role, so in other words you are either a user or not. Like Flask-Principal it does not force you to use any particular method to authenticate those users.

Example Code

The code below is sort of a basic outline that we will fill in using proper code in a moment.

The User model is very silly. If you create a User by passing an id of '1' you get a User with a username of 'user1' and a password of 'user1_secret'. To help with the examples we then create users one to twenty (user1 to user20). Obviously you would normally have them stored in a database or something, but we're keeping it simple.

At this stage there is nothing protecting access to the 'home' resource and 'login' and 'logout' don't work yet either.

from flask import Flask, Response, redirect, \
    url_for, request, session, abort

app = Flask(__name__)

# config
app.config.update(
    DEBUG = True,
    SECRET_KEY = 'secret_xxx'
)

# middlewares etc go here
      
# silly user model
class User(object):

    def __init__(self, id):
        self.id = id
        self.name = "user" + str(id)
        self.password = self.name + "_secret"        
    def __repr__(self):
        return "%d/%s/%s" % (self.id, self.name, self.password)


# create some users with ids 1 to 20       
users = [User(id) for id in range(1, 21)]


# we'd like to protect this resource
@app.route('/')
def home():
    return Response("Hello World!")


# somewhere to login    
@app.route('/login', methods=['GET', 'POST'])
def login():
    pass # this does not work yet

# somewhere to logout
@app.route("/logout")
def logout():
    pass # this does not work yet
    
        
# handle login failed
@app.errorhandler(401)
def page_not_found(e):
    return Response('Login failed')


@app.errorhandler(403)
def page_not_found(e):
    return Response('Unauthorized')


if __name__ == "__main__":
    app.run()

Download the code!

Flask-Principal Example
Now that we know how that works, or doesn't, let's look first at how flask principal might set about protecting the home resource.

As I said earlier, Flask-Principal doesn't tell you what roles you need for your app. In this example I'm only using one called 'normal' which we need to tell the extension about before we tell Flask to initialize the extension:

# flask-principal
principals = Principal()
normal_role = RoleNeed('normal')
normal_permission = Permission(normal_role)
principals._init_app(app)

After that we can protect the 'home' resource thus:

@normal_permission.require(http_exception=403)

If your request's session doesn't have all the stuff it's asked for (ie. has not logged in) then Flask-Principal gives it a HTTP 403 (Forbidden) response which redirects it to:

@app.errorhandler(403)
def page_not_found(e):
    session['redirected_from'] = request.url
    return redirect(url_for('login'))

Which, as you can see stores the URL that you wanted in the first place ('home' in this case) and then redirects you to the login page. Since by default this request is a GET, you are shown the login form and not the login process which is used for POSTs.

Enter a valid username/password combination, e.g. 'user7' and 'user7_secret' and you get logged in and sent back to the 'home' page. The login sends a signal to the extension to tell it who has logged in and congratulations you get to see the 'Hello World' message!

If you enter an invalid username/password combination you get served an HTTP 401 (Unauthorized) and thus you end up in the 401 errorhandler seeing 'Login Failed'

If you go to 'logout' you the code will destroy any of the relevent session information and tell you you are 'Logged Out':

@app.route("/logout")
def logout():
    for key in ['identity.name', 'identity.auth_type', 'redirected_from']:
        try:
            del session[key]
        except:
            pass
    return Response('Logged out
')

Try out the code!

Flask-Login Example
With Flask-Login you need to tell Flask to use the extension and this time tell the extension where the login page is (in this case "login"):

# flask-login
login_manager = LoginManager()
login_manager.setup_app(app)
login_manager.login_view = "login"

Secondly you have to use a mixin to create your User class. The mixin just makes some standard functions available for the User.

class User(UserMixin):

Thirdly, you need to set up a user_loader that will load your user based on the user id:

# callback to relaad the user object        
@login_manager.user_loader
def load_user(userid):
    return User(userid)

This time when you go to 'home' and you haven't yet logged in, you don't get a HTTP 403 (Forbidden), but instead get sent to the resource you defined earlier; 'login'.

Login successfully and this time the login retrieves the actual User object and tells the extension using the login_user function. Again, congratulations!

Try out the code!

You can work out for yourself how to store the User object in a database or datastore and how to check the passwords. There are a good ways to do both, but this is not an article about that. Should we have one in a future article?

My Top Ten Python Web Frameworks

An updated version of this article can be found here.

I have recently revisited all of the Python Web Frameworks that I know of and re-evaluated them for use in a new project.

I did this first two or three years years ago and so I imagine I will be biased by the decisions that I made then, so I think it's only fair to say that I started using Django because of it's fantastic documentation and I suppose the helper scripts which help the beginner along. However, all that Convention over configuration stuff can be too much for me sometimes.

I then moved onto Tipfy because it slotted in so easily to Google App Engine which I was using a lot at the time  and the Django framework itself was starting to be more of a hindrance than a help.

One day however I stumbled upon Bottle which was a revelation. Being a micro framework it's explicit. That is if you want to use something, you make up your mind to do so; it's not some cleverness built into the framework over which you have no obvious control. If you want to know how it works, then read the code!

Finally I came to Flask, another micro framework, because I like the way it is based on Werkzeug and other well trusted libraries which gives me a some peace of mind that it is supportable, and secondly because it is also explicit like Bottle.

All that being said, here's my top ten:

1. Flask - small, fast, easy to learn and built on reassuringly supported libraries.
2. Bottle - small, fast and a great tool for learning.
3. Tipfy - built for GAE, easy to pick up and well supported.
4. Django - contains most of what you will ever need to build a website, well documented, but starting to bulge at the wasteline.
5. Pyramid - the successor to Pylons, this one has great docs and great ideas. It is a framework, but you can unpick the bits you don't like.
6. CubicWeb - like Django, Convention over configuration, but nice in some ways. 
7. GAE framework - Simple, built for GAE, and well documented.
8. Web2py - I hate it, but love it. All that framework kinda helps, kinda hinders. Give it a little time and you can definitely pull a website together, it depends on your personality whether or not you go mad first though.
9. Nagare - I admit I haven't built a full system with this one, but I have a feeling I will be soon. I'd rather it was a wee bit more mature though, but the libraries/concepts it's built on is all good stuff.
10. Weblayer - Another micro framework. Some good ideas here, but could do with a little love, If you are a developer looking for a worthwhile project, you should consider lending a hand.

[Please note I am currently updating this article here]

For more information, please read:

• http://wiki.python.org/moin/WebFrameworks
• http://en.wikipedia.org/wiki/Comparison_of_Web_application_frameworks

Let Your Imagination Run Riot On Our Flask-Script Example

In my last article in this series, "Adding Shell Access to Our Flask-Script Example" I showed you how you could indeed get shell access from our script.

You see the whole point of Flask-Script is that it gives you scripted access not to the application, but to the environment in which the app runs. You don't actually need to be running the app, or importantly, to interfere with the app to do some task or other.

An Example
For example, lets say you want to load some data into the database model.

There are three ways you could do this:

1. Write a script that understands your app's data model and get it to insert the data. You may be able to import the model module straight from your app, but you may not, due to dependencies etc.
2. Run the code straight from your app, possibly using an admin screen somewhere. This means designing and implementing the screens - including the obvious security requirements.
3. Use Flask-Script like this:

from application.model import Articles
import csv


@manager.command
def ingest_csv():
    '''read data from a csv file and ingest it into the database'''
    reader = csv.reader(open('articles.csv'))
    for row in reader:
        a = Article()
        a.init_from_csv_row(row)

and then;

python manager.py ingest_csv

Simpler I think you'll agree?

Scheduling Task
There is one other huge advantage to using Flask-Script; you can run them from crontabs.

*/10 * * * * /some/path/bin/python /some/path/manager.py ingest_csv >>/tmp/ingest.log 2>&1

Other Ideas
Here are some other examples of things you might want to dovia script:

• Purge old data from your database.
• Backup data from your database.
• Export data from your database.
• Do anything else you want from your database!
• Flush your caches.
• Mail out reports.
• Other stuff I haven't thought of. Let me know!

Okay, enough of all this, go and enjoy!

Install Postgres/PostGIS on Ubuntu 11.04 (Natty) or Mint 11

One of my most visited entries on my old blog was "Install Postgres/PostGIS on Ubuntu Maverick/10.10". So I was keen to update what had obviously been quite a useful article.

After all it was only a couple of dozen lines of code.

Turns out that it's one of those things I know that will never be used again; like how to hotwire a 1972 Ford Cortina, or how to get into the Tullyglass Hotel Disco for free.

Ah, for a misspent youth!

What's that? You want to know?

sudo apt-get install postgresql-8.4-postgis

Adding Shell Access to Our Flask-Script Example

The starting example of how you can use flask-script was given in my earlier article;
A Simple Flask-Script Example.

Next thing we are going to add is the ability to run a shell which that shares your applications environment. You need to decide what you ant to import and put it in the shell_context function. Call it whatever you want, but it has to return a dict with your context items inside. To keep it simple I just import the app itself. The lines you need to change are:

from flaskext.script import Manager, Server, Shell

and

def shell_context():
    return dict(app=app)            
            
if __name__ == "__main__":
    manager.add_command('dev', DevServer())
    manager.add_command('test', Test())
    manager.add_command('zen', ZenTest())
    manager.add_command('shell', Shell(make_context=shell_context))
    manager.run()

now test it:

python manager.py shell

>>> app.debug
True
>>> app.logger
<flask.logging.DebugLogger instance at 0x2ee3518>

and so on.

There's one final installment to follow - Let Your Imagination Run Riot On Our Flask-Script Example


The full listing for what we did here is below:

from flaskext.script import Manager, Server, Shell
from flaskext.zen import Test, ZenTest
import application


app = application.create_app()

manager = Manager(app)

class DevServer(Server):
    
    def handle(self, app, host, port, use_debugger, use_reloader):
        try:
            from flaskext.lesscss import lesscss
            lesscss(app)
        except: pass

        app.run(host=host,
            port=port,
            debug=use_debugger,
            use_debugger=use_debugger,
            use_reloader=use_reloader,
            **self.server_options)
    
def shell_context():
    return dict(app=app)            
            
if __name__ == "__main__":
    manager.add_command('dev', DevServer())
    manager.add_command('test', Test())
    manager.add_command('zen', ZenTest())
    manager.add_command('shell', Shell(make_context=shell_context))
    manager.run()

Installing Node.js on Ubuntu 11.04 and Mint 11

Installing Node.js is simple and let's you get play with all the Javascript goodness that it offers.

Another reason to install it is to use Flask-CSS (and hence lesscss) which I mentioned in my last article, so I thought I'd better cover that now.

Let's install it in our home directory:

cd ~
rm -fr node # in case this is a fresh install

sudo apt-get install -y g++ curl libssl-dev apache2-utils git-core
wget http://nodejs.org/dist/node-v0.4.9.tar.gz
tar xfz http://nodejs.org/dist/node-v0.4.9.tar.gz
cd node-v0.4.9 && ./configure 
# or the bleeding edge version
# git clone git://github.com/joyent/node.git 
# cd node && ./configure 
make 
sudo make install

and finally you need to add the following to your ~/.bashrc file:

export PATH="$HOME/node/bin:$PATH"

Finally install NPM (Node Package Manager):

source ~/.bashrc
curl http://npmjs.org/install.sh | sudo sh

Simples!

Thanks to:

• http://thinkdevcode.wordpress.com/2011/05/14/installing-node-js-on-ubuntu-11-04-natty-narwhal 

A Simple Flask-Script Example

Here is a simple flask-script example that you can use to start your application in development mode, or test modes.

I know that you can do that without flask-script, but the point of this script is that you can do so much more from this starting point. Notice for example that I'm trying to load flask-lesscss and execute it (which won't work, but won't fail if you don't have lesscss installed.)

To get the application (in the 'application' module) running you just need to do one of the following:

python manager.py dev
python manager.py test
python manager.py zen

I show how to add shell access in the second article in this series. Meanwhile why not let us see what you have in your manager script?

from flaskext.script import Manager, Server
from flaskext.zen import Test, ZenTest
import application


app = application.create_app()

manager = Manager(app)

class DevServer(Server):
    
    def handle(self, app, host, port, use_debugger, use_reloader):
        try:
            from flaskext.lesscss import lesscss
            lesscss(app)
        except: pass

        app.run(host=host,
            port=port,
            debug=use_debugger,
            use_debugger=use_debugger,
            use_reloader=use_reloader,
            **self.server_options)
            
            
if __name__ == "__main__":
    manager.add_command("dev", DevServer())
    manager.add_command('test', Test())
    manager.add_command('zen', ZenTest())
    manager.run()
    

Install Git, Pip and VirtualEnv and VirtualEnvWrapper on Ubuntu in a Minute

From a terminal, type:

sudo apt-get install git python-setuptools
sudo easy_install pip
sudo pip install virtualenv virtualenvwrapper

Now, you may want to set some defaults in your ~/.bashrc. My relevant entries look like this:

export WORKON_HOME=$HOME/Projects
export VIRTUALENVWRAPPER_HOOK_DIR=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_LOG_DIR=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--no-site-packages --distribute'
source /usr/local/bin/virtualenvwrapper.sh

git config --global user.name "Terse Col"
git config --global user.email myemail@gmail.com
git config --global http.sslVerify false

Now away you go:

mkdir $HOME/.virtualenvs
source $HOME/.bashrc
mkvirtualenv mynewproject 
cdvirtualenv

Use Twitter and Facebook with Ubuntu 11.04 (Natty) or Mint 11

Ubuntu's preferred Internet Messenger is not Pidgin which with a little love can happily act as your Twitter and Facebook  messenger/client.

You want to install the pidgin-microblog package which I think used to be called purple-pidgin:

sudo apt-get install pidgin-microblog

Strictly, Facebook doesn't need this, but it makes the experience nicer.

Now you want to start up Pidgin and do "Accounts/Manage Accounts" (or CTRL+a).

Add Twitter

"Add" (or ALT+a) then the protocol should be "TwitterIM"

and then "add and that's it"

Add Facebook

Facebook is almost the same, EXCEPT that you may or not actually know your Facebook username if you, like me have always used your email address and password.

You can find it at Facebook, in the top right hand corner there's drop down called "Account" then "Account Settings/Settings/Username" which will show you what it is.

Then back in Pidgin do "Accounts/Manage Accounts" (or CTRL+a) "Add" (or ALT+a) then the protocol should be "Facebook (XMPP)" and enter your details:

Note the Resource = Pidgin entry.

Have fun and please comment on any problems you come across.

A Review of Free Code Repositories

In this article, I’m going to assume you are part of a *One Sized Team*, because there are loads of alternatives and I’m trying to keep the word count at a reasonable level. If you don’t fit that outline then you can still read on because most of the available offerings also have options that might well be right for you, but at a small monthly cost. I cover these in my article on *Code Repositories for Fun Sized Teams*.

Where to store your code?

So you've got some code and you need somewhere to put it so that it won't get lost when someone steals your laptop.

Well there are options:

• Copy it to a USB drive
• Upload the folder to Google Docs
• Email it to yourself
• Paste it into Pastebin

Well I know that I have done of all of these things at times and that’s just fine for snippets of code. With bigger bits of code you’ll need something a bit cleverer. Something that let’s you keep track of changes. A Code Repository.

Assembla
Assembla offers one free private Git or Subversion repository (which they call a space) although you have to find it because they'd rather you took a paid for option. You can find the free offerings at https://www.assembla.com/free. You are allowed as many public spaces as you want. The Git  implementation uses Git over ssh.

Also be aware that they automatically throw in the free trial additions which is fair enough, but you can drop these extras immediately if want and just get used to the free set of tools which include you're storage space and a ticketing system.

They also provide nice clear instructions on how to get going with setting up your repository, although putting your stuff under 'start' in their menu system confused me because I expected it to be under 'Home' or 'Your spaces' or something similar.

Berlios
Berlios offer Git and Subversion hosting (and other tools) for proper grown up Open Source projects. Too much hassle for me, which is probably their plan. Looks much like SourceForge. Have a read at the Wikipedia entry.

Bettercodes
Bettercodes is rather cool for a number of reasons. Firstly, they allow you multiple projects each with it's own Subversion or Git (Smart HTTP) repository. You can either make your code public, private of hidden and also invite others to work on your projects. You also get milestones and tasks and flat file storage. One word of caution; this site is in beta and although the repository stuff worked fine, there are definitely some teething problems. There's no word yet of a pricing plan, so get on there and give them some feedback.

Bitbucket
Bitbucket offers multiple, free, private project repositories with wikis and issue tracking. It's easy to use and reassuringly swish.

My only issue is that it only offers Mercurial repositories and I much prefer Git. That being said, let me just repeat. Bitbcket offers multiple, free, private project repositories with wikis and issue tracking."

Codeplex
Codeplex is Microsoft's version of proper grown up Open Source projects. Free for Open Source stuff and uses Mercurial or Subversion. Again, like Berlios,too much hassle for me.

Fedora Hosted
"Fedora Hosted is a project sponsored by the Fedora Project to allow upstream developers to host their code and collaborate online." See Berlios, Codeplex and grown ups. Not for me at this stage.

GitHub
GitHub is the most famous Git repository out there. Free plans are all public so other people will be able to see your code, but the interface, instructions etc are lovely and there's a nice repository explorer. Only Git (over ssh) is available.

Gitorious
Again, like GitHub, Gitorious is an attractive and easy to use site. Repositories are required to be under an open source licence (not just public) which may or may not suit your purposes. Only Git (over ssh) is available.

GNU Savannah
Savannah is another site for grown ups. This time focusing on free software projects (hence GNU). The rules here are particularly strictly enforced and you aren't even allowed to utilize non-free formats in your code. They provide repositories in CVS, GNU arch, Subversion, Git, Mercurial, Bazaar which is quite astonishing and there are loads of extra tools such as mailing lists and bug tracking etc. Looks like SourceForge. Don't get me wrong, this is a mighty site, but it doesn't suit everybody's requirements.

Is your's an Open Source Project?

Google Code
The sheer number of Google tools that integrate into this site must make it tempting for anyone. It's really useful, but as obvious as the advantages are, I'm going to mention a couple negative points here, but please take them in the context of an excellent site.

Firstly it only offers Mercurial and Subversion code repositories. These are perfectly good, but I still prefer Git.

Secondly projects have to be Open Source. This is not so different from lots of other sites mentioned here, but since Google offers almost all it's tools freely, including Gigs of space for email, sites, docs etc, I don't really understand why they don't offer a few megabytes for private code repositories. Any Google guys out there?

JavaForge
JavaForge looks excellent and the number of tools is impressive (" Web hosting, Document Management, Wiki, Forum, Online chat, Issue tracking integrated with optional Git, Mercurial or Subversion revision control"), but in short I don't do Java as a hobby and this is definitely a Java site. The Open Source rule applies. Java for grown ups.

LaunchPad
I haven't used LaunchPad, but it looks really rather useful. Firstly it's only for Open Source projects (for which it is free), and secondly it uses Bazaar as it's version control system. If you can work with those then I think you should have a good hard look at this offering as it offers stuff like Bug Tracking, Mailing Lists, Teams, FAQs, Code Review and more.

ProjectKenai
ProjectKenai is now owned by Oracle and is being closed/shifted/shut down.

SourceForge
The daddy of them all, SourceForge has some great statistics. I looked just now and it claims to be hosting 294,229 projects and had 3.8 million downloads today. If you don't mind people downloading your code for free or with an Open Source license, you can host it here. It offers all the version control systems already mentioned (and good old CVS) and gives you room for WIKI, files and other niceties.

Surely there's some kudos in just having a SourceForge Project. You could print your own t-shirts!

Anyway, don't forget about SourceForge, there's a reason the other grown up sites tend to emulate it.

Finally

Have I missed anything? Please ley me know.

There's some other information on wikipedia:

• http://en.wikipedia.org/wiki/Comparison_of_free_software_hosting_facilities

What a Smart Git

This article is a primer on Git protocols so that you get to grips with what specific Git repository providers are offering. I'll cover Code Repositories in a follow up article.

Traditionally Git lets you communicate with the Git Server (Repository) via four protocols:

• Local (file system)
• Secure Shell (SSH)
• Git
• Dumb HTTP

Local is probably little used because after all what's the point of having a repository for your code  on your hard disk if you're hard disk crashes, or your laptop gets stolen.

SSH and Git protocols worked just fine because they could talk to the server and get things done efficiently.

Dumb HTTP was a bit of a bodge because it could not be used to talk to the server and things were very inefficient (see the references below).

So, What's The Problem?

Firewalls, especially in the workplace, just hate it when people try to use anything except the standard ports. You know, HTTP, HTTPS, SMTP and after that they get sulky. Maybe, you'll be allowed to use FTP, but certainly not port 9418 (Git) or 22 (SSH).

Smart HTTP

The Smart HTTP protocol is really still just HTTP, but it's the server side that got smarter. Since Git version 1.6.6, Git servers can communicate with the client in a much smarter way using HTTP which makes it just as efficient as SSH or Git protocols AND allowing us to use Git anywhere we can browse the Internet; so long as the repository provider has things set up to use Smart HTTP.

If they aren't now, they soon will be.

References:

• http://progit.org/2010/03/04/smart-http.html
• http://progit.org/book/ch4-1.html