Twitter Streaming API - Almost Useful

Twitter's Streaming API is a splendid idea. It gives developer's access to a good splodge of Tweets and let's us filter them in different ways. 

However it has a few flaws.

For Twitter, the benefit of a Streaming API is probably one of scalability. Instead of us using the old REST API to ask for specific data and causing tens of thousands of data look ups, all they do is give us the end of their own data stream once it has been used in house and is now halfway across the back garden. All they have to do is allow us to filter the stream a bit to make it a bit more relevant to our needs and put an absolute cap on the throughput (about 1% for most of us.)

This looks good. 1% is enough for most development needs and streams down your connection like a low bandwidth radio station. I don't really know what the bandwidth or download is, but it's not much. Once we've developed our new and wonderful website, then we can ask, or possibly pay, Twitter to turn up the pressure a bit.

So, now let's look at the filters.

There are several ways that the stream can be filtered

• follow - filter by userid
• track - filter by keyword
• location - filter by geographic location
• retweets - just the retweets ma'am
• links - only tweets containing a link
• random - I think they just mean unfiltered

On the face of it this is pretty useful. You could filter the stream by adding the location of your home town so that you can get the jist of what is important to your townsfolk. You could filter the stream by the keyword 'Elvis' to see who has spotted Elvis lately and plot the results on a map.

My own first idea was inspired by the M5 motorway accident just a few miles from where I live and astounded that even in this day and age, the scale of the incident was only uncovered somewhat slowly. Surely what the quantity and content of the tweets from the people who were NOT in the incident itself, would help scale the incident? So what I wanted to do was:

1. Listen to what people Tweet at known traffic jam locations.
2. Identify some fingerprint of common words, maybe "traffic, jam, standstill, miles" or whatever.
3. Look for clusters of these words near to motorways.
4. Plot the clusters based on the location of the phones that made the tweets.

So I get an updating list of current traffic incidents from the Traffic England RSS feed and start listening within 5 kms of the stated location using a bounding box centred on the location. The first thing I notice is that some of the tweets are in the bounding box, some are around it and some, quite a few are far away, sometimes 200km away or more!

I don't know how Twitter do the filtering, but evidently it's based on something fairly broad brushed. I can live with that maybe, all I have to do is check the Tweets geo location, which is added if you tweet by most modern phones. I was expecting most of the useful tweets to be from a mobile anyway, so that would work if I can just get used to maybe 2% of the tweets actually being in the bounding box. 2% 0f 1% is after all only 0.02% of all Tweets or 1 Tweet in 5000.

So what happens if I assume that the word "traffic" will occur in the most useful Tweets? This is either bad science or common sense data filtering depending on how you look at it.


Alas, it appears that the Streaming API does not allow you to filter by location AND keyword! All you can do is do an OR filter, so I can filter the stream to include certain areas OR certain keywords, but not certain keywords within a certain location.

To me this just renders the API all but useless, but no doubt you lot are much smarter than I and will dazzle me with your great ideas.

Please let me know.

Google App Engine Python2.5 Development in Ubuntu 11.10

How to set up a development environment for Google App Engine, Python2.5 in Ubuntu 11.10.
Firstly, sorry if the code below is badly formatted, but there's a clearer copy of the code at the bottom.

Python2.5 isn't in the Ubuntu 11.10 sources by default so in a console:

sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update

and install python2.5 and some other stuff:

sudo apt-get install python2.5 python-virtualenv virtualenvwrapper python-pip

Next for project 'oinkyoinker':

export CURRENT=oinkyoinker
mkvirtualenv --no-site-packages --distribute --python python2.5 ${CURRENT}
workon ${CURRENT}
cdvirtualenv
pip install fabric yolk ipython readline

Now download GAE and fix the path:

wget -O /tmp/gae.zip http://googleappengine.googlecode.com/files/google_appengine_1.5.5.zip
unzip /tmp/gae.zip
echo "../../../google_appengine" > lib/python2.5/site-packages/gae.pth

and create a simple file server app:

mkdir -p application/static

echo """application: oinkyoinker
version: 1
runtime: python
api_version: 1
default_expiration: "7d"
handlers:
- url: /
  static_dir: static
""" > application/app.yaml

and run it:

./google_appengine/dev_appserver.py application/

Hope this helped you too.

MapReduce in Python

Over the last while I've been using NodeJS, MongoDB and Google's App Engine. Everything seems to have MapReduce functionality and frankly I was missing it when I went back to do some work in plain old Python. This is a nice short example of how it can be done.

The basic format that you need is something like this:

Now, a simple working example using the usual MapReduce example:

The output will look something like this:

• I got most of this from the following blog: http://goo.gl/nW8iA .
• Docs are here: http://docs.python.org/library/multiprocessing.html

MapReduce with Mongoose and CoffeeScript

After searching the InterWeb for a decent MapReduce example coded in CoffeeScript I came up blank and decided to write my own. This one uses Mongoose too - well why use anything else?

I haven't written a whole lot of explanation in the text, but commented the code quite heavily so you can copy and paste and still have it with you.


Load Mongoose and connect to your database.Describe your schema and model and then load up some dummy data.

Now for the MapReduce stuff. If you still don't get MapReduce then you will once you read this Star Trek based article.

Define a Map function.

Define a Reduce function.

Get some output

The results will now be in a collection called "results" which you can look at with JMongoBrowser or whatever.

I hope this is of some use to you. If there are any corrections or improvements then please post something below.

Staying Up Late - CoffeeScript and Zappa

Over the last couple of years NodeJS has become the darling of "coding for pleasure" crowd. All that cross fertilization from the client side buffs into the server side and the server side coders ranting about the whole event-driveness of NodeJS.

Yes, it's the next big thing. Well, it's probably not, but it's certainly "A" next big thing or maybe we'll all get bored dream up something else with a catchy name... "perl for parallelism" or "Friendly Java (without the memory leaks)", or what about "C+=3" or "Oil for Android", "Lube for Palm OS". Well you know what I mean.

We see something; we explore it and once it starts getting a bit boring we convince ourselves and then our customers that we need to use some new technology or other. It's all true of course, or partly true, or lies.

Anyway, I'm not here to rant about that, after all I as guilty as anyone else.

A Messy Affair

What I want to rant about is Zappa which is my own personal cure for technical disfunction at the moment. Well that sounds real bad, but I've been doing this job for years and it's only these little affairs over the years that keep me coming home; "Assembler", "C", "C++", "Ruby", "Python", how I remember their sweet names, but sooner or later they all start to look like their mother and have to go.

Zappa is a young virgin of a technology, only now reaching a pubescent version 0.3. Essentially it's a lightweight framework built on top of known technologies such like NodeJS, CoffeeScript, Express and Socket.IO in much the same way as Python's Flask is built on top of Werkzeug, Jinja 2 and so on.

It's that Socket.IO bit that makes this one so sexy (I'm overdoing the procreation metaphors aren't I?). Socket.IO provides cross-browser WebSocket goodness which means you can do stuff like stop using the Model-View-Controller paradigm and start thinking outside the cubical with your "Model-View-Controller-View-Controller-Controller-View Paradigm" or whatever. Mind you, that's when you'll realise that MVC wasn't about limiting your options, it was about simplifying them.

Anyway, back to Zappa itself. Go and have a look, there are great examples that will get your inventive juices flowing (stop it!) and inspire you to spend a little time alone locked it the bathroom... I mean office. The community is starting to grow and the mailing list is big enough to be useful, but not so big as to be faceless and scarey.

Have fun but take precautions.

Back of an Envelope Node.js Framework - Discuss

I had a quiet afternoon today and while "keeping up with recent developments in my field" or as you might call it "wasting my afternoon googling" I got to thinking what I'd like a Node.js MVC framework should or could contain.

How design shouldn't be done

Firstly, I've been playing with Node.js for about three weeks now and I'm impressed, but I can't seem to find a framework that I'm happy with. Backbone looked good, but I wasted three days on it and it was too much like hard work. Similarly Spine which tried to be less complicated, but eventually failed. Finally I really wanted Brunch to work but it was a release behind backbone (which is a major component of it) and I didn't want to get lost in the mismatching documentation (the Backbone release was pretty major).

I have spent a whole lot of time over the last couple of years using python frameworks, especially Flask and I like their approach of starting with a good foundation (in their case Werkzeug) and adding the bits that are still needed. Could that work with Node? I mean writing a basic HTTP server looks pretty straight forward and there are tons of modules to add the other functionality.

So with this possibly naive starting position I grabbed my pen and scribble down some stuff I thought I'd need. Now bear in mind that I haven't much used any of these modules yet, and I pinched several of them from the bits that go to make up Brunch.


I'd appreciate any feedback, suggestions and so on that those more learned than I might care to offer. Few of you will be less learned I imagine.


Here are the transcribed notes with some links:

Server

• Simple Node.js
• Express

Routing

• choose a module
• Express?

Data Store

• Mongo?
• CouchDB?
• Barricane-db

Models

• JS Object/JSON?
• ORM?

Views
    -CSS

• lesscss ?
• Stylus ?

    -Templating

• mustache(js) ?
• Eco (coffeescript)

Controllers

• Plain Old JS
• CoffeeScript?
• jQuery
• Underscore.js
• Quip (responses) ?

Middleware

• how? Express

Other Stuff

• node-boilerplate?
• Stitch (glue)
• Cake

If you spot any typos, inaccuracies or just plain lies then please let me know.

Binding Updates to Model Changes in Backbone/CoffeeScript

My previous post showed a View that looked like this:



But what happens if we have a more complicated app and something else changes the Model?

Well as I said I'm just learning myself here, co the correct way that should be handled is to bind Model changes to the Views render function which looks like this:



Note the bind instruction which means that every time the counter changes, it calls the render function. You might also notice the fat arrow now on the render which make "this" still mean the View rather than the caller (which would be the Model otherwise.)

The explanation for all this is at http://jashkenas.github.com/coffee-script under "Function Binding".