Generating Gibberish in Python - Markov Chains

Markov Chains allow us to generate letter sequences which can contain sensible keywords and bulk up your blog. 

I was once taught to speed read. A sham, a party trick, but interesting too. Essentially you just have to practice gliding over pages in a book or whatever and pick out key information (as opposed to just words). You do this by focusing on starts of paragraphs, which tend to be information rich, and capitalized words which tend to be names and useful information and ignore the rest.

What I'm suggesting here is almost the opposite, generating key words which are beefed up by noise in such a way that, I believe, most search engines will not see the join.

Markov Chains have been around for the most of a hundred years and work by splitting a text into groups of letters of a fixed length and record which groups follow that group anywhere in the text. To generate gibberish all you have to do is choose any group of letters and randomly pick one of the other groups that you know can follow that group; loop until you get enough text. If you use short groups of letters (say two) the text contains less information from the original text than if you use longer groups (say six), because with short groups there are more choices available. You should try this yourself to see what I mean, but I find that a group length of about four is a good start.

Now to the code. Firstly I would love to see your improvements, so please let me know or branch it from GitHub. And don't forget to read the extra information at the bottom of this page to understand how it works.

Essentially you just have to practice gliding over pages in a book or whatever and pick out key information (as opposite, generate gibberish all you use short groups of letters of the join. Firstly ins have been around for the most of a hundred years and use longer groups (say six), because obviously there are more choices available. Essentially you just have to do is choose any group of letters and randomly pick one of the other groups that you know can follow that groups of letters of a fixed length and randomly pick one of a fixed length and randomly pick one of a fixed length and randomly pick one of the code. 

Chains have been around for the most of a hundred years and words which are beefed up by noise in such a way that, I believe, most the join. GitHub op until you have to do is choose any group of letter sequences which can contain sensible keywords and capitalized words which groups follow that groups of letters (say two) the text contains less information from the original text that you know can follow that groups of letters (say two) the text contains less information from the original text. 

What up by noise in such a way that, I believe, most the other groups than if you get enough text than if you use short group anywhere information rich, and capitalized words which tent to the opposed to just words which tent to be names and work by splitting a text into groups of paragraphs, which tend to be information from the original text than if you use short group anywhere in the most of a hundred years and words which are been around for the most of a hundred years and work by splitting a text into group; loop until you have to do is choose any group of letter sequences which can contain sensible keywords and bulk up your blog. Chains have beefed up by noise in such a way that, I believe, most the opposed to just words which groups (say six), because obviously there are more choices available.

Get Real Data from the Semantic Web - Finding Resources

In my last article, I briefly explained how to get data from a resource using python and SPARQL. This article explains how to find the resource in the first place.
Have you ever been taught how to knit? I you have, then you'll know that you are not usually taught how to cast on (or start off) on your first lesson. That's because it much easier to learn how to knit than it is to cast on.

So it is with the Semantic Web. Once you have a resource URL, it's reasonably easy to extract information linked to that resource, but finding the starting resource is a bit trickier.
So let's just recap how we might get the abstract description for London from DBpedia.

If we know the URL then that's pretty straight forward:
(If you want to follow this tutorial, then you had better copy the sparql.py file from there.)

RDF types for the DBpedia entry for London

If you don't however, then you'll have to search for it. According to the dbpedia entry, London is many things, including a owl:Thing, there are a lot of Things out there, probably enough to make even the DBpdia  endpoint time out, so let's choose something more restrictive such as yago:Locations but not too restrictive, for example yago:BritishCapitals.


Just to be a smart ass as I finish off, you can get both at the same time by doing this, but don't forget that doing this will stress the SPARQL endpoint more than is probably necessary. Be kind.

Get Real Data from the Semantic Web

Semantic Web this, Semantic Web that, what actual use is the Semantic Web in the real world? I mean how can you actually use it?

If you haven't heard the term "Semantic Web" over the last couple of years then you must have been in... well somewhere without this interweb they're all talking about.

Basically, by using metadata (see RDF), disparate bits of data floating around the web can be joined up. In otherwords they stop being disparate. Better than that, theoretically you can query the connections between the data and get lots of lovely information back. This last bit is done via SPARQL, and yes, the QL does stand for Query Language.

I say theoretically because in reality it's a bit of a pain. I may be an intelligent agent capable of finding linked bits of data through the web, but how exactly would you do that in python.

It is possible to use rdflib to find information, but it's very long winded. It's much easier to use SPARQLWrapper andin fact in the simple example below, I've used a SPARQLWrapperWrapper to make asking for lots of similarly sourced data, in this case DBPedia, even easier.

To use this try importing the DBpediaEndpoint and feeding it some SPARQL:

Your homework is - How do you identify the resource_uri in the first place?

That's for another evening.

Github: Who needs it?

Do you ever think that you just don't want all your code on Github? I mean it's only a quick hack right?

Truth is, once you start using git you probably use it automatically for all your code, but you don't always want all your code floating around the net. What about those hard-coded email addresses and API tokens, or those references to your private net servers?

The answer is probably so simple that you have just overlooked it. You don't need to set up a local git server or hire one from Amazon. All you need to do is use DropBox or Ubuntu One as your remote origin repository.

Here's how, using Ubuntu One on Ubuntu:

Write a short shell script something like this and save it on your path as repo.sh.

Now when you want to create a new repository all you have to do is:

If you use Python and virtualenv you may be interested in the slightly extended script at http://pythonic-apis.blogspot.com/2012/01/using-ubuntu-one-as-git-repository.html.

Twitter Streaming API - Almost Useful

Twitter's Streaming API is a splendid idea. It gives developer's access to a good splodge of Tweets and let's us filter them in different ways. 

However it has a few flaws.

For Twitter, the benefit of a Streaming API is probably one of scalability. Instead of us using the old REST API to ask for specific data and causing tens of thousands of data look ups, all they do is give us the end of their own data stream once it has been used in house and is now halfway across the back garden. All they have to do is allow us to filter the stream a bit to make it a bit more relevant to our needs and put an absolute cap on the throughput (about 1% for most of us.)

This looks good. 1% is enough for most development needs and streams down your connection like a low bandwidth radio station. I don't really know what the bandwidth or download is, but it's not much. Once we've developed our new and wonderful website, then we can ask, or possibly pay, Twitter to turn up the pressure a bit.

So, now let's look at the filters.

There are several ways that the stream can be filtered

• follow - filter by userid
• track - filter by keyword
• location - filter by geographic location
• retweets - just the retweets ma'am
• links - only tweets containing a link
• random - I think they just mean unfiltered

On the face of it this is pretty useful. You could filter the stream by adding the location of your home town so that you can get the jist of what is important to your townsfolk. You could filter the stream by the keyword 'Elvis' to see who has spotted Elvis lately and plot the results on a map.

My own first idea was inspired by the M5 motorway accident just a few miles from where I live and astounded that even in this day and age, the scale of the incident was only uncovered somewhat slowly. Surely what the quantity and content of the tweets from the people who were NOT in the incident itself, would help scale the incident? So what I wanted to do was:

1. Listen to what people Tweet at known traffic jam locations.
2. Identify some fingerprint of common words, maybe "traffic, jam, standstill, miles" or whatever.
3. Look for clusters of these words near to motorways.
4. Plot the clusters based on the location of the phones that made the tweets.

So I get an updating list of current traffic incidents from the Traffic England RSS feed and start listening within 5 kms of the stated location using a bounding box centred on the location. The first thing I notice is that some of the tweets are in the bounding box, some are around it and some, quite a few are far away, sometimes 200km away or more!

I don't know how Twitter do the filtering, but evidently it's based on something fairly broad brushed. I can live with that maybe, all I have to do is check the Tweets geo location, which is added if you tweet by most modern phones. I was expecting most of the useful tweets to be from a mobile anyway, so that would work if I can just get used to maybe 2% of the tweets actually being in the bounding box. 2% 0f 1% is after all only 0.02% of all Tweets or 1 Tweet in 5000.

So what happens if I assume that the word "traffic" will occur in the most useful Tweets? This is either bad science or common sense data filtering depending on how you look at it.


Alas, it appears that the Streaming API does not allow you to filter by location AND keyword! All you can do is do an OR filter, so I can filter the stream to include certain areas OR certain keywords, but not certain keywords within a certain location.

To me this just renders the API all but useless, but no doubt you lot are much smarter than I and will dazzle me with your great ideas.

Please let me know.

Google App Engine Python2.5 Development in Ubuntu 11.10

How to set up a development environment for Google App Engine, Python2.5 in Ubuntu 11.10.
Firstly, sorry if the code below is badly formatted, but there's a clearer copy of the code at the bottom.

Python2.5 isn't in the Ubuntu 11.10 sources by default so in a console:

sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update

and install python2.5 and some other stuff:

sudo apt-get install python2.5 python-virtualenv virtualenvwrapper python-pip

Next for project 'oinkyoinker':

export CURRENT=oinkyoinker
mkvirtualenv --no-site-packages --distribute --python python2.5 ${CURRENT}
workon ${CURRENT}
cdvirtualenv
pip install fabric yolk ipython readline

Now download GAE and fix the path:

wget -O /tmp/gae.zip http://googleappengine.googlecode.com/files/google_appengine_1.5.5.zip
unzip /tmp/gae.zip
echo "../../../google_appengine" > lib/python2.5/site-packages/gae.pth

and create a simple file server app:

mkdir -p application/static

echo """application: oinkyoinker
version: 1
runtime: python
api_version: 1
default_expiration: "7d"
handlers:
- url: /
  static_dir: static
""" > application/app.yaml

and run it:

./google_appengine/dev_appserver.py application/

Hope this helped you too.

MapReduce in Python

Over the last while I've been using NodeJS, MongoDB and Google's App Engine. Everything seems to have MapReduce functionality and frankly I was missing it when I went back to do some work in plain old Python. This is a nice short example of how it can be done.

The basic format that you need is something like this:

Now, a simple working example using the usual MapReduce example:

The output will look something like this:

• I got most of this from the following blog: http://goo.gl/nW8iA .
• Docs are here: http://docs.python.org/library/multiprocessing.html