Sunday, 22 January 2012

Get Real Data from the Semantic Web - Finding Resources

In my last article, I briefly explained how to get data from a resource using python and SPARQL. This article explains how to find the resource in the first place.
Have you ever been taught how to knit? I you have, then you'll know that you are not usually taught how to cast on (or start off) on your first lesson. That's because it much easier to learn how to knit than it is to cast on.

So it is with the Semantic Web. Once you have a resource URL, it's reasonably easy to extract information linked to that resource, but finding the starting resource is a bit trickier.
So let's just recap how we might get the abstract description for London from DBpedia.

If we know the URL then that's pretty straight forward:
(If you want to follow this tutorial, then you had better copy the sparql.py file from there.)


RDF types for the DBpedia entry for London
If you don't however, then you'll have to search for it. According to the dbpedia entry, London is many things, including a owl:Thing, there are a lot of Things out there, probably enough to make even the DBpdia  endpoint time out, so let's choose something more restrictive such as yago:Locations but not too restrictive, for example yago:BritishCapitals.


Just to be a smart ass as I finish off, you can get both at the same time by doing this, but don't forget that doing this will stress the SPARQL endpoint more than is probably necessary. Be kind.

Thursday, 19 January 2012

Get Real Data from the Semantic Web

Semantic Web this, Semantic Web that, what actual use is the Semantic Web in the real world? I mean how can you actually use it?

If you haven't heard the term "Semantic Web" over the last couple of years then you must have been in... well somewhere without this interweb they're all talking about.

Basically, by using metadata (see RDF), disparate bits of data floating around the web can be joined up. In otherwords they stop being disparate. Better than that, theoretically you can query the connections between the data and get lots of lovely information back. This last bit is done via SPARQL, and yes, the QL does stand for Query Language.

I say theoretically because in reality it's a bit of a pain. I may be an intelligent agent capable of finding linked bits of data through the web, but how exactly would you do that in python.

It is possible to use rdflib to find information, but it's very long winded. It's much easier to use SPARQLWrapper andin fact in the simple example below, I've used a SPARQLWrapperWrapper to make asking for lots of similarly sourced data, in this case DBPedia, even easier.

To use this try importing the DBpediaEndpoint and feeding it some SPARQL:

Your homework is - How do you identify the resource_uri in the first place?

That's for another evening.

Tuesday, 17 January 2012

Github: Who needs it?

Do you ever think that you just don't want all your code on Github? I mean it's only a quick hack right?

Truth is, once you start using git you probably use it automatically for all your code, but you don't always want all your code floating around the net. What about those hard-coded email addresses and API tokens, or those references to your private net servers?

The answer is probably so simple that you have just overlooked it. You don't need to set up a local git server or hire one from Amazon. All you need to do is use DropBox or Ubuntu One as your remote origin repository.

Here's how, using Ubuntu One on Ubuntu:

Write a short shell script something like this and save it on your path as repo.sh.

Now when you want to create a new repository all you have to do is:

If you use Python and virtualenv you may be interested in the slightly extended script at http://pythonic-apis.blogspot.com/2012/01/using-ubuntu-one-as-git-repository.html.

Thursday, 15 December 2011

Twitter Streaming API - Almost Useful

Twitter's Streaming API is a splendid idea. It gives developer's access to a good splodge of Tweets and let's us filter them in different ways. 

However it has a few flaws.

For Twitter, the benefit of a Streaming API is probably one of scalability. Instead of us using the old REST API to ask for specific data and causing tens of thousands of data look ups, all they do is give us the end of their own data stream once it has been used in house and is now halfway across the back garden. All they have to do is allow us to filter the stream a bit to make it a bit more relevant to our needs and put an absolute cap on the throughput (about 1% for most of us.)

This looks good. 1% is enough for most development needs and streams down your connection like a low bandwidth radio station. I don't really know what the bandwidth or download is, but it's not much. Once we've developed our new and wonderful website, then we can ask, or possibly pay, Twitter to turn up the pressure a bit.

So, now let's look at the filters.

There are several ways that the stream can be filtered
  • follow - filter by userid
  • track - filter by keyword
  • location - filter by geographic location
  • retweets - just the retweets ma'am
  • links - only tweets containing a link
  • random - I think they just mean unfiltered
On the face of it this is pretty useful. You could filter the stream by adding the location of your home town so that you can get the jist of what is important to your townsfolk. You could filter the stream by the keyword 'Elvis' to see who has spotted Elvis lately and plot the results on a map.

My own first idea was inspired by the M5 motorway accident just a few miles from where I live and astounded that even in this day and age, the scale of the incident was only uncovered somewhat slowly. Surely what the quantity and content of the tweets from the people who were NOT in the incident itself, would help scale the incident? So what I wanted to do was:
  1. Listen to what people Tweet at known traffic jam locations.
  2. Identify some fingerprint of common words, maybe "traffic, jam, standstill, miles" or whatever.
  3. Look for clusters of these words near to motorways.
  4. Plot the clusters based on the location of the phones that made the tweets.
So I get an updating list of current traffic incidents from the Traffic England RSS feed and start listening within 5 kms of the stated location using a bounding box centred on the location. The first thing I notice is that some of the tweets are in the bounding box, some are around it and some, quite a few are far away, sometimes 200km away or more!

I don't know how Twitter do the filtering, but evidently it's based on something fairly broad brushed. I can live with that maybe, all I have to do is check the Tweets geo location, which is added if you tweet by most modern phones. I was expecting most of the useful tweets to be from a mobile anyway, so that would work if I can just get used to maybe 2% of the tweets actually being in the bounding box. 2% 0f 1% is after all only 0.02% of all Tweets or 1 Tweet in 5000.

So what happens if I assume that the word "traffic" will occur in the most useful Tweets? This is either bad science or common sense data filtering depending on how you look at it.


Alas, it appears that the Streaming API does not allow you to filter by location AND keyword! All you can do is do an OR filter, so I can filter the stream to include certain areas OR certain keywords, but not certain keywords within a certain location.

To me this just renders the API all but useless, but no doubt you lot are much smarter than I and will dazzle me with your great ideas.

Please let me know.








Sunday, 6 November 2011

Google App Engine Python2.5 Development in Ubuntu 11.10

How to set up a development environment for Google App Engine, Python2.5 in Ubuntu 11.10.
Firstly, sorry if the code below is badly formatted, but there's a clearer copy of the code at the bottom.

Python2.5 isn't in the Ubuntu 11.10 sources by default so in a console:

sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update

and install python2.5 and some other stuff:
sudo apt-get install python2.5 python-virtualenv virtualenvwrapper python-pip

Next for project 'oinkyoinker':
export CURRENT=oinkyoinker
mkvirtualenv --no-site-packages --distribute --python python2.5 ${CURRENT}
workon ${CURRENT}
cdvirtualenv
pip install fabric yolk ipython readline

Now download GAE and fix the path:
wget -O /tmp/gae.zip http://googleappengine.googlecode.com/files/google_appengine_1.5.5.zip
unzip /tmp/gae.zip
echo "../../../google_appengine" > lib/python2.5/site-packages/gae.pth
and create a simple file server app:
mkdir -p application/static

echo """application: oinkyoinker
version: 1
runtime: python
api_version: 1
default_expiration: "7d"
handlers:
- url: /
  static_dir: static
""" > application/app.yaml
and run it:
./google_appengine/dev_appserver.py application/
Hope this helped you too.

MapReduce in Python

Over the last while I've been using NodeJS, MongoDB and Google's App Engine. Everything seems to have MapReduce functionality and frankly I was missing it when I went back to do some work in plain old Python. This is a nice short example of how it can be done.

The basic format that you need is something like this:

Now, a simple working example using the usual MapReduce example:

The output will look something like this:



Wednesday, 19 October 2011

MapReduce with Mongoose and CoffeeScript

After searching the InterWeb for a decent MapReduce example coded in CoffeeScript I came up blank and decided to write my own. This one uses Mongoose too - well why use anything else?

I haven't written a whole lot of explanation in the text, but commented the code quite heavily so you can copy and paste and still have it with you.


Load Mongoose and connect to your database.Describe your schema and model and then load up some dummy data.

Now for the MapReduce stuff. If you still don't get MapReduce then you will once you read this Star Trek based article.

Define a Map function.

Define a Reduce function.

Get some output

The results will now be in a collection called "results" which you can look at with JMongoBrowser or whatever.

I hope this is of some use to you. If there are any corrections or improvements then please post something below.