The basic format that you need is something like this:
Now, a simple working example using the usual MapReduce example:
The output will look something like this:
- I got most of this from the following blog: http://goo.gl/nW8iA .
- Docs are here: http://docs.python.org/library/multiprocessing.html
Hey there!
ReplyDeleteWhile your multiprocessing usage is nice, it seems to me that you missed the point of MapReduce in your example because most of the work actually happens in partition().
A more classic example would look something like this: https://gist.github.com/1406554
There you have proper partitioning BEFORE the mapping happens, solving sub-problems in the map phase and finally reducing sub-results to a single aggregate result.
Is there a way to reduce the part of the partition part?
ReplyDeleteEasy jumpstart to python Mapreduce : http://pymapreduce.com
ReplyDelete