Disco/MapReduce: Using results of previous iteration as input to new iteration

Posted by muckabout on Stack Overflow See other posts from Stack Overflow or by muckabout
Published on 2010-04-02T11:36:16Z Indexed on 2010/04/02 11:53 UTC
Read the original article Hit count: 604

Filed under:

python

|

disco

|

mapreduce

Currently am implementing PageRank on Disco. As an iterative algorithm, the results of one iteration are used as input to the next iteration.

I have a large file which represents all the links, with each row representing a page and the values in the row representing the pages to which it links.

For Disco, I break this file into N chunks, then run MapReduce for one round. As a result, I get a set of (page, rank) tuples.

I'd like to feed this rank to the next iteration. However, now my mapper needs two inputs: the graph file, and the pageranks.

I would like to "zip" together the graph file and the page ranks, such that each line represents a page, it's rank, and it's out links.
Since this graph file is separated into N chunks, I need to split the pagerank vector into N parallel chunks, and zip the regions of the pagerank vectors to the graph chunks

This all seems more complicated than necessary, and as a pretty straightforward operation (with the quintessential mapreduce algorithm), it seems I'm missing something about Disco that could really simplify the approach.

Any thoughts?

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about disco

How to force remove a package if dpkg removal script fails?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I'm trying to remove a package where I deleted the /etc/init.d/disco-master file (in an attempt to remove the package manually). I want to remove the disco-master package. How do I do this now? This is what happens when I do sudo apt-get remove disco-master: removing disco-master ... invoke-rc.d:… >>> More
ayuda con disco duro

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
al momento de instalar el ubuntu me equivoque seleccionar el disco (en este caso quise instalar en el disco e:/) pero casi finalizar la instalacion ve dio error (cd defectuoso, grabe el cd en menos velocidad intente desde usb o error en el disco) inicie con windows xp y el disco D:/ no se muestra… >>> More
Disco/MapReduce: Using results of previous iteration as input to new iteration

as seen on Stack Overflow - Search for 'Stack Overflow'
Currently am implementing PageRank on Disco. As an iterative algorithm, the results of one iteration are used as input to the next iteration. I have a large file which represents all the links, with each row representing a page and the values in the row representing the pages to which it links. For… >>> More
Reading Data from DDFS ValueError: No JSON object could be decoded

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm running dozens of map reduce jobs for a number of different purposes using disco. My data has grown enormous and I thought I would try using DDFS for a change rather than standard txt files. I've followed the DISCO map/reduce example Counting Words as a map/reduce job, without to much difficulty… >>> More
¿Cómo utilizar 2º disco duro de Ubuntu?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
Recientemente he instalado Windows y Ubuntu 12.04. Tengo 2 discos duros ( 160 GB y 320 GB). Datos de la instalación: Partición Sistema de archivos Etiqueta Tamaño Punto de montaja /dev/sda1 ntfs Sistema Windows 21 GB ^/dev/sda2 extendida 128,54… >>> More