How can I implement MapReduce using shell commands?

Posted by alex on Stack Overflow See other posts from Stack Overflow or by alex
Published on 2010-04-16T16:28:12Z Indexed on 2010/04/17 13:13 UTC
Read the original article Hit count: 437

Filed under:

parallel-processing

|

rpc

|

mapreduce

How do you execute a Unix shell command (e.g awk one liner) on a cluster in parallel (step 1) and collect the results back to a central node (step 2)?

Update: I've just found http://blog.last.fm/2009/04/06/mapreduce-bash-script It seems to do exactly what I need.

© Stack Overflow or respective owner

Related posts about parallel-processing

Intel Core 2 duo / AMD athlon X2 parallel processing capability

as seen on Super User - Search for 'Super User'
Does Intel Core 2 duo/ AMD athlon X2 really have 2 separate processors? i.e are they capable of doing real parallel processing? What I don't understand is the difference when somebody says Cores or Processors. >>> More
JVM (embarrasingly) parallel processing libraries/tools

as seen on Stack Overflow - Search for 'Stack Overflow'
I am looking for something that will make it easy to run (correctly coded) embarrassingly parallel JVM code on a cluster (so that I can use Clojure + Incanter). I have used Parallel Python in the past to do this. We have a new PBS cluster and our admin will soon set up IPython nodes that use PBS… >>> More
Parallel processing in R 2.11 Windows 64-bit using SNOW not quite working

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm running R 2.11 64-bit on a WinXP64 machine with 8 processors. With R 2.10.1 the following code spawned 6 R processes for parallel processing: require(foreach) require(doSNOW) cl = makeCluster(6, type='SOCK') registerDoSNOW(cl) bl2 = foreach(i=icount(length(unqmrno))) %dopar% { (Some code… >>> More
Databases supporting parallel processing across multiple servers

as seen on Server Fault - Search for 'Server Fault'
I need a database engine that can utilize multiple servers for processing a single SQL query in parallel. So far I know that this is possible with the some engines, though none of them are feasible for me either because of pricing or missing features. The engines currently known to me are: MS SQL… >>> More
Multithreading/Parallel Processing in PHP

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a PHP script that will generate a report using PHPExcel from data queried from a MySQL DB. Currently, it is linear in processing in that it gets the data back from MySQL, reads in the Excel template, writes the data to the template, then outputs it. I have optimized the code to the point that… >>> More

Related posts about rpc

JSON-RPC and Json-rpc service discovery specifications

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I'm going to implement JSON-PRC web service. I need specifications for this. So far I had found only one resource that can be called as real specifications: JSON-RPC 1.0 http://json-rpc.org/wiki/specification Proposal of JSON-PRC 2.0: http://groups.google.com/group/json-rpc/web/json-rpc-2-0… >>> More
Slow XML-RPC in Windows 7 with XML-RPC.NET

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm considering to use XML-RPC.NET to communicate with a Linux XML-RPC server written in Python. I have tried a sample application (MathApp) from Cook Computing's XML-RPC.NET but it took 30 seconds for the app to add two numbers within the same LAN with server. I have also tried to run a simple… >>> More
Windows RPC vs XML-RPC

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there any benchmark about encoding/decoding certain common typed data in Microsoft RPC NDR engine (DCE 1.1) in comparison with that in XML-RPC-C/C++ in the de-facto C/C++ implementation in XML-RPC? Actually I have to choose between Windows RPC and XML-RPC-C/C++ to implement my own common object… >>> More
Cross-site json rpc : Python server side and Mozilla extension using Javascript client side

as seen on Stack Overflow - Search for 'Stack Overflow'
hello, I am building a mozilla extension that contacts a python application on a remote server to send and receive data. The python application can be used using xml-rpc from a python console.I am given the task to design a json-rpc that would contact the same application.Making the python server… >>> More
GWT - occasional com.google.gwt.user.client.rpc.SerializationException

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello we are haunted by occasional occurences of exceptions such as: com.google.gwt.user.client.rpc.SerializationException: Type 'xxx' was not assignable to 'com.google.gwt.user.client.rpc.IsSerializable' and did not have a custom field serializer.For security purposes, this type will not be… >>> More