How to Profile R Code that Includes SNOW Cluster

Posted by James on Stack Overflow See other posts from Stack Overflow or by James
Published on 2010-04-12T17:53:47Z Indexed on 2010/04/12 18:53 UTC
Read the original article Hit count: 333

Filed under:

r

|

parallel-processing

|

distributed-computing

Hi,

I have a nested loop that I'm using foreach, DoSNOW, and a SNOW socket cluster to solve for. How should I go about profiling the code to make sure I'm not doing something grossly inefficient.

Also is there anyway to measure the data flows going between the master and nodes in a Snow cluster?

Thanks,

James

Related posts about parallel-processing

Intel Core 2 duo / AMD athlon X2 parallel processing capability

as seen on Super User - Search for 'Super User'
Does Intel Core 2 duo/ AMD athlon X2 really have 2 separate processors? i.e are they capable of doing real parallel processing? What I don't understand is the difference when somebody says Cores or Processors. >>> More
JVM (embarrasingly) parallel processing libraries/tools

as seen on Stack Overflow - Search for 'Stack Overflow'
I am looking for something that will make it easy to run (correctly coded) embarrassingly parallel JVM code on a cluster (so that I can use Clojure + Incanter). I have used Parallel Python in the past to do this. We have a new PBS cluster and our admin will soon set up IPython nodes that use PBS… >>> More
Parallel processing in R 2.11 Windows 64-bit using SNOW not quite working

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm running R 2.11 64-bit on a WinXP64 machine with 8 processors. With R 2.10.1 the following code spawned 6 R processes for parallel processing: require(foreach) require(doSNOW) cl = makeCluster(6, type='SOCK') registerDoSNOW(cl) bl2 = foreach(i=icount(length(unqmrno))) %dopar% { (Some code… >>> More
Databases supporting parallel processing across multiple servers

as seen on Server Fault - Search for 'Server Fault'
I need a database engine that can utilize multiple servers for processing a single SQL query in parallel. So far I know that this is possible with the some engines, though none of them are feasible for me either because of pricing or missing features. The engines currently known to me are: MS SQL… >>> More
Multithreading/Parallel Processing in PHP

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a PHP script that will generate a report using PHPExcel from data queried from a MySQL DB. Currently, it is linear in processing in that it gets the data back from MySQL, reads in the Excel template, writes the data to the template, then outputs it. I have optimized the code to the point that… >>> More

Developer IT

How to Profile R Code that Includes SNOW Cluster - Developer IT

How to Profile R Code that Includes SNOW Cluster

r

parallel-processing

distributed-computing

Related posts about r

Related posts about parallel-processing

Intel Core 2 duo / AMD athlon X2 parallel processing capability

JVM (embarrasingly) parallel processing libraries/tools

Parallel processing in R 2.11 Windows 64-bit using SNOW not quite working

Databases supporting parallel processing across multiple servers

Multithreading/Parallel Processing in PHP

Categories cloud