How do you save the state of a computation in while using SNOW (or Multicore or ...)

Posted by James on Stack Overflow See other posts from Stack Overflow or by James
Published on 2010-04-14T01:54:55Z Indexed on 2010/04/14 2:03 UTC
Read the original article Hit count: 299

Filed under:

From hard experience I've found it useful to occasionally save the state of my long computations to disk to start them up later if something fails. Can I do this in a distributed computation package in R (like SNOW or multicore)?
It does not seem clear how this could be done since the master is collecting things from the slaves in a non-transparent way.

Related posts about distributed-computing

Distributed Computing Framework (.NET) - Specifically for CPU Instensive operations

as seen on Stack Overflow - Search for 'Stack Overflow'
I am currently researching the options that are available (both Open Source and Commercial) for developing a distributed application. "A distributed system consists of multiple autonomous computers that communicate through a computer network." Wikipedia The application is focused on distributing… >>> More
Where do I start with distributed computing?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm interested in learning techniques for distributed computing. As a Java developer, I'm probably willing to start with Hadoop. Could you please recommend some books/tutorials/articles to begin with? >>> More
running a python script where dependencies are not avail: distributed computing

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I have access to a grid (running condor) that would (potentially) allow to very substantially reduce how long by nltk based nlp tasks take. unfortunately, i dont have root access on the cluster so cannot install new packages, only run whatever is available on the linux boxes. python is of course… >>> More
Java - System design with distributed Queues and Locks

as seen on Programmers - Search for 'Programmers'
Looking for inputs to evaluate a design for a system (java) which would have a distributed queue serving several (but not too many) nodes. These nodes would process objects present in the distributed queue and on occasion require a distributed lock across the cluster on an arbitrary (distributed)… >>> More
Can an issue tracking system be distributed?

as seen on Programmers - Search for 'Programmers'
I was thinking about issue tracking software like Redmine, Trac or even the one that is in Fossil and something hit me: Is there a reason why Redmine and Trac are not possible to be distributed? Or maybe it's possible and I just don't know how it's possible? If it's not possible, why? By distributed… >>> More

Developer IT

How do you save the state of a computation in while using SNOW (or Multicore or ...) - Developer IT

How do you save the state of a computation in while using SNOW (or Multicore or ...)

r

distributed-computing

parallel-processing

Related posts about r

Related posts about distributed-computing

Distributed Computing Framework (.NET) - Specifically for CPU Instensive operations

Where do I start with distributed computing?

running a python script where dependencies are not avail: distributed computing

Java - System design with distributed Queues and Locks

Can an issue tracking system be distributed?

Categories cloud