Communicate between separate MPI-Programs

Posted by Fyg on Stack Overflow See other posts from Stack Overflow or by Fyg
Published on 2010-05-23T14:50:36Z Indexed on 2010/05/23 15:10 UTC
Read the original article Hit count: 247

Filed under:

sockets

|

mpi

I have the following problem:

Program 1 has a huge amount of data, say 10GB. The data in question consists of large integer- and double-arrays. Program 2 has 1..n MPI processes that use tiles of this data to compute results.

How can I send the data from program 1 to the MPI Processes?

Using File I/O is out of question. The compute node has sufficient RAM.

© Stack Overflow or respective owner

Related posts about sockets

Performance implications of Synchronous Sockets vs Asynchronous Sockets

as seen on Stack Overflow - Search for 'Stack Overflow'
We are trying to build an SMTP Server to receive mail notifications from various clients over internet. As each of the communication will be longer and it needs to log everything, doing this Asynchronous way is little challenging as well as by using Socket's Asynchronous methods we are not sure of… >>> More
Multiple INET sockets (mulple IP's too) connected to UNIX sockets

as seen on Server Fault - Search for 'Server Fault'
HOST = same host all the time, accepts multiple connection. I have a dedicated server and I will buy extra IP's. Socket 1 connects to HOST:PORT, from IP-1 Socket 2 connects to HOST:PORT, from IP-1 Socket 3 connects to HOST:PORT, from IP-1 Socket 4 connects to HOST:PORT, from IP-2 Socket 5 connects… >>> More
Multiple INET sockets (mulple IP's too) connected to UNIX sockets

as seen on Stack Overflow - Search for 'Stack Overflow'
HOST = same host all the time, accepts multiple connection. I have a dedicated server and I will buy extra IP's. Socket 1 connects to HOST:PORT, from IP-1 Socket 2 connects to HOST:PORT, from IP-1 Socket 3 connects to HOST:PORT, from IP-1 Socket 4 connects to HOST:PORT, from IP-2 Socket 5 connects… >>> More
Do WebSockets have exclusive access to their sockets?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm curious to know if, after a WebSocket has been established (after having received the proper handshake from a server that supports them), whether or not the TCP socket used by the "WebSocket connection" is used exclusively by the WebSocket, or if the browser may still make regular HTTP requests… >>> More
What Parallel computing APIs make good use of sockets?

as seen on Stack Overflow - Search for 'Stack Overflow'
My program uses sockets, what Parallel computing APIs could I use that would help me without obligating me to go from sockets to anything else? When we are on a cluster with a special, non-socket infrastructure system this API would emulate something like sockets but using that infrastructure (so… >>> More

Related posts about mpi

Message Passing Interface (MPI)

as seen on Daniel Moth - Search for 'Daniel Moth'
So you have installed your cluster and you are done with introductory material on Windows HPC. Now you want to develop an application with the most common programming model: Message Passing Interface.The MPI programming model is a standard with implementations from many vendors. For newbies (like… >>> More
How do I debug an MPI program?

as seen on Stack Overflow - Search for 'Stack Overflow'
I have an MPI program which compiles and runs, but I would like to step through it to make sure nothing bizarre is happening. Ideally, I would like a simple way to attach GDB to any particular process, but I'm not really sure whether that's possible or how to do it. An alternative would be having… >>> More
trying to build Boost MPI, but the lib files are not created. What's going on?

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to run a program with Boost MPI, but the thing is I don't have the .lib. So I try to create one by following the instruction at http://www.boost.org/doc/libs/1_43_0/doc/html/mpi/getting_started.html#mpi.config The instruction says "For many users using LAM/MPI, MPICH, or OpenMPI, configuration… >>> More
MPI: is there mpi libraries capable of message compression?

as seen on Stack Overflow - Search for 'Stack Overflow'
Sometimes MPI is used to send low-entropy data in messages. So it can be useful to try to compress messages before sending it. I know that MPI can work on very fast networks (10 Gbit/s and more), but many MPI programs are used with cheap network like 0,1G or 1Gbit/s Ethernet and with cheap (slow,… >>> More
Slides and code for MPI Cluster Debugger

as seen on Daniel Moth - Search for 'Daniel Moth'
I've blogged before about the MPI Cluster Debugger in VS2010 that facilitates launching the application on the cluster and attaching the debugger (btw, a shorter version of the screencast I link to there, is here).There have been requests for the code I use in the screencast, so please find a ZIP… >>> More