MPI Large Data all to all transfer

Posted by csslayer on Stack Overflow See other posts from Stack Overflow or by csslayer
Published on 2010-06-13T08:33:04Z Indexed on 2010/06/13 8:42 UTC
Read the original article Hit count: 225

Filed under:
|
|

My application of MPI has some process that generate some large data. Say we have N+1 process (one for master control, others are workers), each of worker processes generate large data, which is now simply write to normal file, named file1, file2, ..., fileN. The size of each file may be quite different. Now I need to send all fileM to rank M process to do the next job, So it's just like all to all data transfer.

My problem is how should I use MPI API to send these files efficiently? I used to use windows share folder to transfer these before, but I think it's not a good idea.

I have think about MPI_file and MPI_All_to_all, but these functions seems not to be so suitable for my case. Simple MPI_Send and MPI_Recv seems hard to be used because every process need to transfer large data, and I don't want to use distributed file system for now.

© Stack Overflow or respective owner

Related posts about parallel

Related posts about mpi