python, cluster computing, design help [closed]
- by j dawg
I would like to create my own parallel computing server. Can you please point me to some resources I can use to help me develop my server.
Sorry, like I said I need help getting started.
Yes, I am limited to python, I cannot use C. I am using a bunch of workstations and I want to use all the cpus in those machines.
So what I am looking for is blog posts, books, articles that can help me develop my own client/server tools to send code from the client to the servers and spawn python processes based on the number of cpus.
I know how to do the subprocessing/multiprocessing part of the program, I do not know how to create the server that will take the client's requests. I also need to figure out what is the best way to handle sending file data, like netcdf files or other spatial data.
Any suggestions very welcome.