Parallelism in Python
Posted
by fmark
on Stack Overflow
See other posts from Stack Overflow
or by fmark
Published on 2010-06-07T08:22:40Z
Indexed on
2010/06/07
8:32 UTC
Read the original article
Hit count: 336
What are the options for achieving parallelism in Python? I want to perform a bunch of CPU bound calculations over some very large rasters, and would like to parallelise them. Coming from a C background, I am familiar with three approaches to parallelism:
- Message passing processes, possibly distributed across a cluster, e.g. MPI.
- Explicit shared memory parallelism, either using pthreads or fork(), pipe(), et. al
- Implicit shared memory parallelism, using OpenMP.
Deciding on an approach to use is an exercise in trade-offs.
In Python, what approaches are available and what are their characteristics? Is there a clusterable MPI clone? What are the preferred ways of achieving shared memory parallelism? I have heard reference to problems with the GIL, as well as references to tasklets.
In short, what do I need to know about the different parallelization strategies in Python before choosing between them?
© Stack Overflow or respective owner