Parallel doseq for Clojure
Posted
by
andrew cooke
on Stack Overflow
See other posts from Stack Overflow
or by andrew cooke
Published on 2012-06-10T14:56:43Z
Indexed on
2012/06/11
4:40 UTC
Read the original article
Hit count: 193
I haven't used multithreading in Clojure at all so am unsure where to start.
I have a doseq
whose body can run in parallel. What I'd like is for there always to be 3 threads running (leaving 1 core free) that evaluate the body in parallel until the range is exhausted. There's no shared state, nothing complicated - the equivalent of Python's multiprocessing would be just fine.
So something like:
(dopar 3 [i (range 100)]
; repeated 100 times in 3 parallel threads...
...)
Where should I start looking? Is there a command for this? A standard package? A good reference?
So far I have found pmap
, and could use that (how do I restrict to 3 at a time? looks like it uses 32 at a time - no, source says 2 + number of processors), but it seems like this is a basic primitive that should already exist somewhere.
clarification: I really would like to control the number of threads. I have processes that are long-running and use a fair amount of memory, so creating a large number and hoping things work out OK isn't a good approach (example which uses a significant chunk available mem).
update: Starting to write a macro that does this, and I need a semaphore (or a mutex, or an atom i can wait on). Do semaphores exist in Clojure? Or should I use a ThreadPoolExecutor? It seems odd to have to pull so much in from Java - I thought parallel programming in Clojure was supposed to be easy... Maybe I am thinking about this completely the wrong way? Hmmm. Agents?
© Stack Overflow or respective owner