Parallel Tasking Concurrency with Dependencies on Python like GNU Make
- by Brian Bruggeman
I'm looking for a method or possibly a philosophical approach for how to do something like GNU Make within python. Currently, we utilize makefiles to execute processing because the makefiles are extremely good at parallel runs with changing single option: -j x. In addition, gnu make already has the dependency stacks built into it, so adding a secondary processor or the ability to process more threads just means updating that single option. I want that same power and flexibility in python, but I don't see it.
As an example:
all: dependency_a dependency_b dependency_c
dependency_a: dependency_d
stuff
dependency_b: dependency_d
stuff
dependency_c: dependency_e
stuff
dependency_d: dependency_f
stuff
dependency_e:
stuff
dependency_f:
stuff
If we do a standard single thread operation (-j 1), the order of operation might be:
dependency_f -> dependency_d -> dependency_a -> dependency_b -> dependency_e \
-> dependency_c
For two threads (-j 2), we might see:
1: dependency_f -> dependency_d -> dependency_a -> dependency_b
2: dependency_e -> dependency_c
Does anyone have any suggestions on either a package already built or an approach? I'm totally open, provided it's a pythonic solution/approach.
Please and Thanks in advance!