Map Reduce Frameworks/Infrastructure
- by Johannes Rudolph
Map Reduce is a pattern that seems to get a lot of traction lately and I start to see it manifest in one of my projects that is focused on an event processing pipeline (iPhone Accelerometer and GPS data). I needed to built a lot of infrastructure for this project, in fact it overweighs the logic code interacting with it by 2x. Some of the components I built where EventProcessors (with in- and output plus buffering, timing etc.), multiplexers and aggregators.
This leads me to my question what the "common" required infrastrucutre for map reduce is. Since I am working with .Net a lot I can see map reduce infrastructure built into the Framework and language constructs. Functional languages support this paradigm per se. It seems every language can be used with map reduce, some have better support than others, others again are built around that concept (e.g. Go). And there are Frameworks like Apache Hadoop to support map reduce.