Low coupling processing big quantities of data
- by vitalik
Usually I achieve low coupling by creating classes that exchange lists, sets, and maps between them. Now I am developing a batch application and I can't put all the data inside a data structure because there isn't enough memory. I have to read and process one chunk of data and then going to the next one. So having low coupling is much more difficult because I have to check somewhere if there is still data to read, etc.
What I am using now is:
Source - Process - Persist
The classes that process have to ask to the Source classes if there are more rows to read.
What are the best practices and or useful patterns in such situations?
I hope I am explaining myself, if not tell me.
