Data Integration Solution?

Posted by Shlomo on Stack Overflow See other posts from Stack Overflow or by Shlomo
Published on 2013-06-19T18:32:15Z Indexed on 2013/06/26 16:21 UTC
Read the original article Hit count: 151

Filed under:
|

At my company we have a number of data feeds and processing that run on any given day. The number of feeds and processing steps is starting to out-number the ability to manage it ad-hoc as it is managed currently. Is there a good solution that helps with logging and managing/scheduling dependencies?

For example:
A: When file x is FTP dropped into directory D1, kick off processing step B
B: Load flat file into DB1
C: When file y is FTP dropped into directory D2, kick off processing Step D
D: Load flat file into DB11
E: When B and D are done, churn through the data, and load new data into DB111.
F: When Step E is done, launch application process P
G: etc...

I want those steps to run at the appropriate times, not to mention if B fails, there's no reason to run steps E & F, but I could still run C & D. When I re-run B successfully, it should trigger just E & F to re-run, not C & D.

We're a .NET/C#/Sql Server shop, and I'm already familiar with SSIS. Is that really the best there is? That manages steps well, but not external dependencies, or logging. Open source (.NET) preferred, but not required.

© Stack Overflow or respective owner

Related posts about data

Related posts about integration