Data Integration Solution?
Posted
by
Shlomo
on Stack Overflow
See other posts from Stack Overflow
or by Shlomo
Published on 2013-06-19T18:32:15Z
Indexed on
2013/06/26
16:21 UTC
Read the original article
Hit count: 154
data
|integration
At my company we have a number of data feeds and processing that run on any given day. The number of feeds and processing steps is starting to out-number the ability to manage it ad-hoc as it is managed currently. Is there a good solution that helps with logging and managing/scheduling dependencies?
For example:
A: When file x is FTP dropped into directory D1, kick off processing step B
B: Load flat file into DB1
C: When file y is FTP dropped into directory D2, kick off processing Step D
D: Load flat file into DB11
E: When B and D are done, churn through the data, and load new data into DB111.
F: When Step E is done, launch application process P
G: etc...
I want those steps to run at the appropriate times, not to mention if B fails, there's no reason to run steps E & F, but I could still run C & D. When I re-run B successfully, it should trigger just E & F to re-run, not C & D.
We're a .NET/C#/Sql Server shop, and I'm already familiar with SSIS. Is that really the best there is? That manages steps well, but not external dependencies, or logging. Open source (.NET) preferred, but not required.
© Stack Overflow or respective owner