Threading best practice when using SFTP in C#
- by Christian
Ok,
this is more one of these "conceptual questions", but I hope I got some pointers in the right direction. First the desired scenario:
I want to query an SFTP server for directory and file lists
I want to upload or download files simulaneously
Both things are pretty easy using a SFTP class provided by Tamir.SharpSsh, but if I only use one thread, it is kind of slow. Especially the recursion into subdirs gets very "UI blocking", because we are talking about 10.000 of directories.
My basic approach is simple, create some kind of "pool" where I keep 10 open SFTP connections. Then query the first worker for a list of dirs. If this list was obtained, send the next free workers (e.g. 1-10, first one is also free again) to get the subdirectory details. As soon as there is a worker free, send him for the subsubdirs. And so on...
I know the ThreadPool, simple Threads and did some Tests. What confuses me a little bit is the following: I basically need...
A list of threads I create, say 10
Connect all threads to the server
If a connection drops, create a new thread / sftp client
If there is work to do, take the first free thread and handle the work
I am currently not sure about the implementation details, especially the "work to do" and the "maintain list of threads" parts.
Is it a good idea to:
Enclose the work in an object, containing a job description (path) and a callback
Send the threads into an infinite loop with 100ms wait to wait for work
If SFTP is dead, either revive it, or kill the whole thread and create a new one
How to encapsulate this, do I write my own "10ThreadsManager" or are there some out
Ok, so far...
Btw, I could also use PRISM events and commands, but I think the problem is unrelated. Perhaps the EventModel to signal a done processing of a "work package"...
Thanks for any ideas, critic..
Chris
