There is a folder that contains 1000 of small text files.

I aim to parse and process all of them while more files are being populated in to the folder.

My intention is to multithread this operation as the single threaded prototype took 6 minutes to process 1000 files.

I like to have reader and writer thread(s) as following : while the reader thread(s) are reading the files, I d like to have writer thread(s) to process them. Once the reader is started reading a file, I d like to mark it as being processed, such as by renaming it, once it s read, rename it to completed.

How to approach such multithreaded application ?

Is it better to use a distributed hash table or a queue?

Which data structure to use that would avoid locks?

Would you have a better approach to this scheme that you like to share?