Read non-blocking from multiple fifos in parallel

Posted by Ole Tange on Super User See other posts from Super User or by Ole Tange
Published on 2012-10-03T14:39:58Z Indexed on 2014/05/28 9:38 UTC
Read the original article Hit count: 187

Filed under:
|
|
|

I sometimes sit with a bunch of output fifos from programs that run in parallel. I would like to merge these fifos. The naïve solution is:

cat fifo* > output

But this requires the first fifo to complete before reading the first byte from the second fifo, and this will block the parallel running programs.

Another way is:

(cat fifo1 & cat fifo2 & ... ) > output

But this may mix the output thus getting half-lines in output.

When reading from multiple fifos, there must be some rules for merging the files. Typically doing it on a line by line basis is enough for me, so I am looking for something that does:

parallel_non_blocking_cat fifo* > output

which will read from all fifos in parallel and merge the output on with a full line at a time.

I can see it is not hard to write that program. All you need to do is:

  1. open all fifos
  2. do a blocking select on all of them
  3. read nonblocking from the fifo which has data into the buffer for that fifo
  4. if the buffer contains a full line (or record) then print out the line
  5. if all fifos are closed/eof: exit
  6. goto 2

So my question is not: can it be done?

My question is: Is it done already and can I just install a tool that does this?

© Super User or respective owner

Related posts about merge

Related posts about cat