Read non-blocking from multiple fifos in parallel
Posted
by
Ole Tange
on Super User
See other posts from Super User
or by Ole Tange
Published on 2012-10-03T14:39:58Z
Indexed on
2014/05/28
9:38 UTC
Read the original article
Hit count: 187
I sometimes sit with a bunch of output fifos from programs that run in parallel. I would like to merge these fifos. The naïve solution is:
cat fifo* > output
But this requires the first fifo to complete before reading the first byte from the second fifo, and this will block the parallel running programs.
Another way is:
(cat fifo1 & cat fifo2 & ... ) > output
But this may mix the output thus getting half-lines in output.
When reading from multiple fifos, there must be some rules for merging the files. Typically doing it on a line by line basis is enough for me, so I am looking for something that does:
parallel_non_blocking_cat fifo* > output
which will read from all fifos in parallel and merge the output on with a full line at a time.
I can see it is not hard to write that program. All you need to do is:
- open all fifos
- do a blocking select on all of them
- read nonblocking from the fifo which has data into the buffer for that fifo
- if the buffer contains a full line (or record) then print out the line
- if all fifos are closed/eof: exit
- goto 2
So my question is not: can it be done?
My question is: Is it done already and can I just install a tool that does this?
© Super User or respective owner