Trouble with piping through sed
- by Joel
I am having trouble piping through sed. Once I have piped output to sed, I cannot pipe the output of sed elsewhere.
wget -r -nv http://127.0.0.1:3000/test.html
Outputs:
2010-03-12 04:41:48 URL:http://127.0.0.1:3000/test.html [99/99] -> "127.0.0.1:3000/test.html" [1]
2010-03-12 04:41:48 URL:http://127.0.0.1:3000/robots.txt [83/83] -> "127.0.0.1:3000/robots.txt" [1]
2010-03-12 04:41:48 URL:http://127.0.0.1:3000/shop [22818/22818] -> "127.0.0.1:3000/shop.29" [1]
I pipe the output through sed to get a clean list of URLs:
wget -r -nv http://127.0.0.1:3000/test.html 2>&1 | grep --line-buffered -v ERROR | sed 's/^.*URL:\([^ ]*\).*/\1/g'
Outputs:
http://127.0.0.1:3000/test.html
http://127.0.0.1:3000/robots.txt
http://127.0.0.1:3000/shop
I would like to then dump the output to file, so I do this:
wget -r -nv http://127.0.0.1:3000/test.html 2>&1 | grep --line-buffered -v ERROR | sed 's/^.*URL:\([^ ]*\).*/\1/g' > /tmp/DUMP_FILE
I interrupt the process after a few seconds and check the file, yet it is empty.
Interesting, the following yields no output (same as above, but piping sed output through cat):
wget -r -nv http://127.0.0.1:3000/test.html 2>&1 | grep --line-buffered -v ERROR | sed 's/^.*URL:\([^ ]*\).*/\1/g' | cat
Why can I not pipe the output of sed to another program like cat?
