wget not behaving via IPC::Open3 vs bash

Posted by Ryley on Stack Overflow See other posts from Stack Overflow or by Ryley
Published on 2010-06-08T20:11:37Z Indexed on 2010/06/08 22:12 UTC
Read the original article Hit count: 334

Filed under:
|
|

I'm trying to stream a file from a remote website to a local command and am running into some problems when trying to detect errors.

The code looks something like this:

use IPC::Open3;

my @cmd = ('wget','-O','-','http://10.10.1.72/index.php');#any website will do here

my ($wget_pid,$wget_in,$wget_out,$wget_err);
if (!($wget_pid = open3($wget_in,$wget_out,$wget_err,@cmd))){
    print STDERR "failed to run open3\n";
    exit(1)
}
close($wget_in);
my @wget_outs = <$wget_out>;
my @wget_errs = <$wget_err>;
print STDERR "wget stderr: ".join('',@wget_errs);
#page and errors outputted on the next line, seems wrong
print STDERR "wget stdout: ".join('',@wget_outs);

#clean up after this, not shown is running the filtering command, closing and waitpid'ing

When I run that wget command directly from the command-line and redirect stderr to a file, something sane happens - the stdout will be the downloaded page, the stderr will contain the info about opening the given page.

wget -O - http://10.10.1.72/index.php 2> stderr_test_file

When I run wget via open3, I'm getting both the page and the info mixed together in stdout. What I expect is the loaded page in one stream and STDERR from wget in another.

I can see I've simplified the code to the point where it's not clear why I want to use open3, but the general plan is that I wanted to stream stdout to another filtering program as I received it, and then at the end I was going to read the stderr from both wget and the filtering program to determine what, if anything went wrong.

    Other important things:
  • I was trying to avoid writing the wget'd data to a file, then filtering that file to another file, then reading the output.
  • It's key that I be able to see what went wrong, not just reading $? >> 8 (i.e. I have to tell the user, hey, that IP address is wrong, or isn't the right kind of website, or whatever).
  • Finally, I'm choosing system/open3/exec over other perl-isms (i.e. backticks) because some of the input is provided by untrustworthy users.

© Stack Overflow or respective owner

Related posts about perl

Related posts about ipc