Replacing whitespace with sed in a CSV (to use w/ postgres copy command)

Posted by Wells on Stack Overflow See other posts from Stack Overflow or by Wells
Published on 2012-04-14T23:20:45Z Indexed on 2012/04/14 23:30 UTC
Read the original article Hit count: 187

Filed under:
|
|

I iterate through a collection of CSV files in bash, running:

iconv --from-code=ISO-8859-1 --to-code=UTF-8 ${FILE} | \
    sed -e 's/\"//g' | \
    sed -e 's/, /,/g' \
    > ${FILE}.utf8

Running iconv to fix UTF-8 characters, then the first sed call removes the double quote characters, and the final sed call is supposed to remove leading and trailing whitespace around the commas.

HOWEVER, I still have a line like this in the saved file:

FALSE,,,, 2.40,,

The COPY command in postgres is kind of dumb, so it thinks " 2.40" is not valid syntax for a numeric value.

Where am I going wrong w/ my processing of the CSV file? Thanks!

© Stack Overflow or respective owner

Related posts about linux

Related posts about postgresql