Replacing whitespace with sed in a CSV (to use w/ postgres copy command)
Posted
by
Wells
on Stack Overflow
See other posts from Stack Overflow
or by Wells
Published on 2012-04-14T23:20:45Z
Indexed on
2012/04/14
23:30 UTC
Read the original article
Hit count: 187
I iterate through a collection of CSV files in bash, running:
iconv --from-code=ISO-8859-1 --to-code=UTF-8 ${FILE} | \
sed -e 's/\"//g' | \
sed -e 's/, /,/g' \
> ${FILE}.utf8
Running iconv
to fix UTF-8 characters, then the first sed
call removes the double quote characters, and the final sed
call is supposed to remove leading and trailing whitespace around the commas.
HOWEVER, I still have a line like this in the saved file:
FALSE,,,, 2.40,,
The COPY
command in postgres is kind of dumb, so it thinks " 2.40" is not valid syntax for a numeric value.
Where am I going wrong w/ my processing of the CSV file? Thanks!
© Stack Overflow or respective owner