Replacing whitespace with sed in a CSV (to use w/ postgres copy command)
- by Wells
I iterate through a collection of CSV files in bash, running:
iconv --from-code=ISO-8859-1 --to-code=UTF-8 ${FILE} | \
sed -e 's/\"//g' | \
sed -e 's/, /,/g' \
> ${FILE}.utf8
Running iconv to fix UTF-8 characters, then the first sed call removes the double quote characters, and the final sed call is supposed to remove leading and trailing whitespace around the commas.
HOWEVER, I still have a line like this in the saved file:
FALSE,,,, 2.40,,
The COPY command in postgres is kind of dumb, so it thinks " 2.40" is not valid syntax for a numeric value.
Where am I going wrong w/ my processing of the CSV file? Thanks!
