Wierd characters in exported csv files when converting
- by Ahue
Hey guys,
I came across a problem I cannot solve on my own concerning the downloadable csv formatted trends data files from Google Insights for Search.
I'm to lazy to reformat the files I4S gives me manually what means: Extracting the section with the actual trends data and reformatting the columns so that I can use it with a modelling program I do for school.
So I wrote a tiny script the should do the work for me: Taking a file, do some magic and give me a new file in proper format.
What it's supposed to do is reading the file contents, extracting the trends section, splitting it by newlines, splitting each line and then reorder the columns and maybe reformat them.
When looking at a untouched I4S csv file it looks normal containing CR LF caracters at line breaks (maybe thats only because I'm using Windows).
When just reading the contents and then writing them to a new file using the script wierd asian characters appear between CR and LF. I tried the script with a manually written similar looking file and even tried a csv file from Google Trends and it works fine.
I use Python and the script (snippet) I used for the following example
looks like this:
# Read from an input file
file = open(file,"r")
contents = file.read()
file.close()
cfile = open("m.log","w+")
cfile.write(contents)
cfile.close()
Has anybody an idea why those characters appear??? Thank you for you help!
I'll give you and example:
First few lines of I4S csv file:
Web Search Interest: foobar
Worldwide; 2004 - present
Interest over time
Week foobar
2004-01-04 - 2004-01-10 44
2004-01-11 - 2004-01-17 44
2004-01-18 - 2004-01-24 37
2004-01-25 - 2004-01-31 40
2004-02-01 - 2004-02-07 49
2004-02-08 - 2004-02-14 51
2004-02-15 - 2004-02-21 45
2004-02-22 - 2004-02-28 61
2004-02-29 - 2004-03-06 51
2004-03-07 - 2004-03-13 48
2004-03-14 - 2004-03-20 50
2004-03-21 - 2004-03-27 56
2004-03-28 - 2004-04-03 59
Output file when reading and writing contents:
Web Search Interest: foobar
??????????? ? ? ? ?????????
????????? ???? ??????
Week foobar
?? ?? ?? ? ? ? ?? ??? ?????
2004-01-11 - 2004-01-17 44
?? ?? ???? ? ? ?? ?????????
2004-01-25 - 2004-01-31 40
?? ?? ?? ? ? ? ?? ?? ??????
2004-02-08 - 2004-02-14 51
?? ?? ???? ? ? ?? ?????????
2004-02-22 - 2004-02-28 61
?? ?? ???? ? ? ?? ?? ??????
2004-03-07 - 2004-03-13 48
?? ?? ???? ? ? ?? ??? ?? ??
2004-03-21 - 2004-03-27 56
?? ?? ???? ? ? ?? ?? ??????
2004-04-04 - 2004-04-10 69
?? ?? ???? ? ? ?? ?????????
2004-04-18 - 2004-04-24 51
?? ?? ???? ? ? ?? ?? ??????
2004-05-02 - 2004-05-08 56
?? ?? ?? ? ? ? ?? ?????????
2004-05-16 - 2004-05-22 54
?? ?? ???? ? ? ?? ?????????
2004-05-30 - 2004-06-05 74
?? ?? ?? ? ? ? ?? ?????????
2004-06-13 - 2004-06-19 50
?? ?? ??? ? ? ?? ?????????
2004-06-27 - 2004-07-03 58
?? ?? ?? ? ? ? ?? ??? ?????
2004-07-11 - 2004-07-17 59
?? ?? ???? ? ? ?? ?????????
