Dealing with a badly formatted CSV file

Posted by Josh K on Stack Overflow See other posts from Stack Overflow or by Josh K
Published on 2010-04-15T21:16:35Z Indexed on 2010/04/15 21:23 UTC
Read the original article Hit count: 278

Filed under:

csv

|

php

|

mysql

|

table

|

formatting-nightmare

I have an exceptionally bad CSV file. Although I "solved" the problem in the end by manually writing scripts to process and reprocess this specific file I wanted to know if there were any other solutions out there.

You have a CSV file that has all the fields terminated by | (pipe) characters. Running a quick check shows you that there are 53 fields in the file. The person who gave you the file claims there there are only 28 fields. Not all of the fields have information in them. For example there are five custom_field_{num} fields which may or may not have data.

How would you get this into a database nicely?

The ideal solution (and one I searched high and low for) would be to just throw it all into a table with no column names or specifications. Then remove any columns that were completely blank and then give them titles and specifications.

© Stack Overflow or respective owner

Related posts about csv

DBD::CSV: How can I generate different behavior with the two f_ext-options ".csv" and ".csv/r"?

as seen on Stack Overflow - Search for 'Stack Overflow'
This is from the DBD::File-documentation: f_ext This attribute is used for setting the file extension where (CSV) files are opened. There are several possibilities. DBI:CSV:f_dir=data;f_ext=.csv In this case, DBD::File will open only table.csv if both table.csv and table exist… >>> More
SUPER CSV write bean to CSV.

as seen on Stack Overflow - Search for 'Stack Overflow'
Here is my class, public class FreebasePeopleResults { public String intendedSearch; public String weight; public Double heightMeters; public Integer age; public String type; public String parents; public String profession; public String alias; public String children; public String siblings; public… >>> More
export to csv using fastercsv and CSV::Writer (Ruby on Rails)

as seen on Stack Overflow - Search for 'Stack Overflow'
What am I trying to do: export data to csv. I have a form which allows user to select the format (from a drop down menu). So based on the selection of the format the ouput is displayed using a ajax call. Works fine for html but when I select the format as csv I don't see any pop up on the screen… >>> More
How can i make changes to this file Encoding?

as seen on Super User - Search for 'Super User'
I have these 3 files 21/08/2014 07:15 PM 122 Tw2AWK.csv 21/08/2014 07:15 PM 125 Tw2Notepad.csv 21/08/2014 07:15 PM 119 Tw2REPL.csv C:\myfilesfile Tw2AWK.csv TwREPL.csv Tw2Notepad.csv Tw2AWK.csv; UTF-8 Unicode text, with CRLF line terminators Tw2REPL… >>> More
Unix sort keys cause performance problems

as seen on Super User - Search for 'Super User'
My data: It's a 71 MB file with 1.5 million rows. It has 6 fields All six fields combine to form a unique key - so that's what I need to sort on. Sort statement: sort -t ',' -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 -k6,6 -o output.csv input.csv The problem: If I sort without keys, it takes 30 seconds… >>> More

Related posts about php

Magento, NGINX, PHP-FPM, APC, MEMCACHED, 16gb Ram CentOS, Spiking PHP-FPM to 100% CPU

as seen on Server Fault - Search for 'Server Fault'
I have been trying to resolve my issue of spiking cpu caused by php-fpm processes. I've reduced the php-fpm config settings to: pm = ondemand pm.max_children = 12 pm.start_servers = 2 pm.min_spare_servers = 2 pm.max_spare_servers = 10 pm.max_requests = 500 php_admin_value[memory_limit] = 128M Problem… >>> More
PHP Pear Installation on CentOS

as seen on Server Fault - Search for 'Server Fault'
[root@ip ~]# yum install php-pear* Reducing CentOS-5 Testing to included packages only Finished Setting up Install Process Package 1:php-pear-1.8.1-2.el5.centos.noarch already installed and latest versio … >>> More
Apache configurations for php "AddType text/html php" or "AddType application/x-httpd-php php .php"

as seen on Server Fault - Search for 'Server Fault'
I am taking over an application server and discover that it contain the following settings: AddType text/html php Although it works, but my understanding is that it should set as following: AddType application/x-httpd-php php .php What are the key differences between the two settings?… >>> More
mod_rewrite settings causes server to throw HTTP 500 errors instead of 404

as seen on Server Fault - Search for 'Server Fault'
Hello. I have a server with VBulletin forum (working under Apache 2.2, CentOS). The default settings for it in .htaccess are as follows: RewriteEngine on RewriteCond %{HTTP_HOST} ^gsmforum\.ru RewriteRule (.*) http://www.gsmforum.ru/$1 [R=301,L] # If you are having problems or are using VirtualDocumentRoot… >>> More
Problems installing Memcache (PECL extension)

as seen on Server Fault - Search for 'Server Fault'
I have installed memcached fine, and now I will need to install PECL extension memcache. Im running RedHat x86_64 es5. The installation gives me this: downloading memcache-2.2.6.tgz ... Starting to download memcache-2.2.6.tgz (35,957 bytes) ..........done: 35,957 bytes 11 source files, building running:… >>> More