Challege: merging csv files intelligently!

Posted by Evenz495 on Stack Overflow See other posts from Stack Overflow or by Evenz495
Published on 2010-03-31T14:18:03Z Indexed on 2010/03/31 14:33 UTC
Read the original article Hit count: 235

Filed under:

csv

|

database

|

data

We are in the middle of changing web store platform and we need to import products' data from different sources.

We currently have several different csv files from different it systems/databases because each system is missing some information. Fortunatly the product ids are the same so it's possible to relate the data using ids.

We need to merge this data into one big csv file so we can import in into our new e-commerce site.

My question: is there a general approach when you need to merge csv files with related data into one csv file? Are there any applications or tools that helps you out?

© Stack Overflow or respective owner

Related posts about csv

DBD::CSV: How can I generate different behavior with the two f_ext-options ".csv" and ".csv/r"?

as seen on Stack Overflow - Search for 'Stack Overflow'
This is from the DBD::File-documentation: f_ext This attribute is used for setting the file extension where (CSV) files are opened. There are several possibilities. DBI:CSV:f_dir=data;f_ext=.csv In this case, DBD::File will open only table.csv if both table.csv and table exist… >>> More
SUPER CSV write bean to CSV.

as seen on Stack Overflow - Search for 'Stack Overflow'
Here is my class, public class FreebasePeopleResults { public String intendedSearch; public String weight; public Double heightMeters; public Integer age; public String type; public String parents; public String profession; public String alias; public String children; public String siblings; public… >>> More
export to csv using fastercsv and CSV::Writer (Ruby on Rails)

as seen on Stack Overflow - Search for 'Stack Overflow'
What am I trying to do: export data to csv. I have a form which allows user to select the format (from a drop down menu). So based on the selection of the format the ouput is displayed using a ajax call. Works fine for html but when I select the format as csv I don't see any pop up on the screen… >>> More
How can i make changes to this file Encoding?

as seen on Super User - Search for 'Super User'
I have these 3 files 21/08/2014 07:15 PM 122 Tw2AWK.csv 21/08/2014 07:15 PM 125 Tw2Notepad.csv 21/08/2014 07:15 PM 119 Tw2REPL.csv C:\myfilesfile Tw2AWK.csv TwREPL.csv Tw2Notepad.csv Tw2AWK.csv; UTF-8 Unicode text, with CRLF line terminators Tw2REPL… >>> More
Unix sort keys cause performance problems

as seen on Super User - Search for 'Super User'
My data: It's a 71 MB file with 1.5 million rows. It has 6 fields All six fields combine to form a unique key - so that's what I need to sort on. Sort statement: sort -t ',' -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 -k6,6 -o output.csv input.csv The problem: If I sort without keys, it takes 30 seconds… >>> More

Related posts about database

SQL SERVER Retrieve and Explore Database Backup without Restoring Database Idera virtual database

as seen on Dot net Slackers - Search for 'Dot net Slackers'
I recently downloaded Ideras SQL virtual database, and tested it. There are a few things about this tool which caught my attention.My ScenarioIt is quite common in real life that sometimes observing or retrieving older data is necessary; however, it had changed as time passed by. The full database… >>> More
Cloning A Database On The Same Server Using Rman Duplicate From Active Database

as seen on Oracle Blogs - Search for 'Oracle Blogs'
To clone a database using Rman we used to require an existing Rman backup, on 11g we can clone databases using the "from active" database option. In this case we do not require an existing backup, the active datafiles will be used as the source for the clone. In order to clone with the source database… >>> More
cPickle ImportError: No module named multiarray

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I'm using cPickle to save my Database into file. The code looks like that: def Save_DataBase(): import cPickle from scipy import * from numpy import * a=Results.VersionName #filename='D:/results/'+a[a.find('/')+1:-a.find('/')-2]+Results.AssType[:3]+str(random.randint(0,100))+Results.Distribution+"… >>> More
SQL SERVER – 2008 – Introduction to Snapshot Database – Restore From Snapshot

as seen on SQL Authority - Search for 'SQL Authority'
Snapshot database is one of the most interesting concepts that I have used at some places recently. Here is a quick definition of the subject from Book On Line: A Database Snapshot is a read-only, static view of a database (the source database). Multiple snapshots can exist on a source database and… >>> More
OTN ???? ?????? ???????

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Database ?? Database ??????? Database ?????????? Java WebLogic Server/????????·???? SOA/BPM/????? ???????/???? ID??/?????? ?????EPM/BI EPM/BI ??????? EPM/BI ???? OS/??? ???? ????? MySQL Database ?? ???? ?? ????????? ??? ?? ORACLE MASTER… >>> More