Search Results

Search found 1639 results on 66 pages for 'csv'.

Page 2/66 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

  • comparing csv files

    - by Surresh2121
    I want to write a shell script to compare two .csv files. First one contains filename,path the second .csv file contains filename,paht,target. Now, I want to compare the two .csv files and output the target name where the file from the first .csv exists in the second .csv file. Ex. a.csv build.xml,/home/build/NUOP/project1 eesX.java,/home/build/adm/acl b.csv build.xml,/home/build/NUOP/project1,M1 eesX.java,/home/build/adm/acl,M2 ddexse3.htm,/home/class/adm/33eFg I want the output to be something like this. M1 and M2 Please help Thanks,

    Read the article

  • Read random lines from huge CSV file in Python

    - by jbssm
    I have this quite big CSV file (15 Gb) and I need to read about 1 million random lines from it. As far as I can see - and implement - the CSV utility in Python only allows to iterate sequentially in the file. It's very memory consuming to read the all file into memory to use some random choosing and it's very time consuming to go trough all the file and discard some values and choose others, so, is there anyway to choose some random line from the CSV file and read only that line? I tried without success: import csv with open('linear_e_LAN2A_F_0_435keV.csv') as file: reader = csv.reader(file) print reader[someRandomInteger] A sample of the CSV file: 331.093,329.735 251.188,249.994 374.468,373.782 295.643,295.159 83.9058,0 380.709,116.221 352.238,351.891 183.809,182.615 257.277,201.302 61.4598,40.7106

    Read the article

  • Writing to CSV issue in Spyder

    - by 0003
    I am doing the Kaggle Titanic beginner contest. I generally work in Spyder IDE, but I came across a weird issue. The expected output is supposed to be 418 rows. When I run the script from terminal the output I get is 418 rows (as expected). When I run it in Spyder IDE the output is 408 rows not 418. When I re-run it in the current python process, it outputs the expected 418 rows. I posted a redacted portion of the code that has all of the relevant bits. Any ideas? import csv import numpy as np csvFile = open("/train.csv","ra") csvFile = csv.reader(csvFile) header = csvFile.next() testFile = open("/test.csv","ra") testFile = csv.reader(testFile) testHeader = testFile.next() writeFile = open("/gendermodelDebug.csv", "wb") writeFile = csv.writer(writeFile) count = 0 for row in testFile: if row[3] == 'male': do something to row writeFile.writerow(row) count += 1 elif row[3] == 'female': do something to row writeFile.writerow(row) count += 1 else: raise ValueError("Did not find a male or female in %s" % row)

    Read the article

  • php csv high load request

    - by msaif
    i have PHP serve and one csv file . i need to read csv file and send the data to the browser. if individual request = 10,000 or more (may be) then reading csv file from harddisk may be costly.how can i efficiently read csv file from PHP and send the data to the browser. there is no option to read data form relational db. only browser<-------------PHP(apache)<----------------CSV flow pattern.

    Read the article

  • printing only the last 8 entries in a .csv file

    - by user2341103
    I have an input csv file as below,I want to print only the most recent 8 entries..can anyone provide inputs on how to do this? INPUT:- trend.csv ['2013-06-25 20:01', '10'] ['2013-06-25 20:06', '9'] ['2013-06-25 20:06', '8'] ['2013-06-26 20:06', '7'] ['2013-06-26 20:06', '6'] ['2013-06-26 20:06', '5'] ['2013-06-26 20:06', '4'] ['2013-06-26 20:06', '3'] ['2013-06-26 20:06', '2'] ['2013-06-26 20:08', '1'] OUTPUT:- ['2013-06-25 20:06', '8'] ['2013-06-26 20:06', '7'] ['2013-06-26 20:06', '6'] ['2013-06-26 20:06', '5'] ['2013-06-26 20:06', '4'] ['2013-06-26 20:06', '3'] ['2013-06-26 20:06', '2'] ['2013-06-26 20:08', '1'] import csv #Now read the recent 8 entries and print cr = csv.reader(open("trend.csv","rb")) for row in cr: #print only the recent most 8 entries print row

    Read the article

  • Looking for app to work fluidly with CSV data in graph form

    - by Aszurom
    It often occurs to me that if I had a good tool for viewing CSV data in graphical format, and comparing two sets of numbers to each other, I could do a great deal of meaningful trend watching and data interpretation. For example, perfmon can output quite a lot of data about a server into a CSV file, but there's no good way to view it. A lot of scripts could/have been written that would populate CSV files. I could write these all day long. My problem is that I need a great viewer. I've seen quite a few things that will take a CSV file and after a lot of tweaking and user adjustment produce a static gif/png image. A static image doesn't do me a lot of good, because I have to look at it, then re-calibrate the parameters of the program, regenerate the image, repeat. That sucks. I could do this in Excel. Ideally, I would want a FLUID graph viewer. On the fly, I can adjust how much of my timeline I'm viewing. I could adjust the scaling so that one big spike doesn't make 99.9% of the data an unreadable line across the bottom of the X axis. Stuff like that. I should be able to say "show me CSV column 3 and column 5 as graphs. Show me the data scaled for 20 or 150 entries, and let me slide that window up and down the column of data. Auto scale to fit 95% of data within the Y axis and let crazy spikes go off the screen." Maybe I'm terribly spoiled by how you can drag, zoom, and slide data around on my iPad. I want to be able to view a spreadsheet of data with that fluidity and not have to guess at what sort of static snapshot I want to create from it. I don't want to have to make a study of how to tweak some data plotting program to let me import my file and do what I could just do in Excel. I want to scale, zoom, and transform my graph on the fly and then export a snapshot of it once I have it the way I want it. Is there anything out there that fills this need? I'll take linux, osx, win32 or even iOS suggestions.

    Read the article

  • Ruby on Rails export to csv - maintain mysql select statement order

    - by zekial
    Exporting some data from mysql to a csv file using FasterCSV. I'd like the columns in the outputted CSV to be in the same order as the select statement in my query. Example: rows = Data.find( :all, :select=>'name, age, height, weight' ) headers = rows[0].attributes.keys FasterCSV.generate do |csv| csv << headers rows.each do |r| csv << r.attributes.values end end CSV Output: height,weight,name,age 74,212,bob,23 70,201,fred,24 . . . I want the CSV columns in the same order as my select statement. Obviously the attributes method is not going to work. Any ideas on the best way to ensure that the columns in my csv file will be in the same order as the select statement? Got a lot of data and performance is an issue. The select statement is not static. I realize I could loop through column names within the rows.each loop but it seems kinda dirty.

    Read the article

  • Autodetect Presence of CSV Headers in a File

    - by banzaimonkey
    Short question: How do I automatically detect whether a CSV file has headers in the first row? Details: I've written a small CSV parsing engine that places the data into an object that I can access as (approximately) an in-memory database. The original code was written to parse third-party CSV with a predictable format, but I'd like to be able to use this code more generally. I'm trying to figure out a reliable way to automatically detect the presence of CSV headers, so the script can decide whether to use the first row of the CSV file as keys / column names or start parsing data immediately. Since all I need is a boolean test, I could easily specify an argument after inspecting the CSV file myself, but I'd rather not have to (go go automation). I imagine I'd have to parse the first 3 to ? rows of the CSV file and look for a pattern of some sort to compare against the headers. I'm having nightmares of three particularly bad cases in which: The headers include numeric data for some reason The first few rows (or large portions of the CSV) are null There headers and data look too similar to tell them apart If I can get a "best guess" and have the parser fail with an error or spit out a warning if it can't decide, that's OK. If this is something that's going to be tremendously expensive in terms of time or computation (and take more time than it's supposed to save me) I'll happily scrap the idea and go back to working on "important things". I'm working with PHP, but this strikes me as more of an algorithmic / computational question than something that's implementation-specific. If there's a simple algorithm I can use, great. If you can point me to some relevant theory / discussion, that'd be great, too. If there's a giant library that does natural language processing or 300 different kinds of parsing, I'm not interested.

    Read the article

  • parsing python to csv

    - by user185955
    I'm trying to download some game stats to do some analysis, only problem is each season the data their isn't 100% consistent. I grab the json file from the site, then wish to save it to a csv with the first line in the csv containing the heading for that column, so the heading would be essentially the key from the python data type. #!/usr/bin/env python import requests import json import csv base_url = 'http://www.afl.com.au/api/cfs/afl/' token_url = base_url + 'WMCTok' player_url = base_url + 'matchItems/round' def printPretty(data): print(json.dumps(data, sort_keys=True, indent=2, separators=(',', ': '))) session = requests.Session() # session makes it simple to use the token across the requests token = session.post(token_url).json()['token'] # get the token session.headers.update({'X-media-mis-token': token}) # set the token Season = 2014 Roundno = 4 if Roundno<10: strRoundno = '0'+str(Roundno) else: strRoundno = str(Roundno) # get some data (could easily be a for loop, might want to put in a delay using Sleep so that you don't get IP blocked) data = session.get(player_url + '/CD_R'+str(Season)+'014'+strRoundno) # print everything printPretty(data.json()) with open('stats_game_test.csv', 'w', newline='') as csvfile: spamwriter = csv.writer(csvfile, delimiter="'",quotechar='|', quoting=csv.QUOTE_ALL) for profile in data.json()['items']: spamwriter.writerow(['%s' %(profile)]) #for key in data.json().keys(): # print("key: %s , value: %s" % (key, data.json()[key])) The above code grabs the json and writes it to a csv, but it puts the key in each individual cell next to the value (eg 'venueId': 'CD_V190'), the key needs to be just across the first row as a heading. It gives me a csv file with data in the cells like this Column A B 'tempInCelsius': 17.0 'totalScore': 32 'tempInCelsius': 16.0 'totalScore': 28 What I want is the data like this tempInCelsius totalScore 17 32 16 28 As I mentioned up the top, the data isn't always consistent so if I define what fields to grab with spamwriter.writerow([profile['tempInCelsius'], profile['totalScore']]) then it will error out on certain data grabs. This is why I'm now trying the above method so it just grabs everything regardless of what data is there.

    Read the article

  • Outlook 2011 Contact Import from CSV with Notes containing new lines / cr / lf

    - by Paul Hargreaves
    I'm trying to import several thousand contacts into Outlook 2011 for Mac. Everything is working well except the Notes field as I cannot figure out how to get new lines / carriage returns into it. There is no documentation for the exact format that Outlook supports. After searching the web and experimenting I have tried: Creating a single contact in Outlook with Notes containing several lines of text. I then export the contact to a csv, deleting the contact in Outlook, then re-import. All lines in Notes merge together :-/ Following tips I found such as containing new lines around quotes. e.g. http://creativyst.com/Doc/Articles/CSV/CSV01.htm (search for line-break) Switching the CSV format from DOS to Unix, experimenting using manually injected ctrl-characters such as ^M etc. I would include an example export/import but unfortunately the the new breaks included do not work well in a SU code block.

    Read the article

  • mongoexport csv output array values

    - by 9point6
    I'm using mongoexport to export some collections into CSV files, however when I try to target fields which are members of an array I cannot get it to export correctly. command I'm using: mongoexport -d db -c collection -fieldFile fields.txt --csv > out.csv and the contents of fields.txt is similar to id name address[0].line1 address[0].line2 address[0].city address[0].country address[0].postcode where the BSON data would be: { "id": 1, "name": "example", "address": [ { "line1": "flat 123", "line2": "123 Fake St.", "city": "London", "country": "England", "postcode": "N1 1AA" } ] } what is the correct syntax for exporting the contents of an array?

    Read the article

  • Read data from separate CSV file in OpenOffice Calc

    - by Thomi
    Hi, I have a very large CSV file (many thousands of rows) that I want to work with. Due to the size of the file, I don't want to import it into openoffice. Rather, I want to create a spreadsheet that contains formulas & graphs that read from this (or any other) CSV file I point it it. Ideally the spreadsheet will ask me what CSV file I want to use, allowing me to change the data source dynamically. Any ideas?

    Read the article

  • Getting Excel to handle CRLF's correctly in CSV

    - by Ben Fulton
    I am creating CSV files to be opened in Excel. The rows are separated by CRLF and that's fine, but some of the input data contains CRLF data in it as well. Per the usual standards, I surround them with quotes, but Excel doesn't seem to recognize the CR character and puts a little box with a question mark in it instead. I can strip the CR's out of the CSV file, but it seems like an unnecessary step. Is there an easy way to get Excel to recognize a CRLF inside a row of a CSV file?

    Read the article

  • Exported csv file is not in right lining with pgadmin

    - by user938363
    We exported a pg 9.3 table to csv file in pgadmin. The problem is that from about 10th line, the lining of the columns were messed up and did not line up correctly with the columns above. We tried a few times and every output has the same problem. We follow the instruction on http://www.question-defense.com/2010/10/15/how-to-export-from-pgadmin-export-pgadmin-data-to-csv for export. The only difference that UTF8 is selected instead of localcharset. What's the right way to export csv in pg?

    Read the article

  • Custom CSV (.csv) filter for OpenOffice.org or LibreOffice?

    - by anon
    Is it possible to create a some kind of 'custom CSV filter' for OpenOffice.org or LibreOffice spreadsheet program. What I need is to have the program to use predefined CSV settings for loading and saving when I open, let's say file named 'somefile.myext'. Also I would need the loaded data to be placed in a prestyled spreadsheet. In this particular case, I would need the CSV settings to have tab as a field delimiter and no text delimiter at all. Prestyled spreadsheet would contain Blue gray coloring for every odd row (achieved with conditional formatting formula), some font styling and probably some column width definitions.

    Read the article

  • Can the csv format be defined by a regex?

    - by Spencer Rathbun
    A colleague and I have recently argued over whether a pure regex is capable of fully encapsulating the csv format, such that it is capable of parsing all files with any given escape char, quote char, and separator char. The regex need not be capable of changing these chars after creation, but it must not fail on any other edge case. I have argued that this is impossible for just a tokenizer. The only regex that might be able to do this is a very complex PCRE style that moves beyond just tokenizing. I am looking for something along the lines of: ... the csv format is a context free grammar and as such, it is impossible to parse with regex alone ... Or am I wrong? Is it possible to parse csv with just a POSIX regex? For example, if both the escape char and the quote char are ", then these two lines are valid csv: """this is a test.""","" "and he said,""What will be, will be."", to which I replied, ""Surely not!""","moving on to the next field here..."

    Read the article

  • Rails CSV import, adding to a related table

    - by Jack
    Hi, I have a csv importing system on my app (used locally only) which parses the csv file line by line and adds the data to the database table. This is based on a tutorial here. require 'csv' def csv_import @parsed_file=CSV::Reader.parse(params[:dump][:file]) n = 0 @parsed_file.each_with_index do |row, i| next if i == 0 #ignore the first row course = Course.new course.title = row[0] course.unit_code = row[1] course.course_type = row[2] course.value = row[3] course.pass_mark = row[4] if course.save n = n+1 GC.start if n%50==0 end flash.now[:message] = "CSV Import Successful, #{n} new courses added to the database." end redirect_to(courses_url) end This is all in the courses controller and works fine. There is a relationship that courses HABTM years and years HABTM courses. In the csv file (effectively in row[5] to row[8]) are the year_id s. Is there a way that I can add this within the method above. I am confused as to how to loop over the 4 items and add them to the courses_years table. Thank you Jack

    Read the article

  • MySQL import in phpmyadmin (CSV) chokes on quotes

    - by Andrew Swift
    I am trying to import a .csv file into a MySQL table via phpMyAdmin. The .csv file is separated by pipes, formated like this: data|d'ata|d'a"ta|dat"a| data|"da"ta|data|da't'a| dat'a|data|da"ta"|da'ta| The data contains quotes. I have no control over the format in which I recieve the data -- it is generated by a third party. The problem comes when there is a | followed by a double quote. I always get an "invalid field count in CSV input on line N" error. I am uploading the file from the import page, using Latin1, CSV, terminated by |, separated by ". I would like to just change the "enclosed by" character, but I keep getting "Invalid parameter for CSV import: Fields enclosed by". I have tried various characters with no success. How can I tell MySQL to accept this format in phpMyAdmin? Setting up these tables is the first step in writing a program that will use uploaded gzipped .csv files to maintain the catalog of an e-commerce site.

    Read the article

  • Tool to convert from XML to CSV and back to XML without losing information?

    - by Lanaru
    I would like to convert an XML file into CSV, and then be able to convert that CSV back into the original XML file. Is there a tool out there that makes this possible? The reason I want to do this is that I want to generate a ton of data to be stored in an xml file that my program will parse. I can easily generate a lot of data with a CSV file through the use of Excel macros, so that's why I'm looking for a tool to convert between these two formats. Any alternative suggestions are greatly appreciated.

    Read the article

  • Set CSV import default to UTF-8 in Calc

    - by picca
    Every time I open a CSV (comma separated values) document in OpenOffice.org Calc I get a dialog with CSV preferences. The current default character set is "Eastern Europe (ISO-8859-2)". I would like "UTF-8" to be selected by default instead.

    Read the article

  • oocalc csv defaults to utf-8

    - by picca
    Everytime I open csv (comma separated values) document in oocalc (Open Office) i get dialog with csv preferences. And everytime character set is "Eastern Europe (ISO-8859-2)". I would like "UTF-8" to be defaultly selected.

    Read the article

  • How to bulk insert a CSV file into SQLite C#

    - by Lirik
    I have seen similar questions (1, 2), but none of them discuss how to insert CSV files into SQLite. About the only thing I could think of doing is to use a CSVDataAdapter and fill the SQLiteDataSet, then use the SQLiteDataSet to update the tables in the database: The only DataAdapter for CSV files I found is not actually available: CSVDataAdapter CSVda = new CSVDataAdapter(@"c:\MyFile.csv"); CSVda.HasHeaderRow = true; DataSet ds = new DataSet(); // <-- Use an SQLiteDataSet instead CSVda.Fill(ds); To write to a CSV file: CSVDataAdapter CSVda = new CSVDataAdapter(@"c:\MyFile.csv"); bool InclHeader = true; CSVda.Update(MyDataSet,"MyTable",InclHeader); I found the above code @ http://devintelligence.com/2005/02/dataadapter-for-csv-files/ The CSVDataAdapter was supposed to come with OpenNetCF's SDF, but it doesn't seem to be available anymore. Does anybody know where I can get a CSVDataAdapter? Perhaps somebody knows the much simpler thing: how to do bulk inserts of CSV files into SQLite... your help would be greatly appreciated!

    Read the article

  • How to bulk insert a CSV file into SQLite

    - by Lirik
    I have seen similar questions (1, 2), but none of them discuss how to insert CSV files into SQLite. About the only thing I could think of doing is to use a CSVDataAdapter and fill the SQLiteDataSet, then use the SQLiteDataSet to update the tables in the database: The only DataAdapter for CSV files I found is not actually available: CSVDataAdapter CSVda = new CSVDataAdapter(@"c:\MyFile.csv"); CSVda.HasHeaderRow = true; DataSet ds = new DataSet(); // <-- Use an SQLiteDataSet instead CSVda.Fill(ds); To write to a CSV file: CSVDataAdapter CSVda = new CSVDataAdapter(@"c:\MyFile.csv"); bool InclHeader = true; CSVda.Update(MyDataSet,"MyTable",InclHeader); I found the above code @ http://devintelligence.com/2005/02/dataadapter-for-csv-files/ The CSVDataAdapter was supposed to come with OpenNetCF's SDF, but it doesn't seem to be available anymore. Does anybody know where I can get a CSVDataAdapter? Perhaps somebody knows the much simpler thing: how to do bulk inserts of CSV files into SQLite... your help would be greatly appreciated!

    Read the article

  • Generating a .CSV with Several Columns - Use a Dictionary?

    - by Qanthelas
    I am writing a script that looks through my inventory, compares it with a master list of all possible inventory items, and tells me what items I am missing. My goal is a .csv file where the first column contains a unique key integer and then the remaining several columns would have data related to that key. For example, a three row snippet of my end-goal .csv file might look like this: 100001,apple,fruit,medium,12,red 100002,carrot,vegetable,medium,10,orange 100005,radish,vegetable,small,10,red The data for this is being drawn from a couple sources. 1st, a query to an API server gives me a list of keys for items that are in inventory. 2nd, I read in a .csv file into a dict that matches keys with item name for all possible keys. A snippet of the first 5 rows of this .csv file might look like this: 100001,apple 100002,carrot 100003,pear 100004,banana 100005,radish Note how any key in my list of inventory will be found in this two column .csv file that gives all keys and their corresponding item name and this list minus my inventory on hand yields what I'm looking for (which is the inventory I need to get). So far I can get a .csv file that contains just the keys and item names for the items that I don't have in inventory. Give a list of inventory on hand like this: 100003,100004 A snippet of my resulting .csv file looks like this: 100001,apple 100002,carrot 100005,radish This means that I have pear and banana in inventory (so they are not in this .csv file.) To get this I have a function to get an item name when given an item id that looks like this: def getNames(id_to_name, ids): return [id_to_name[id] for id in ids] Then a function which gives a list of keys as integers from my inventory server API call that returns a list and I've run this function like this: invlist = ServerApiCallFunction(AppropriateInfo) A third function takes this invlist as its input and returns a dict of keys (the item id) and names for the items I don't have. It also writes the information of this dict to a .csv file. I am using the set1 - set2 method to do this. It looks like this: def InventoryNumbers(inventory): with open(csvfile,'w') as c: c.write('InvName' + ',InvID' + '\n') missinginvnames = [] with open("KeyAndItemNameTwoColumns.csv","rb") as fp: reader = csv.reader(fp, skipinitialspace=True) fp.readline() # skip header invidsandnames = {int(id): str.upper(name) for id, name in reader} invids = set(invidsandnames.keys()) invnames = set(invidsandnames.values()) invonhandset = set(inventory) missinginvidsset = invids - invonhandset missinginvids = list(missinginvidsset) missinginvnames = getNames(invidsandnames, missinginvids) missinginvnameswithids = dict(zip(missinginvnames, missinginvids)) print missinginvnameswithids with open(csvfile,'a') as c: for invname, invid in missinginvnameswithids.iteritems(): c.write(invname + ',' + str(invid) + '\n') return missinginvnameswithids Which I then call like this: InventoryNumbers(invlist) With that explanation, now on to my question here. I want to expand the data in this output .csv file by adding in additional columns. The data for this would be drawn from another .csv file, a snippet of which would look like this: 100001,fruit,medium,12,red 100002,vegetable,medium,10,orange 100003,fruit,medium,14,green 100004,fruit,medium,12,yellow 100005,vegetable,small,10,red Note how this does not contain the item name (so I have to pull that from a different .csv file that just has the two columns of key and item name) but it does use the same keys. I am looking for a way to bring in this extra information so that my final .csv file will not just tell me the keys (which are item ids) and item names for the items I don't have in stock but it will also have columns for type, size, number, and color. One option I've looked at is the defaultdict piece from collections, but I'm not sure if this is the best way to go about what I want to do. If I did use this method I'm not sure exactly how I'd call it to achieve my desired result. If some other method would be easier I'm certainly willing to try that, too. How can I take my dict of keys and corresponding item names for items that I don't have in inventory and add to it this extra information in such a way that I could output it all to a .csv file? EDIT: As I typed this up it occurred to me that I might make things easier on myself by creating a new single .csv file that would have date in the form key,item name,type,size,number,color (basically just copying in the column for item name into the .csv that already has the other information for each key.) This way I would only need to draw from one .csv file rather than from two. Even if I did this, though, how would I go about making my desired .csv file based on only those keys for items not in inventory?

    Read the article

  • Import CSV into Org-mode properties

    - by lecodesportif
    I would like to import a CSV into Org-mode. Others have already asked about importing CSV to Org-mode tables. That's not what I am trying to do. I need to import CSV to Org-mode properties. For example, a CSV like this: Name,Tel,Mobile,Fax John,11111,22222,33333 should become: :PROPERTIES: :Name: John :Tel: 11111 :Mobile: 22222 :Fax: 33333 :END: Do you happen to know a painless way to do it?

    Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >