How to insert several thousand columns into sqlite3?

Posted by user291071 on Stack Overflow See other posts from Stack Overflow or by user291071
Published on 2010-04-24T05:41:13Z Indexed on 2010/04/24 5:43 UTC
Read the original article Hit count: 209

Filed under:
|
|

Similar to my last question, but I ran into problem lets say I have a simple dictionary like below but its Big, when I try inserting a big dictionary using the methods below I get operational error for the c.execute(schema) for too many columns so what should be my alternate method to populate an sql databases columns? Using the alter table command and add each one individually?

import sqlite3
con = sqlite3.connect('simple.db')
c = con.cursor()

dic = {
    'x1':{'y1':1.0,'y2':0.0},
    'x2':{'y1':0.0,'y2':2.0,'joe bla':1.5},
    'x3':{'y2':2.0,'y3 45 etc':1.5}
    }

# 1. Find the unique column names.
columns = set()
for _, cols in dic.items():
    for key, _ in cols.items():
       columns.add(key)

# 2. Create the schema.
col_defs = [
    # Start with the column for our key name
    '"row_name" VARCHAR(2) NOT NULL PRIMARY KEY'
    ]
for column in columns:
    col_defs.append('"%s" REAL NULL' % column)
schema = "CREATE TABLE simple (%s);" % ",".join(col_defs)
c.execute(schema)

# 3. Loop through each row
for row_name, cols in dic.items():

    # Compile the data we have for this row.
    col_names = cols.keys()
    col_values = [str(val) for val in cols.values()]

    # Insert it.
    sql = 'INSERT INTO simple ("row_name", "%s") VALUES ("%s", "%s");' % (
        '","'.join(col_names),
        row_name,
        '","'.join(col_values)
        )

© Stack Overflow or respective owner

Related posts about sqlite3

Related posts about python