Search Results

Search found 4769 results on 191 pages for 'places sqlite'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 10/191 | < Previous Page | 6 7 8 9 10 11 12 13 14 15 16 17  | Next Page >

  • SQLite issues, escaping certain characters...

    - by CODe
    I'm working on my first database application. It is a WinForms application written in C# using a SQLite database. I've come across some problems, when a apostrophe is used, my SQLite query fails. Here is the structure of my queries. string SQL = "UPDATE SUBCONTRACTOR SET JobSite = NULL WHERE JobSite = '" + jobSite + "'"; For instance, if an apostrophe is used in the jobSite var, it offsets the other apostrophes in the command, and fails. So my questions are: 1. How do I escape characters like the apostrophe and semicolon in the above query example? 2. What characters do I need to escape? I know I should escape the apostrophe, what else is dangerous? Thanks for your help!

    Read the article

  • SQLite Queries for dates

    - by user2909616
    I have a SQLite data base which I am pulling data for a specific set of dates (lets say 01-01-2011 to 01-01-2011). What is the best way to implement this query into SQL. Ideally I would like the following line to run: SELECT * FROM database where start_date < date_stamp and end_date date_stamp This obviously does not work when I store the dates as strings. My solution (which I think is messy and I am hoping for another one) is to convert the dates into integers in the following format: YYYYMMDD Which makes the above line able to run (theoretically). IS there a better method? Using python sqlite3 Would the answer be any different if I were using SQL not SQLite

    Read the article

  • Android ListView with SQLite

    - by soclose
    Hi I'd like to refresh the Listview items. These items are populated from SQLite database. My code is below public class Weeve extends Activity { private String[] lv_arr; protected ListView CView; private DBAdapter mDbHelper; public void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); mDbHelper = new DBAdapter(this); mDbHelper.open(); Cursor c = mDbHelper.getAll(); if (c.getCount() > 0) {if (c.moveToFirst()) { ArrayList strings = new ArrayList(); do { String mC = c.getString(0); strings.add(mC); } while (c.moveToNext()); lv_arr = (String[]) strings.toArray(new String[strings.size()]); } } else Toast.makeText(this, "No more Records", Toast.LENGTH_SHORT).show(); c.close(); ListView CView = new ListView(this); CView.setAdapter(new ArrayAdapter<String>(this, android.R.layout.simple_list_item_1, lv_arr)); setContentView(CView);}} I'd like to make refreshing this list view after adding, updating or deleting SQLite table. These operations are called by content or option menu. I tried to create these code into a separated function and call it after every operation. But can't. I think setContentView(CView) statement. I also tried to use SimpleCursorAdapter like notepad sample from Android.com. I got Thread error. Help me.

    Read the article

  • How to bulk insert a CSV file into SQLite C#

    - by Lirik
    I have seen similar questions (1, 2), but none of them discuss how to insert CSV files into SQLite. About the only thing I could think of doing is to use a CSVDataAdapter and fill the SQLiteDataSet, then use the SQLiteDataSet to update the tables in the database: The only DataAdapter for CSV files I found is not actually available: CSVDataAdapter CSVda = new CSVDataAdapter(@"c:\MyFile.csv"); CSVda.HasHeaderRow = true; DataSet ds = new DataSet(); // <-- Use an SQLiteDataSet instead CSVda.Fill(ds); To write to a CSV file: CSVDataAdapter CSVda = new CSVDataAdapter(@"c:\MyFile.csv"); bool InclHeader = true; CSVda.Update(MyDataSet,"MyTable",InclHeader); I found the above code @ http://devintelligence.com/2005/02/dataadapter-for-csv-files/ The CSVDataAdapter was supposed to come with OpenNetCF's SDF, but it doesn't seem to be available anymore. Does anybody know where I can get a CSVDataAdapter? Perhaps somebody knows the much simpler thing: how to do bulk inserts of CSV files into SQLite... your help would be greatly appreciated!

    Read the article

  • How to bulk insert a CSV file into SQLite

    - by Lirik
    I have seen similar questions (1, 2), but none of them discuss how to insert CSV files into SQLite. About the only thing I could think of doing is to use a CSVDataAdapter and fill the SQLiteDataSet, then use the SQLiteDataSet to update the tables in the database: The only DataAdapter for CSV files I found is not actually available: CSVDataAdapter CSVda = new CSVDataAdapter(@"c:\MyFile.csv"); CSVda.HasHeaderRow = true; DataSet ds = new DataSet(); // <-- Use an SQLiteDataSet instead CSVda.Fill(ds); To write to a CSV file: CSVDataAdapter CSVda = new CSVDataAdapter(@"c:\MyFile.csv"); bool InclHeader = true; CSVda.Update(MyDataSet,"MyTable",InclHeader); I found the above code @ http://devintelligence.com/2005/02/dataadapter-for-csv-files/ The CSVDataAdapter was supposed to come with OpenNetCF's SDF, but it doesn't seem to be available anymore. Does anybody know where I can get a CSVDataAdapter? Perhaps somebody knows the much simpler thing: how to do bulk inserts of CSV files into SQLite... your help would be greatly appreciated!

    Read the article

  • SQLite vs Firebird

    - by rwallace
    The scenario I'm looking at is "This program uses Postgres. Oh, you want to just use it single-user for the moment, and put off having to deal with installing a database server? Okay, in the meantime you can use it with the embedded single-user database." The question is then which embedded database is best. As I understand it, the two main contenders are SQLite and Firebird; so which is better? Criteria: Full SQL support, or as close as reasonably possible. Full text search. Easy to call from C# Locks, or allows you to lock, the database file to make sure nobody tries to run it multiuser and ends up six months down the road with intermittent data corruption in all their backups. Last but far from least, reliability. As I understand it, the disadvantages of SQLite are, No right outer join. Workaround: use left outer join instead. Not much integrity checking. Workaround: be really careful in the application code. No decimal numbers. Workaround: lots of aspirin. None of the above are showstoppers. Are there any others I'm missing? (I know it doesn't support some administrative and code-within-database SQL features, that aren't relevant for this kind of use case.) I don't know anything much about Firebird. What are its disadvantages?

    Read the article

  • display sqlite datatable in a jtable

    - by tuxou
    Hi I'm trying to display an sqlite data table in a jtable but i have an error " sqlite is type forward only" how could I display it in a jtable try { long start = System.currentTimeMillis(); Statement state = ConnectionBd.getInstance().createStatement( ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY ); ResultSet res = state.executeQuery("SELECT * FROM data"); ResultSetMetaData meta = res.getMetaData(); Object[] column = new Object[meta.getColumnCount()]; for(int i = 1 ; i <= meta.getColumnCount(); i++){ column[i-1] = meta.getColumnName(i); } res.last(); int rowCount = res.getRow(); Object[][] data = new Object[res.getRow()][meta.getColumnCount()]; res.beforeFirst(); int j = 1; while(res.next()){ for(int i = 1 ; i <= meta.getColumnCount(); i++) data[j-1][i-1] = res.getObject(i); j++; } res.close(); state.close(); long totalTime = System.currentTimeMillis() - start; result.removeAll(); result.add(new JScrollPane(new JTable(data, column)), BorderLayout.CENTER); result.add(new JLabel("execute in " + totalTime + " ms and has " + rowCount + " ligne(s)"), BorderLayout.SOUTH); result.revalidate(); } catch (SQLException e) { result.removeAll(); result.add(new JScrollPane(new JTable()), BorderLayout.CENTER); result.revalidate(); JOptionPane.showMessageDialog(null, e.getMessage(), "ERREUR ! ", JOptionPane.ERROR_MESSAGE); } thank you

    Read the article

  • Image Wells, Core Data, and Sqlite files.

    - by Sway
    I've got a mac application that I've developed. I use it to create sqlite files that are bundled with my iphone app. The mac app uses Core Data and bindings and is working fine except for one "weird" issue. I use an NSImageView (or Image Well) to allow me to drag and drop jpg files. This is bound through to an optional binary attribute in my model class. For some reason when I drag and drop a 4k jpg file it onto the image well and save the sqlite file. The data saved to the binary column is over 15 times larger than it should be. Whereas if I use an application like SQLiteManager and add the image into the row in the database. The binary data is the correct (expected size). File 4k jpg Actual size: 2371. Persisted via Core Data size: 35810. Can anyone give me a suggestion as to why this might be happening? Do I need to set some setting in Interface Builder or write some custom code?

    Read the article

  • Looking for advice on importing large dataset in sqlite and Cocoa/Objective-C

    - by jluckyiv
    I have a fairly large hierarchical dataset I'm importing. The total size of the database after import is about 270MB in sqlite. My current method works, but I know I'm hogging memory as I do it. For instance, if I run with Zombies, my system freezes up (although it will execute just fine if I don't use that Instrument). I was hoping for some algorithm advice. I have three hierarchical tables comprising about 400,000 records. The highest level has about 30 records, the next has about 20,000, the last has the balance. Right now, I'm using nested for loops to import. I know I'm creating an unreasonably large object graph, but I'm also looking to serialize to JSON or XML because I want to break up the records into downloadable chunks for the end user to import a la carte. I have the code written to do the serialization, but I'm wondering if I can serialize the object graph if I only have pieces in memory. Here's pseudocode showing the basic process for sqlite import. I left out the unnecessary detail. [database open]; [database beginTransaction]; NSArray *firstLevels = [[FirstLevel fetchFromURL:url retain]; for (FirstLevel *firstLevel in firstLevels) { [firstLevel save]; int id1 = [firstLevel primaryKey]; NSArray *secondLevels = [[SecondLevel fetchFromURL:url] retain]; for (SecondLevel *secondLevel in secondLevels) { [secondLevel saveWithForeignKey:id1]; int id2 = [secondLevel primaryKey]; NSArray *thirdLevels = [[ThirdLevel fetchFromURL:url] retain]; for (ThirdLevel *thirdLevel in thirdLevels) { [thirdLevel saveWithForeignKey:id2]; } [database commit]; [database beginTransaction]; [thirdLevels release]; } [secondLevels release]; } [database commit]; [database release]; [firstLevels release];

    Read the article

  • Problems when I try to see databases in SQLite

    - by Sabau Andreea
    I created in code a database and two tables: static final String dbName="graficeCirculatie"; static final String ruteTable="Rute"; static final String colRuteId="RutaID"; static final String colRuta="Ruta"; static final String statiaTable="Statia"; static final String colStatiaID="StatiaID"; static final String colIdRuta="IdRuta"; static final String colStatia="Statia"; public DatabaseHelper(Context context) { super(context, dbName, null,33); } public void onCreate(SQLiteDatabase db) { // TODO Auto-generated method stub db.execSQL("CREATE TABLE " + statiaTable + " (" + colStatiaID + " INTEGER PRIMARY KEY , " + colIdRuta + " INTEGER, " + colStatia + " TEXT)"); db.execSQL("CREATE TABLE " + ruteTable + "(" + colRuteId + " INTEGER PRIMARY KEY AUTOINCREMENT, " + colRuta + " TEXT);"); InsertDepts(db); } void InsertDepts(SQLiteDatabase db) { ContentValues cv = new ContentValues(); cv.put(colRuteId, 1); cv.put(colRuta, "Expres8"); db.insert(ruteTable, colRuteId, cv); cv.put(colRuteId, 2); cv.put(colRuta, "Expres2"); db.insert(ruteTable, colRuteId, cv); cv.put(colRuteId, 3); cv.put(colRuta, "Expres3"); db.insert(ruteTable, colRuteId, cv); } Now I want to see tables inputs from command line. I try in this way: C:\Program Files\Android\android-sdk\tools sqlite3 SQLite version 3.7.4 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite sqlite3 graficeCirculatie ... select * from ruteTable; And I got an error: Error: near "squlite3": syntax error. Can someone help me?

    Read the article

  • SQLite transaction doesn't work as expected

    - by troll
    I prepared 2 files, "1.php" and "2.php". "1.php" is like this. <?php $dbh = new PDO('sqlite:test1'); $dbh->beginTransaction(); print "aaa<br>"; sleep(55); $dbh->commit(); print "bbb"; ?> and "2.php" is like this. <?php $dbh = new PDO('sqlite:test1'); $dbh->beginTransaction(); print "ccc<br>"; $dbh->commit(); print "ddd"; ?> and I excute "1.php". It starts a transaction and waits 55 seconds. So when I immediately excute "2.php", my expectation is this: "1.php" is getting transaction and "1" holds a database lock "2" can not begin a transaction "2" can not get database lock so "2" have to wait 55 seconds BUT, but the test went another way. When I excute "2",then "2" immediately returned it's result "2" did not wait so I have to think that "1" could not get transaction, or could not get database lock. Can anyone help?

    Read the article

  • SQLite Databases and Grid Hosting

    - by jocull
    I'm considering moving my site from a GoDaddy shared hosting account to a Media Temple grid hosting account in anticipation of traffic. However, I first have some concerns with the grid hosting setup. My site stores a large personal set of data on a per-user basis (possibly 3-4MB per user). At this rate I was worried about blowing over a 1GB MySQL limit in no time. To deal with this I created distributed SQLite databases per user to store large data objects. It's worked wonderfully so far. SQLite is super fast and simple. I know that reading from and writing to files is different in a Grid Hosting environment. I need to know if this setup is going to cause serious problems. These databases are not (and will not be) highly trafficked. They are personal to the user and will only be touched maybe two locations at the same time (one updating the data hourly at the most, and one or more reading on demand). I'd like to keep this setup as getting additional space (beyond 4GB) on a MySQL database seems to be a real trouble point. Will Grid Hosting cause me serious problems? Thanks.

    Read the article

  • Chinese records of sqlite databse are in not accessable in iphone app

    - by Mas
    Hi! I'm creating an iphone app. In which I'm reading data from the sqlite db and presenting it in the tableview control. Problem is that the data is in chinese language. Due to unknown reason, when I read/fetch record from the sqlite table, some records are presented and other are missed by objc. Even objc reads some columns of the missing values, But returns the nil results of the some columns. In reality these columns are not empty. I tried many solutions but didn't find any suitable. Same database is working perfectly in the android version of the app. Any one can help here is the code of reading the chinese records from table Quote *q = [[Quote alloc] init]; q.catId = sqlite3_column_int(statement, 0); q.subCatId = sqlite3_column_int(statement, 1); q.quote = [NSString stringWithUTF8String:(char *)sqlite3_column_text(statement, 2)]; and here is the some records which iphone misses ??????·??? ??????,????????! ??????,??????? Thanks

    Read the article

  • values not equal in sqlite and json array in android

    - by Venkat
    I am trying to compare the value in sqlite table and id of the webservice what i have done so far is if(data_exist!=bookProduct.length()){ Log.i("in update","m here"); Cursor cursors = getRawEvents("select id from bcuk_book"); try{ for (int i = 0; i < bookProduct.length(); i++) { JSONObject c = bookProduct.getJSONObject(i); String Bid = c.getString(TAG_ID); ArrayList<String> mapId = new ArrayList<String>(); while(cursors.moveToNext()) { Log.e("cursors",cursors.getString(0)); Log.i(Bid,Bid); if(cursors.getString(0)!=c.getString(TAG_ID)){ Log.e("fas",Bid); } } mapId.add(TAG_ID); Log.e(Bid,Bid); } } My issue is i am getting same values in logs.. if(cursors.getString(0)!=c.getString(TAG_ID)){ this condition says if they are not equal then print the log..But the issue is i am entering into that block even i am getting same values from sqlite and TAG_ID i.e from json webservice..How to solve this.Where i done wrong?

    Read the article

  • C# sqlite syntax in ASP.NET?

    - by acidzombie24
    -edit- This is no longer relevant and the question doesnt make sense to me anymore. I think i wanted to know how to create tables or know if the syntax is the same from winform to ASP.NET I am very use to sqlite http://sqlite.phxsoftware.com/ and would like to create a DB in a similar style. How do i do this? it doesnt need to be the same, just similar enough for me to enjoy. An example of the syntax. connection = new SQLiteConnection("Data Source=" + sz +";Version=3"); command = new SQLiteCommand(connection); connection.Open(); command.CommandText = "CREATE TABLE if not exists miscInfo(key TEXT, " + "value TEXT, UNIQUE (key));"; command.ExecuteNonQuery(); The @name is a symbol and command.Parameters.Add("@userDesc", DbType.String).Value = d.userDesc; replaces the symbol with escaped values/texts/blob command.CommandText = "INSERT INTO discData(rootfolderID, time, volumeName, discLabel, " + "userTitle, userDesc) "+ "VALUES(@rootfolderID, @time, @volumeName, @discLabel, " + "@userTitle, @userDesc); " + "SELECT last_insert_rowid() AS RecordID;"; command.Parameters.Add("@rootfolderID", DbType.Int64).Value = d.rootfolderID; command.Parameters.Add("@time", DbType.Int64).Value = d.time; command.Parameters.Add("@volumeName", DbType.String).Value = d.volumeName; command.Parameters.Add("@discLabel", DbType.String).Value = d.discLabel; command.Parameters.Add("@userTitle", DbType.String).Value = d.userTitle; command.Parameters.Add("@userDesc", DbType.String).Value = d.userDesc; d.discID = (long)command.ExecuteScalar();

    Read the article

  • How to create an Entity Framework model from an existing SQLite database in Visual Studio 2008?

    - by splattne
    I've installed the System.Data.SQLite ADO.NET Provider from http://sqlite.phxsoftware.com/. I can connect to the database from within Visual Studio, I can open table schemas, views etc. I'd like to use an existing SQLite database to create an Entity Framework model in Visual Studio 2008. When I try to create a new ADO.NET Entity Data Model (.edmx) file using the wizard, the existing SQLite connection is not in the list though. Also, it's not possible to create a SQLite connection because there is no provider for SQLite. It only lists SQL Server, SQL Server file and SQL Server Compact 3.5. Any idea how to solve this problem?

    Read the article

  • How to make a database service in Netbeans 6.5 to connect to SQLite databases?

    - by farzad
    I use Netbeans IDE (6.5) and I have a SQLite 2.x database. I installed a JDBC SQLite driver from zentus.com and added a new driver in Nebeans services panel. Then tried to connect to my database file from Services Databases using this URL for my database: jdbc:sqlite:/home/farzad/netbeans/myproject/mydb.sqlite but it fails to connect. I get this exception: org.netbeans.modules.db.dataview.meta.DBException: Unable to Connect to database : DatabaseConnection[name='jdbc:sqlite://home/farzad/netbeans/myproject/mydb.sqlite [ on session]'] at org.netbeans.modules.db.dataview.output.SQLExecutionHelper.initialDataLoad(SQLExecutionHelper.java:103) at org.netbeans.modules.db.dataview.output.DataView.create(DataView.java:101) at org.netbeans.modules.db.dataview.api.DataView.create(DataView.java:71) at org.netbeans.modules.db.sql.execute.SQLExecuteHelper.execute(SQLExecuteHelper.java:105) at org.netbeans.modules.db.sql.loader.SQLEditorSupport$SQLExecutor.run(SQLEditorSupport.java:480) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:572) [catch] at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:997) What should I do? :(

    Read the article

  • How to write custom SQLite functions in Javascript inside a Webkit browser?

    - by Jay Godse
    I have just learned how to use the SQLite database for local storage in a Webkit web browser (e.g. Google Chrome or Apple Safari) using the Javascript API. For example the "Sticky Notes" application. However, I know that SQLite has a function called sqlite_create_function() that lets you add custom functions to your instance of SQLite on the fly which can then be used inside SQL queries. This function is described at sqlite.org. I also know that you can call an equivalent of this API in Ruby as described here. QUESTION: Can anybody show me how to do this in Javascript - i.e. write a custom function in Javascript that can be bound into the SQLite database at run time to be called by the SQLite engine, and all inside a Webkit browser?

    Read the article

  • Places to share my new blog entry? [closed]

    - by TomasAlabes
    I started yesterday a tech blog called devhike and as my first entry I explained a palette behaviour made with RaphaelJS. I wanted to share the entry the tech people around the world, to see if they like it, find it useful, etc. I submitted the link to my post in dzone.com and hackernews which are the places from where I like to read tech articles, but I feel there should be other places where I could post my blog or entry too. If not, I feel my blog will never be read. Do you know any places or ways to be read?

    Read the article

  • SQLite, python, unicode, and non-utf data

    - by Nathan Spears
    I started by trying to store strings in sqlite using python, and got the message: sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. Ok, I switched to Unicode strings. Then I started getting the message: sqlite3.OperationalError: Could not decode to UTF-8 column 'tag_artist' with text 'Sigur Rós' when trying to retrieve data from the db. More research and I started encoding it in utf8, but then 'Sigur Rós' starts looking like 'Sigur Rós' note: My console was set to display in 'latin_1' as @John Machin pointed out. What gives? After reading this, describing exactly the same situation I'm in, it seems as if the advice is to ignore the other advice and use 8-bit bytestrings after all. I didn't know much about unicode and utf before I started this process. I've learned quite a bit in the last couple hours, but I'm still ignorant of whether there is a way to correctly convert 'ó' from latin-1 to utf-8 and not mangle it. If there isn't, why would sqlite 'highly recommend' I switch my application to unicode strings? I'm going to update this question with a summary and some example code of everything I've learned in the last 24 hours so that someone in my shoes can have an easy(er) guide. If the information I post is wrong or misleading in any way please tell me and I'll update, or one of you senior guys can update. Summary of answers Let me first state the goal as I understand it. The goal in processing various encodings, if you are trying to convert between them, is to understand what your source encoding is, then convert it to unicode using that source encoding, then convert it to your desired encoding. Unicode is a base and encodings are mappings of subsets of that base. utf_8 has room for every character in unicode, but because they aren't in the same place as, for instance, latin_1, a string encoded in utf_8 and sent to a latin_1 console will not look the way you expect. In python the process of getting to unicode and into another encoding looks like: str.decode('source_encoding').encode('desired_encoding') or if the str is already in unicode str.encode('desired_encoding') For sqlite I didn't actually want to encode it again, I wanted to decode it and leave it in unicode format. Here are four things you might need to be aware of as you try to work with unicode and encodings in python. The encoding of the string you want to work with, and the encoding you want to get it to. The system encoding. The console encoding. The encoding of the source file Elaboration: (1) When you read a string from a source, it must have some encoding, like latin_1 or utf_8. In my case, I'm getting strings from filenames, so unfortunately, I could be getting any kind of encoding. Windows XP uses UCS-2 (a Unicode system) as its native string type, which seems like cheating to me. Fortunately for me, the characters in most filenames are not going to be made up of more than one source encoding type, and I think all of mine were either completely latin_1, completely utf_8, or just plain ascii (which is a subset of both of those). So I just read them and decoded them as if they were still in latin_1 or utf_8. It's possible, though, that you could have latin_1 and utf_8 and whatever other characters mixed together in a filename on Windows. Sometimes those characters can show up as boxes, other times they just look mangled, and other times they look correct (accented characters and whatnot). Moving on. (2) Python has a default system encoding that gets set when python starts and can't be changed during runtime. See here for details. Dirty summary ... well here's the file I added: \# sitecustomize.py \# this file can be anywhere in your Python path, \# but it usually goes in ${pythondir}/lib/site-packages/ import sys sys.setdefaultencoding('utf_8') This system encoding is the one that gets used when you use the unicode("str") function without any other encoding parameters. To say that another way, python tries to decode "str" to unicode based on the default system encoding. (3) If you're using IDLE or the command-line python, I think that your console will display according to the default system encoding. I am using pydev with eclipse for some reason, so I had to go into my project settings, edit the launch configuration properties of my test script, go to the Common tab, and change the console from latin-1 to utf-8 so that I could visually confirm what I was doing was working. (4) If you want to have some test strings, eg test_str = "ó" in your source code, then you will have to tell python what kind of encoding you are using in that file. (FYI: when I mistyped an encoding I had to ctrl-Z because my file became unreadable.) This is easily accomplished by putting a line like so at the top of your source code file: # -*- coding: utf_8 -*- If you don't have this information, python attempts to parse your code as ascii by default, and so: SyntaxError: Non-ASCII character '\xf3' in file _redacted_ on line 81, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details Once your program is working correctly, or, if you aren't using python's console or any other console to look at output, then you will probably really only care about #1 on the list. System default and console encoding are not that important unless you need to look at output and/or you are using the builtin unicode() function (without any encoding parameters) instead of the string.decode() function. I wrote a demo function I will paste into the bottom of this gigantic mess that I hope correctly demonstrates the items in my list. Here is some of the output when I run the character 'ó' through the demo function, showing how various methods react to the character as input. My system encoding and console output are both set to utf_8 for this run: '?' = original char <type 'str'> repr(char)='\xf3' '?' = unicode(char) ERROR: 'utf8' codec can't decode byte 0xf3 in position 0: unexpected end of data 'ó' = char.decode('latin_1') <type 'unicode'> repr(char.decode('latin_1'))=u'\xf3' '?' = char.decode('utf_8') ERROR: 'utf8' codec can't decode byte 0xf3 in position 0: unexpected end of data Now I will change the system and console encoding to latin_1, and I get this output for the same input: 'ó' = original char <type 'str'> repr(char)='\xf3' 'ó' = unicode(char) <type 'unicode'> repr(unicode(char))=u'\xf3' 'ó' = char.decode('latin_1') <type 'unicode'> repr(char.decode('latin_1'))=u'\xf3' '?' = char.decode('utf_8') ERROR: 'utf8' codec can't decode byte 0xf3 in position 0: unexpected end of data Notice that the 'original' character displays correctly and the builtin unicode() function works now. Now I change my console output back to utf_8. '?' = original char <type 'str'> repr(char)='\xf3' '?' = unicode(char) <type 'unicode'> repr(unicode(char))=u'\xf3' '?' = char.decode('latin_1') <type 'unicode'> repr(char.decode('latin_1'))=u'\xf3' '?' = char.decode('utf_8') ERROR: 'utf8' codec can't decode byte 0xf3 in position 0: unexpected end of data Here everything still works the same as last time but the console can't display the output correctly. Etc. The function below also displays more information that this and hopefully would help someone figure out where the gap in their understanding is. I know all this information is in other places and more thoroughly dealt with there, but I hope that this would be a good kickoff point for someone trying to get coding with python and/or sqlite. Ideas are great but sometimes source code can save you a day or two of trying to figure out what functions do what. Disclaimers: I'm no encoding expert, I put this together to help my own understanding. I kept building on it when I should have probably started passing functions as arguments to avoid so much redundant code, so if I can I'll make it more concise. Also, utf_8 and latin_1 are by no means the only encoding schemes, they are just the two I was playing around with because I think they handle everything I need. Add your own encoding schemes to the demo function and test your own input. One more thing: there are apparently crazy application developers making life difficult in Windows. #!/usr/bin/env python # -*- coding: utf_8 -*- import os import sys def encodingDemo(str): validStrings = () try: print "str =",str,"{0} repr(str) = {1}".format(type(str), repr(str)) validStrings += ((str,""),) except UnicodeEncodeError as ude: print "Couldn't print the str itself because the console is set to an encoding that doesn't understand some character in the string. See error:\n\t", print ude try: x = unicode(str) print "unicode(str) = ",x validStrings+= ((x, " decoded into unicode by the default system encoding"),) except UnicodeDecodeError as ude: print "ERROR. unicode(str) couldn't decode the string because the system encoding is set to an encoding that doesn't understand some character in the string." print "\tThe system encoding is set to {0}. See error:\n\t".format(sys.getdefaultencoding()), print ude except UnicodeEncodeError as uee: print "ERROR. Couldn't print the unicode(str) because the console is set to an encoding that doesn't understand some character in the string. See error:\n\t", print uee try: x = str.decode('latin_1') print "str.decode('latin_1') =",x validStrings+= ((x, " decoded with latin_1 into unicode"),) try: print "str.decode('latin_1').encode('utf_8') =",str.decode('latin_1').encode('utf_8') validStrings+= ((x, " decoded with latin_1 into unicode and encoded into utf_8"),) except UnicodeDecodeError as ude: print "The string was decoded into unicode using the latin_1 encoding, but couldn't be encoded into utf_8. See error:\n\t", print ude except UnicodeDecodeError as ude: print "Something didn't work, probably because the string wasn't latin_1 encoded. See error:\n\t", print ude except UnicodeEncodeError as uee: print "ERROR. Couldn't print the str.decode('latin_1') because the console is set to an encoding that doesn't understand some character in the string. See error:\n\t", print uee try: x = str.decode('utf_8') print "str.decode('utf_8') =",x validStrings+= ((x, " decoded with utf_8 into unicode"),) try: print "str.decode('utf_8').encode('latin_1') =",str.decode('utf_8').encode('latin_1') except UnicodeDecodeError as ude: print "str.decode('utf_8').encode('latin_1') didn't work. The string was decoded into unicode using the utf_8 encoding, but couldn't be encoded into latin_1. See error:\n\t", validStrings+= ((x, " decoded with utf_8 into unicode and encoded into latin_1"),) print ude except UnicodeDecodeError as ude: print "str.decode('utf_8') didn't work, probably because the string wasn't utf_8 encoded. See error:\n\t", print ude except UnicodeEncodeError as uee: print "ERROR. Couldn't print the str.decode('utf_8') because the console is set to an encoding that doesn't understand some character in the string. See error:\n\t",uee print print "Printing information about each character in the original string." for char in str: try: print "\t'" + char + "' = original char {0} repr(char)={1}".format(type(char), repr(char)) except UnicodeDecodeError as ude: print "\t'?' = original char {0} repr(char)={1} ERROR PRINTING: {2}".format(type(char), repr(char), ude) except UnicodeEncodeError as uee: print "\t'?' = original char {0} repr(char)={1} ERROR PRINTING: {2}".format(type(char), repr(char), uee) print uee try: x = unicode(char) print "\t'" + x + "' = unicode(char) {1} repr(unicode(char))={2}".format(x, type(x), repr(x)) except UnicodeDecodeError as ude: print "\t'?' = unicode(char) ERROR: {0}".format(ude) except UnicodeEncodeError as uee: print "\t'?' = unicode(char) {0} repr(char)={1} ERROR PRINTING: {2}".format(type(x), repr(x), uee) try: x = char.decode('latin_1') print "\t'" + x + "' = char.decode('latin_1') {1} repr(char.decode('latin_1'))={2}".format(x, type(x), repr(x)) except UnicodeDecodeError as ude: print "\t'?' = char.decode('latin_1') ERROR: {0}".format(ude) except UnicodeEncodeError as uee: print "\t'?' = char.decode('latin_1') {0} repr(char)={1} ERROR PRINTING: {2}".format(type(x), repr(x), uee) try: x = char.decode('utf_8') print "\t'" + x + "' = char.decode('utf_8') {1} repr(char.decode('utf_8'))={2}".format(x, type(x), repr(x)) except UnicodeDecodeError as ude: print "\t'?' = char.decode('utf_8') ERROR: {0}".format(ude) except UnicodeEncodeError as uee: print "\t'?' = char.decode('utf_8') {0} repr(char)={1} ERROR PRINTING: {2}".format(type(x), repr(x), uee) print x = 'ó' encodingDemo(x) Much thanks for the answers below and especially to @John Machin for answering so thoroughly.

    Read the article

  • SQLite, python, unicode, and non-utf data

    - by Nathan Spears
    I started by trying to store strings in sqlite using python, and got the message: sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. Ok, I switched to Unicode strings. Then I started getting the message: sqlite3.OperationalError: Could not decode to UTF-8 column 'tag_artist' with text 'Sigur Rós' when trying to retrieve data from the db. More research and I started encoding it in utf8, but then 'Sigur Rós' starts looking like 'Sigur Rós' note: My console was set to display in 'latin_1' as @John Machin pointed out. What gives? After reading this, describing exactly the same situation I'm in, it seems as if the advice is to ignore the other advice and use 8-bit bytestrings after all. I didn't know much about unicode and utf before I started this process. I've learned quite a bit in the last couple hours, but I'm still ignorant of whether there is a way to correctly convert 'ó' from latin-1 to utf-8 and not mangle it. If there isn't, why would sqlite 'highly recommend' I switch my application to unicode strings? I'm going to update this question with a summary and some example code of everything I've learned in the last 24 hours so that someone in my shoes can have an easy(er) guide. If the information I post is wrong or misleading in any way please tell me and I'll update, or one of you senior guys can update. Summary of answers Let me first state the goal as I understand it. The goal in processing various encodings, if you are trying to convert between them, is to understand what your source encoding is, then convert it to unicode using that source encoding, then convert it to your desired encoding. Unicode is a base and encodings are mappings of subsets of that base. utf_8 has room for every character in unicode, but because they aren't in the same place as, for instance, latin_1, a string encoded in utf_8 and sent to a latin_1 console will not look the way you expect. In python the process of getting to unicode and into another encoding looks like: str.decode('source_encoding').encode('desired_encoding') or if the str is already in unicode str.encode('desired_encoding') For sqlite I didn't actually want to encode it again, I wanted to decode it and leave it in unicode format. Here are four things you might need to be aware of as you try to work with unicode and encodings in python. The encoding of the string you want to work with, and the encoding you want to get it to. The system encoding. The console encoding. The encoding of the source file Elaboration: (1) When you read a string from a source, it must have some encoding, like latin_1 or utf_8. In my case, I'm getting strings from filenames, so unfortunately, I could be getting any kind of encoding. Windows XP uses UCS-2 (a Unicode system) as its native string type, which seems like cheating to me. Fortunately for me, the characters in most filenames are not going to be made up of more than one source encoding type, and I think all of mine were either completely latin_1, completely utf_8, or just plain ascii (which is a subset of both of those). So I just read them and decoded them as if they were still in latin_1 or utf_8. It's possible, though, that you could have latin_1 and utf_8 and whatever other characters mixed together in a filename on Windows. Sometimes those characters can show up as boxes, other times they just look mangled, and other times they look correct (accented characters and whatnot). Moving on. (2) Python has a default system encoding that gets set when python starts and can't be changed during runtime. See here for details. Dirty summary ... well here's the file I added: \# sitecustomize.py \# this file can be anywhere in your Python path, \# but it usually goes in ${pythondir}/lib/site-packages/ import sys sys.setdefaultencoding('utf_8') This system encoding is the one that gets used when you use the unicode("str") function without any other encoding parameters. To say that another way, python tries to decode "str" to unicode based on the default system encoding. (3) If you're using IDLE or the command-line python, I think that your console will display according to the default system encoding. I am using pydev with eclipse for some reason, so I had to go into my project settings, edit the launch configuration properties of my test script, go to the Common tab, and change the console from latin-1 to utf-8 so that I could visually confirm what I was doing was working. (4) If you want to have some test strings, eg test_str = "ó" in your source code, then you will have to tell python what kind of encoding you are using in that file. (FYI: when I mistyped an encoding I had to ctrl-Z because my file became unreadable.) This is easily accomplished by putting a line like so at the top of your source code file: # -*- coding: utf_8 -*- If you don't have this information, python attempts to parse your code as ascii by default, and so: SyntaxError: Non-ASCII character '\xf3' in file _redacted_ on line 81, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details Once your program is working correctly, or, if you aren't using python's console or any other console to look at output, then you will probably really only care about #1 on the list. System default and console encoding are not that important unless you need to look at output and/or you are using the builtin unicode() function (without any encoding parameters) instead of the string.decode() function. I wrote a demo function I will paste into the bottom of this gigantic mess that I hope correctly demonstrates the items in my list. Here is some of the output when I run the character 'ó' through the demo function, showing how various methods react to the character as input. My system encoding and console output are both set to utf_8 for this run: '?' = original char <type 'str'> repr(char)='\xf3' '?' = unicode(char) ERROR: 'utf8' codec can't decode byte 0xf3 in position 0: unexpected end of data 'ó' = char.decode('latin_1') <type 'unicode'> repr(char.decode('latin_1'))=u'\xf3' '?' = char.decode('utf_8') ERROR: 'utf8' codec can't decode byte 0xf3 in position 0: unexpected end of data Now I will change the system and console encoding to latin_1, and I get this output for the same input: 'ó' = original char <type 'str'> repr(char)='\xf3' 'ó' = unicode(char) <type 'unicode'> repr(unicode(char))=u'\xf3' 'ó' = char.decode('latin_1') <type 'unicode'> repr(char.decode('latin_1'))=u'\xf3' '?' = char.decode('utf_8') ERROR: 'utf8' codec can't decode byte 0xf3 in position 0: unexpected end of data Notice that the 'original' character displays correctly and the builtin unicode() function works now. Now I change my console output back to utf_8. '?' = original char <type 'str'> repr(char)='\xf3' '?' = unicode(char) <type 'unicode'> repr(unicode(char))=u'\xf3' '?' = char.decode('latin_1') <type 'unicode'> repr(char.decode('latin_1'))=u'\xf3' '?' = char.decode('utf_8') ERROR: 'utf8' codec can't decode byte 0xf3 in position 0: unexpected end of data Here everything still works the same as last time but the console can't display the output correctly. Etc. The function below also displays more information that this and hopefully would help someone figure out where the gap in their understanding is. I know all this information is in other places and more thoroughly dealt with there, but I hope that this would be a good kickoff point for someone trying to get coding with python and/or sqlite. Ideas are great but sometimes source code can save you a day or two of trying to figure out what functions do what. Disclaimers: I'm no encoding expert, I put this together to help my own understanding. I kept building on it when I should have probably started passing functions as arguments to avoid so much redundant code, so if I can I'll make it more concise. Also, utf_8 and latin_1 are by no means the only encoding schemes, they are just the two I was playing around with because I think they handle everything I need. Add your own encoding schemes to the demo function and test your own input. One more thing: there are apparently crazy application developers making life difficult in Windows. #!/usr/bin/env python # -*- coding: utf_8 -*- import os import sys def encodingDemo(str): validStrings = () try: print "str =",str,"{0} repr(str) = {1}".format(type(str), repr(str)) validStrings += ((str,""),) except UnicodeEncodeError as ude: print "Couldn't print the str itself because the console is set to an encoding that doesn't understand some character in the string. See error:\n\t", print ude try: x = unicode(str) print "unicode(str) = ",x validStrings+= ((x, " decoded into unicode by the default system encoding"),) except UnicodeDecodeError as ude: print "ERROR. unicode(str) couldn't decode the string because the system encoding is set to an encoding that doesn't understand some character in the string." print "\tThe system encoding is set to {0}. See error:\n\t".format(sys.getdefaultencoding()), print ude except UnicodeEncodeError as uee: print "ERROR. Couldn't print the unicode(str) because the console is set to an encoding that doesn't understand some character in the string. See error:\n\t", print uee try: x = str.decode('latin_1') print "str.decode('latin_1') =",x validStrings+= ((x, " decoded with latin_1 into unicode"),) try: print "str.decode('latin_1').encode('utf_8') =",str.decode('latin_1').encode('utf_8') validStrings+= ((x, " decoded with latin_1 into unicode and encoded into utf_8"),) except UnicodeDecodeError as ude: print "The string was decoded into unicode using the latin_1 encoding, but couldn't be encoded into utf_8. See error:\n\t", print ude except UnicodeDecodeError as ude: print "Something didn't work, probably because the string wasn't latin_1 encoded. See error:\n\t", print ude except UnicodeEncodeError as uee: print "ERROR. Couldn't print the str.decode('latin_1') because the console is set to an encoding that doesn't understand some character in the string. See error:\n\t", print uee try: x = str.decode('utf_8') print "str.decode('utf_8') =",x validStrings+= ((x, " decoded with utf_8 into unicode"),) try: print "str.decode('utf_8').encode('latin_1') =",str.decode('utf_8').encode('latin_1') except UnicodeDecodeError as ude: print "str.decode('utf_8').encode('latin_1') didn't work. The string was decoded into unicode using the utf_8 encoding, but couldn't be encoded into latin_1. See error:\n\t", validStrings+= ((x, " decoded with utf_8 into unicode and encoded into latin_1"),) print ude except UnicodeDecodeError as ude: print "str.decode('utf_8') didn't work, probably because the string wasn't utf_8 encoded. See error:\n\t", print ude except UnicodeEncodeError as uee: print "ERROR. Couldn't print the str.decode('utf_8') because the console is set to an encoding that doesn't understand some character in the string. See error:\n\t",uee print print "Printing information about each character in the original string." for char in str: try: print "\t'" + char + "' = original char {0} repr(char)={1}".format(type(char), repr(char)) except UnicodeDecodeError as ude: print "\t'?' = original char {0} repr(char)={1} ERROR PRINTING: {2}".format(type(char), repr(char), ude) except UnicodeEncodeError as uee: print "\t'?' = original char {0} repr(char)={1} ERROR PRINTING: {2}".format(type(char), repr(char), uee) print uee try: x = unicode(char) print "\t'" + x + "' = unicode(char) {1} repr(unicode(char))={2}".format(x, type(x), repr(x)) except UnicodeDecodeError as ude: print "\t'?' = unicode(char) ERROR: {0}".format(ude) except UnicodeEncodeError as uee: print "\t'?' = unicode(char) {0} repr(char)={1} ERROR PRINTING: {2}".format(type(x), repr(x), uee) try: x = char.decode('latin_1') print "\t'" + x + "' = char.decode('latin_1') {1} repr(char.decode('latin_1'))={2}".format(x, type(x), repr(x)) except UnicodeDecodeError as ude: print "\t'?' = char.decode('latin_1') ERROR: {0}".format(ude) except UnicodeEncodeError as uee: print "\t'?' = char.decode('latin_1') {0} repr(char)={1} ERROR PRINTING: {2}".format(type(x), repr(x), uee) try: x = char.decode('utf_8') print "\t'" + x + "' = char.decode('utf_8') {1} repr(char.decode('utf_8'))={2}".format(x, type(x), repr(x)) except UnicodeDecodeError as ude: print "\t'?' = char.decode('utf_8') ERROR: {0}".format(ude) except UnicodeEncodeError as uee: print "\t'?' = char.decode('utf_8') {0} repr(char)={1} ERROR PRINTING: {2}".format(type(x), repr(x), uee) print x = 'ó' encodingDemo(x) Much thanks for the answers below and especially to @John Machin for answering so thoroughly.

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 6 7 8 9 10 11 12 13 14 15 16 17  | Next Page >