How to finish a broken data upload to the production Google App Engine server?

Posted by WooYek on Stack Overflow See other posts from Stack Overflow or by WooYek
Published on 2010-03-30T17:30:52Z Indexed on 2010/03/30 17:33 UTC
Read the original article Hit count: 391

Filed under:

google-app-engine

|

google-datastore

|

bulk

|

upload

I was uploading the data to App Engine (not dev server) through loader class and remote api, and I hit the quota in the middle of a CSV file. Based on logs and progress sqllite db, how can I select remaining portion of data to be uploaded?

Going through tens of records to determine which was and which was not transfered, is not appealing task, so I look for some way to limit the number of record I need to check.

Here's relevant (IMO) log portion, how to interpret work item numbers?

[DEBUG    2010-03-30 03:22:51,757 bulkloader.py] [Thread-2] [1041-1050] Transferred 10 entities in 3.9 seconds
[DEBUG    2010-03-30 03:22:51,757 adaptive_thread_pool.py] [Thread-2] Got work item [1071-1080]
<cut>
[DEBUG    2010-03-30 03:23:09,194 bulkloader.py] [Thread-1] [1141-1150] Transferred 10 entities in 4.6 seconds
[DEBUG    2010-03-30 03:23:09,194 adaptive_thread_pool.py] [Thread-1] Got work item [1161-1170]
<cut>
[DEBUG    2010-03-30 03:23:09,226 bulkloader.py] [Thread-3] [1151-1160] Transferred 10 entities in 4.2 seconds
[DEBUG    2010-03-30 03:23:09,226 adaptive_thread_pool.py] [Thread-3] Got work item [1171-1180]
[ERROR    2010-03-30 03:23:10,174 bulkloader.py] Retrying on non-fatal HTTP error: 503 Service Unavailable

© Stack Overflow or respective owner

Related posts about google-app-engine

unittest import error with virtualenv + google-app-engine-django

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm working with google-app-engine-django + zipped django. Just running "python manage.py test" succeeded without error. But with virtualenv, test was failed with "import unittest error". same error with Django 1.1. - OSX 10.5.6 - google-app-engine-django (r101 via svn) : r100 was failed with launcher… >>> More
Could not import Django settings into Google App Engine

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello all you Google App Engine experts, I have used Django a little before but am new to Google App Engine and am trying to use it's development web server with Django for the first time. I don't know if this is relevent but I previously had Django 1.1 and Python 2.6 on my Windows XP and even… >>> More
How does Google App Engine precompile Java ?

as seen on Stack Overflow - Search for 'Stack Overflow'
App Engine uses a "precompilation" process with the Java bytecode of an app to enhance the performance of the app in the Java runtime environment. Precompiled code functions identically to the original bytecode. Is there any detailed information what this does? >>> More
Creating a session scoped bean in Google App Engine using Spring 2.5

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am trying to create a session bean in spring mvc. I am having the following error message when I run my google app engine server in my local box: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'siteController' defined in ServletContext resource [/WEB-INF/springapp-servlet… >>> More
Google App Engine UI Widgets

as seen on Stack Overflow - Search for 'Stack Overflow'
Are there any UI widgets available to the python side of Google App Engine? I'd like something like the collapsed/expanded views of Google Groups threads. Are these type things limited to the GWT side? >>> More

Related posts about google-datastore

Google Datastore w/ JDO: Access Times?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm hitting what appears (to me) strange behavior when I pull data from the google datastore over JDO. In particular, the query executes quickly (say 100 ms), but finding the size of the resulting List< takes about one second! Indeed, whatever operation I try to perform on the resulting list takes… >>> More
How to upload 6000 record to Google Datastore from csv file

as seen on Stack Overflow - Search for 'Stack Overflow'
http://code.google.com/appengine/docs/python/tools/uploadingdata.html is not clearly understand. Where i should call the bulkloader.py or appcfg.py? Should i import the csv file to local Google App Engine SDK first? How to keep the upload and download data process in existing application for datastore… >>> More
ArrayList throwing exception on retrieval from google datastore (with gwt, java)

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm using Google Web Toolkit with java and google datastore as database. The entity class has arraylist and on trying to retrieve the data from data base I'm getting the exception: Type 'org.datanucleus.sco.backed.ArrayList' was not included in the set of types which can be serialized by this SerializationPolicy… >>> More
google datastore - does it do lazy loading?

as seen on Stack Overflow - Search for 'Stack Overflow'
if I have a Customer object with a list of orders, declared using the db.ReferenceProperty after a while I may have huge amount of orders in there, if I pull the Customer object would I be in danger of pulling the complete set of orders? >>> More
From actionscript to google's datastore through java.

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm working on a flash game written in pure actionscript 3.0 in Flex. I've just finished implementing replays for the game, but want to store the top 10 hiscores' replay data on my google-app-engine'd website. I'm using Java for the app-engine stuff in Eclipse in java but I have no idea how to… >>> More