GAE uploading to database - or - how to create big dictionary in GAE?

Posted by Koran on Stack Overflow See other posts from Stack Overflow or by Koran
Published on 2010-05-31T08:22:01Z Indexed on 2010/05/31 8:42 UTC
Read the original article Hit count: 259

Filed under:

google-app-engine

|

upload

|

cron

Hi,

I have a GAE application. A new request is to have a huge database to store many items for some functionality.

The database needs to populated every week or so - and populating it is done by querying many external links which overall takes around 1-2 hours. So, doing it via GAE Cron is now out of the question.

Another option I was thinking was to create it in external server and then uploading it to GAE DB. Is it possible to upload the data created externally to GAE DB straightaway? The question is because there will be close to 10 M items in it, so adding items individually to a DB is not going to work due to 30 second time limit.

Thanks

K

© Stack Overflow or respective owner

Related posts about google-app-engine

unittest import error with virtualenv + google-app-engine-django

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm working with google-app-engine-django + zipped django. Just running "python manage.py test" succeeded without error. But with virtualenv, test was failed with "import unittest error". same error with Django 1.1. - OSX 10.5.6 - google-app-engine-django (r101 via svn) : r100 was failed with launcher… >>> More
Could not import Django settings into Google App Engine

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello all you Google App Engine experts, I have used Django a little before but am new to Google App Engine and am trying to use it's development web server with Django for the first time. I don't know if this is relevent but I previously had Django 1.1 and Python 2.6 on my Windows XP and even… >>> More
How does Google App Engine precompile Java ?

as seen on Stack Overflow - Search for 'Stack Overflow'
App Engine uses a "precompilation" process with the Java bytecode of an app to enhance the performance of the app in the Java runtime environment. Precompiled code functions identically to the original bytecode. Is there any detailed information what this does? >>> More
Creating a session scoped bean in Google App Engine using Spring 2.5

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am trying to create a session bean in spring mvc. I am having the following error message when I run my google app engine server in my local box: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'siteController' defined in ServletContext resource [/WEB-INF/springapp-servlet… >>> More
Google App Engine UI Widgets

as seen on Stack Overflow - Search for 'Stack Overflow'
Are there any UI widgets available to the python side of Google App Engine? I'd like something like the collapsed/expanded views of Google Groups threads. Are these type things limited to the GWT side? >>> More

Related posts about upload

PHP File Upload second file does not upload, first file does without error

as seen on Stack Overflow - Search for 'Stack Overflow'
So I have a script I have been using and it generally works well with multiple files... When I upload a very large file in a multiple file upload, only the first file is uploaded. I am not seeing an errors as to why. I figure this is related to a timeout setting but can not figure it out - Any… >>> More
Upload multiple Images and Ping an address to upload them

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello! I am trying to allow administrators to upload 20-30 images in one go. I have tried jquery uploadify, SWFUpload and other various jquery scripts. However I have had many issues doing this. Uploadify worked fine, however it stopped working and after a day of debugging I couldn't figure out… >>> More
Upload an image using class.upload.php via AJAX, display image in form using jQuery

as seen on Stack Overflow - Search for 'Stack Overflow'
I am using class.upload.php to handle an image upload from a form that submits employee details to a MySQL database. class.upload.php does EXACTLY what I want it to do, resize the image and rename it - what I am now trying to accomplish is to upload via Ajax after the user selects the file and then… >>> More
Upload File to Windows Azure Blob in Chunks through ASP.NET MVC, JavaScript and HTML5

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
Originally posted on: http://geekswithblogs.net/shaunxu/archive/2013/07/01/upload-file-to-windows-azure-blob-in-chunks-through-asp.net.aspxMany people are using Windows Azure Blob Storage to store their data in the cloud. Blob storage provides 99.9% availability with easy-to-use API through .NET SDK… >>> More
javascript replace text with images problem

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm extremely new to JS and have this code that I'm trying to tweak. WHen I was adding the array, I had tested it with only a couple of items and it was working fine, now it just doesn't work, and I can't figure out what is wrong with it!! Basically, I'm trying to change every instance of a card… >>> More