Search Results

Search found 20 results on 1 pages for 'bulkloader'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 1/1 | 1 

  • App Engine Bulkloader

    - by gurkan
    Hi all, I am trying to use Bulkloader of google app engine but unfortunately could not understand what to do from documentation. It says add this part to app.yaml builtins: - remote_api: on ok i have added. Then says that i have to execute this command appcfg.py update but i don't have any appcfg.py file. And also what is the command which executes this line? Please somebody tell me what i am missing I use AppEngineLauncher to upload my project to server.. I have naver used a command to update or upload it. Thanks in advance..

    Read the article

  • overwrite existing entity via bulkloader.Loader

    - by Ray Yun
    I was going to CSV based export/import for large data with app engine. My idea was just simple. First column of CSV would be key of entity. If it's not empty, that row means existing entity and should overwrite old one. Else, that row is new entity and should create new one. I could export key of entity by adding key property. class FrontExporter(bulkloader.Exporter): def __init__(self): bulkloader.Exporter.__init__(self, 'Front', [ ('__key__', str, None), ('name', str, None), ]) But when I was trying to upload CSV, it had failed because bulkloader.Loader.generate_key() was just for "key_name" not "key" itself. That means all exported entities in CSV should have unique 'key_name' if I want to modify-and-reupload them. class FrontLoader(bulkloader.Loader): def __init__(self): bulkloader.Loader.__init__(self, 'Front', [ ('_UNUSED', lambda x: None), ('name', lambda x: x.decode('utf-8')), ]) def generate_key(self,i,values): # first column is key keystr = values[0] if len(keystr)==0: return None return keystr I also tried to load key directly without using generate_key(), but both failed. class FrontLoader(bulkloader.Loader): def __init__(self): bulkloader.Loader.__init__(self, 'Front', [ ('Key', db.Key), # not working. just create new one. ('__key__', db.Key), # same... So, how can I overwrite existing entity which has no 'key_name'? It would be horrible if I should give unique name to all entities..... From the first answer, I could handle this problem. :) def create_entity(self, values, key_name=None, parent=None): # if key_name is None: # print 'key_name is None' # else: # print 'key_name=<',key_name,'> : length=',len(key_name) Validate(values, (list, tuple)) assert len(values) == len(self._Loader__properties), ( 'Expected %d columns, found %d.' % (len(self._Loader__properties), len(values))) model_class = GetImplementationClass(self.kind) properties = { 'key_name': key_name, 'parent': parent, } for (name, converter), val in zip(self._Loader__properties, values): if converter is bool and val.lower() in ('0', 'false', 'no'): val = False properties[name] = converter(val) if key_name is None: entity = model_class(**properties) #print 'create new one' else: entity = model_class.get(key_name) for key, value in properties.items(): setattr(entity, key, value) #print 'overwrite old one' entities = self.handle_entity(entity) if entities: if not isinstance(entities, (list, tuple)): entities = [entities] for entity in entities: if not isinstance(entity, db.Model): raise TypeError('Expected a db.Model, received %s (a %s).' % (entity, entity.__class__)) return entities def generate_key(self,i,values): # first column is key if values[0] is None or values[0] in ('',' ','-','.'): return None return values[0]

    Read the article

  • Which credentials should I put in for Google App Engine BulkLoader at development server?

    - by Hoang Pham
    Hello everyone, I would like to ask which kind of credentials do I need to put on for importing data using the Google App Engine BulkLoader class appcfg.py upload_data --config_file=models.py --filename=listcountries.csv --kind=CMSCountry --url=http://localhost:8178/remote_api vit/ And then it asks me for credentials: Please enter login credentials for localhost Here is an extraction of the content of the models.py, I use this listcountries.csv file class CMSCountry(db.Model): sortorder = db.StringProperty() name = db.StringProperty(required=True) formalname = db.StringProperty() type = db.StringProperty() subtype = db.StringProperty() sovereignt = db.StringProperty() capital = db.StringProperty() currencycode = db.StringProperty() currencyname = db.StringProperty() telephonecode = db.StringProperty() lettercode = db.StringProperty() lettercode2 = db.StringProperty() number = db.StringProperty() countrycode = db.StringProperty() class CMSCountryLoader(bulkloader.Loader): def __init__(self): bulkloader.Loader.__init__(self, 'CMSCountry', [('sortorder', str), ('name', str), ('formalname', str), ('type', str), ('subtype', str), ('sovereignt', str), ('capital', str), ('currencycode', str), ('currencyname', str), ('telephonecode', str), ('lettercode', str), ('lettercode2', str), ('number', str), ('countrycode', str) ]) loaders = [CMSCountryLoader] Every tries to enter the email and password result in "Authentication Failed", so I could not import the data to the development server. I don't think that I have any problem with my files neither my models because I have successfully uploaded the data to the appspot.com application. So what should I put in for localhost credentials? I also tried to use Eclipse with Pydev but I still got the same message :( Here is the output: Uploading data records. [INFO ] Logging to bulkloader-log-20090820.121659 [INFO ] Opening database: bulkloader-progress-20090820.121659.sql3 [INFO ] [Thread-1] WorkerThread: started [INFO ] [Thread-2] WorkerThread: started [INFO ] [Thread-3] WorkerThread: started [INFO ] [Thread-4] WorkerThread: started [INFO ] [Thread-5] WorkerThread: started [INFO ] [Thread-6] WorkerThread: started [INFO ] [Thread-7] WorkerThread: started [INFO ] [Thread-8] WorkerThread: started [INFO ] [Thread-9] WorkerThread: started [INFO ] [Thread-10] WorkerThread: started Password for [email protected]: [DEBUG ] Configuring remote_api. url_path = /remote_api, servername = localhost:8178 [DEBUG ] Bulkloader using app_id: abc [INFO ] Connecting to /remote_api [ERROR ] Exception during authentication Traceback (most recent call last): File "D:\Projects\GoogleAppEngine\google_appengine\google\appengine\tools\bulkloader.py", line 2802, in Run request_manager.Authenticate() File "D:\Projects\GoogleAppEngine\google_appengine\google\appengine\tools\bulkloader.py", line 1126, in Authenticate remote_api_stub.MaybeInvokeAuthentication() File "D:\Projects\GoogleAppEngine\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", line 488, in MaybeInvokeAuthentication datastore_stub._server.Send(datastore_stub._path, payload=None) File "D:\Projects\GoogleAppEngine\google_appengine\google\appengine\tools\appengine_rpc.py", line 344, in Send f = self.opener.open(req) File "C:\Python25\lib\urllib2.py", line 381, in open response = self._open(req, data) File "C:\Python25\lib\urllib2.py", line 399, in _open '_open', req) File "C:\Python25\lib\urllib2.py", line 360, in _call_chain result = func(*args) File "C:\Python25\lib\urllib2.py", line 1107, in http_open return self.do_open(httplib.HTTPConnection, req) File "C:\Python25\lib\urllib2.py", line 1082, in do_open raise URLError(err) URLError: <urlopen error (10061, 'Connection refused')> [INFO ] Authentication Failed Thank you!

    Read the article

  • can load data(google app enngine) from http://localhost:8100/remote_api ..

    - by zjm1126
    i can download data from gae (http://zjm1126.appspot.com/remote_api), this is code: appcfg.py download_data --application=zjm1126 --url=http://zjm1126.appspot.com/remote_api --filename=a.csv and it successful : D:\zjm_demo\app>appcfg.py download_data --application=zjm1126 --url=http://zjm1 126.appspot.com/remote_api --filename=a.csv Downloading data records. [INFO ] Logging to bulkloader-log-20100618.162421 [INFO ] Throttling transfers: [INFO ] Bandwidth: 250000 bytes/second [INFO ] HTTP connections: 8/second [INFO ] Entities inserted/fetched/modified: 20/second [INFO ] Batch Size: 10 [INFO ] Opening database: bulkloader-progress-20100618.162421.sql3 [INFO ] Opening database: bulkloader-results-20100618.162421.sql3 [INFO ] Connecting to zjm1126.appspot.com/remote_api Please enter login credentials for zjm1126.appspot.com Email: [email protected] Password for [email protected]: [INFO ] Downloading kinds: [u'LogText', u'Greeting', u'Forum', u'Thread'] .... [INFO ] Have 0 entities, 0 previously transferred [INFO ] 0 entities (8804 bytes) transferred in 11.3 seconds so i want to know can load data from 127.0.0.1 , this is my code : appcfg.py download_data --application=zjm1126 --url=http://localhost:8100/remote_api --filename=a.csv and the error is : D:\zjm_demo\app>appcfg.py download_data --application=zjm1126 --url=http://loca lhost:8100/remote_api --filename=a.csv Downloading data records. [INFO ] Logging to bulkloader-log-20100618.162325 [INFO ] Throttling transfers: [INFO ] Bandwidth: 250000 bytes/second [INFO ] HTTP connections: 8/second [INFO ] Entities inserted/fetched/modified: 20/second [INFO ] Batch Size: 10 [INFO ] Opening database: bulkloader-progress-20100618.162325.sql3 [INFO ] Opening database: bulkloader-results-20100618.162325.sql3 Please enter login credentials for localhost Email: [email protected] Password for [email protected]: [INFO ] Connecting to localhost:8100/remote_api [ERROR ] Exception during authentication Traceback (most recent call last): File "d:\Program Files\Google\google_appengine\google\appengine\tools\bulkload er.py", line 3169, in Run self.request_manager.Authenticate() File "d:\Program Files\Google\google_appengine\google\appengine\tools\bulkload er.py", line 1178, in Authenticate remote_api_stub.MaybeInvokeAuthentication() File "d:\Program Files\Google\google_appengine\google\appengine\ext\remote_api \remote_api_stub.py", line 542, in MaybeInvokeAuthentication datastore_stub._server.Send(datastore_stub._path, payload=None) File "d:\Program Files\Google\google_appengine\google\appengine\tools\appengin e_rpc.py", line 346, in Send f = self.opener.open(req) File "D:\Python25\lib\urllib2.py", line 387, in open response = meth(req, response) File "D:\Python25\lib\urllib2.py", line 498, in http_response 'http', request, response, code, msg, hdrs) File "D:\Python25\lib\urllib2.py", line 425, in error return self._call_chain(*args) File "D:\Python25\lib\urllib2.py", line 360, in _call_chain result = func(*args) File "D:\Python25\lib\urllib2.py", line 506, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 404: Not Found [INFO ] Authentication Failed so what should i do , thanks

    Read the article

  • How to finish a broken data upload to the production Google App Engine server?

    - by WooYek
    I was uploading the data to App Engine (not dev server) through loader class and remote api, and I hit the quota in the middle of a CSV file. Based on logs and progress sqllite db, how can I select remaining portion of data to be uploaded? Going through tens of records to determine which was and which was not transfered, is not appealing task, so I look for some way to limit the number of record I need to check. Here's relevant (IMO) log portion, how to interpret work item numbers? [DEBUG 2010-03-30 03:22:51,757 bulkloader.py] [Thread-2] [1041-1050] Transferred 10 entities in 3.9 seconds [DEBUG 2010-03-30 03:22:51,757 adaptive_thread_pool.py] [Thread-2] Got work item [1071-1080] <cut> [DEBUG 2010-03-30 03:23:09,194 bulkloader.py] [Thread-1] [1141-1150] Transferred 10 entities in 4.6 seconds [DEBUG 2010-03-30 03:23:09,194 adaptive_thread_pool.py] [Thread-1] Got work item [1161-1170] <cut> [DEBUG 2010-03-30 03:23:09,226 bulkloader.py] [Thread-3] [1151-1160] Transferred 10 entities in 4.2 seconds [DEBUG 2010-03-30 03:23:09,226 adaptive_thread_pool.py] [Thread-3] Got work item [1171-1180] [ERROR 2010-03-30 03:23:10,174 bulkloader.py] Retrying on non-fatal HTTP error: 503 Service Unavailable

    Read the article

  • how to format date when i load data from google-app-engine..

    - by zjm1126
    i use remote_api to load data from google-app-engine. appcfg.py download_data --config_file=helloworld/GreetingLoad.py --filename=a.csv --kind=Greeting helloworld the setting is: class AlbumExporter(bulkloader.Exporter): def __init__(self): bulkloader.Exporter.__init__(self, 'Greeting', [('author', str, None), ('content', str, None), ('date', str, None), ]) exporters = [AlbumExporter] and i download a.csv is : the date is not readable , and the date in appspot.com admin is : so how to get the full date ?? thanks i change this : class AlbumExporter(bulkloader.Exporter): def __init__(self): bulkloader.Exporter.__init__(self, 'Greeting', [('author', str, None), ('content', str, None), ('date', lambda x: datetime.datetime.strptime(x, '%m/%d/%Y').date(), None), ]) exporters = [AlbumExporter] but the error is :

    Read the article

  • appcfg.py upload_data entity kind problem

    - by Dingo
    Hi, I am developing application on app-engine-path and I would like to upload some data to datastore. For example I have a model models/places.py: class Place(db.Model): name = db.StringProperty() longitude = db.FloatProperty() latitude = db.FloatProperty() If I save this in view, kind() of this entity is "models_place". All is ok, Place.all() in view work fine. But: If I upload some next row using appcfg.py upload_data, the kind() of this entities is Place. loader.py look like this: import datetime, os, sys from google.appengine.ext import db from google.appengine.tools import bulkloader libs_path = os.path.join("/home/martin/myproject/src/") if libs_path not in sys.path: sys.path.insert(0, libs_path) from models import places class AlbumLoader(bulkloader.Loader): def __init__(self): bulkloader.Loader.__init__(self, 'Place', [('name', lambda x: x.decode('utf-8')), ('longitude', float), ('latitude', float), ]) loaders = [AlbumLoader] and command for uploading: python /usr/local/google_appengine/appcfg.py upload_data --config_file=places_loader.py --kind=models_place --filename=data/places.csv --url=http://localhost:8000/remote_api /home/martin/myproject/src/

    Read the article

  • AppEngine GeoPt Data Upload

    - by Eric Landry
    I'm writing a GAE app in Java and only using Python for the data upload. I'm trying to import a CSV file that looks like this: POSTAL_CODE_ID,PostalCode,City,Province,ProvinceCode,CityType,Latitude,Longitude 1,A0E2Z0,Monkstown,Newfoundland,NL,D,47.150300000000001,-55.299500000000002 I was able to import this file in my datastore if I import Latitude and Longitude as floats, but I'm having trouble figuring out how to import lat and lng as a GeoPt. Here is my loader.py file: import datetime from google.appengine.ext import db from google.appengine.tools import bulkloader class PostalCode(db.Model): id = db.IntegerProperty() postal_code = db.PostalAddressProperty() city = db.StringProperty() province = db.StringProperty() province_code = db.StringProperty() city_type = db.StringProperty() lat = db.FloatProperty() lng = db.FloatProperty() class PostalCodeLoader(bulkloader.Loader): def __init__(self): bulkloader.Loader.__init__(self, 'PostalCode', [('id', int), ('postal_code', str), ('city', str), ('province', str), ('province_code', str), ('city_type', str), ('lat', float), ('lng', float) ]) loaders = [PostalCodeLoader] I think that the two db.FloatProperty() lines should be replaced with a db.GeoPtProperty(), but that's where my trail ends. I'm very new to Python so any help would be greatly appreciated.

    Read the article

  • Exporting dates properly formatted on Google Appengine in Python

    - by Chris M
    I think this is right but google appengine seems to get to a certain point and cop-out; Firstly is this code actually right; and secondly is there away to skip the record if it cant output (like an ignore errors and continue)? class TrackerExporter(bulkloader.Exporter): def __init__(self): bulkloader.Exporter.__init__(self, 'SearchRec', [('__key__', lambda key:key.name(), None), ('WebSite', str, None), ('DateStamp', lambda x: datetime.datetime.strptime(x, '%d-%m-%Y').date(), None), ('IP', str, None), ('UserAgent', str, None)]) Thanks

    Read the article

  • Tools to work with App Engine data dumps

    - by Thilo
    Using the bulkloader.py utility you can download all data from your application's Datastore. It is not obvious how the data is stored, however. From the looks of it, you get a SQLite file with all data in binary format in a single table: sqlite> .tables bulkloader_database_signature result sqlite> .schema result CREATE TABLE result ( id BLOB primary key, value BLOB not null, sort_key BLOB); Are there any tools to work with this data?

    Read the article

  • GAE/J remote_api ?

    - by Thilo
    The server part of the Remote API that App Engine uses can be installed in the Java version as well, so that you can use for example the bulkloader.py utility to backup data. But is there also a Java client for accessing this API (or is that only available for Python)?

    Read the article

  • how to make data that download from google-app-engine readable..

    - by zjm1126
    i use this to download all data from my google app: i follow this article: http://code.google.com/intl/en/appengine/docs/python/tools/uploadingdata.html#Creating_Exporter_Classes and download data use this: bulkloader.py --dump --url=http://zjm1126.appspot.com/remote_api --filename=b.csv but the data is : so how to make the data readable ? thanks

    Read the article

  • GAE/J datastore backup

    - by jb
    What is the easiest way to do a GAE/J datastore backup? It looks like there is python bulkloader.py tool to do backup for Python apps, but what should I do to backup Java app? Is there any way to use python tool?

    Read the article

  • Bulkloading schema less entities on Google App Engine

    - by Rahul
    The new bulkloader added into SDK 1.3.4 works great for models that have a schema. For models inheriting db.Expando (or loosely defined schemas) i would like to understand what i would have to do to bulk upload them. I defined a custom connector, that implemented the ConnectorInterface and converted my data to the python dict required. How can i use this dict to define entities that get uploaded to the data store ? In the documentation there seems to be a post_import_function that can be used to return the entities that get uploaded. Is there an example on how this function is used ?

    Read the article

  • App Engine remote_api with OpenID

    - by hawkettc
    Hi, I've recently tried to switch my app engine app to using openID, but I'm having an issue authenticating with remote_api. The old authentication mechanism for remote_api doesn't seem to work (which makes sense) - I'm getting a 'urllib2.HTTPError: HTTP Error 302: Found', which I assume is appengine redirecting me to the openid login page I've set up. I guess I'm missing something fairly obvious. Currently my remote_api script has the following in it - remote_api_stub.ConfigureRemoteDatastore(app_id=app_id, path='/remote_api', auth_func=auth_func, servername=host, secure=secure) where auth_func is def auth_func(): return raw_input('Username:'), getpass.getpass('Password:') Any ideas what I need to supply to remote_api? I guess similar issues would be encountered with bulkloader too. Cheers, Colin

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

1