Emptying the datastore in GAE

Posted by colwilson on Stack Overflow See other posts from Stack Overflow or by colwilson
Published on 2010-01-27T14:52:36Z Indexed on 2010/04/24 16:03 UTC
Read the original article Hit count: 281

I know what you're thinking, 'O not that again!', but here we are since Google have not yet provided a simpler method.

I have been using a queue based solution which worked fine:

import datetime from models import *

DELETABLE_MODELS = [Alpha, Beta, AlphaBeta]

def initiate_purge():
    for e in config.DELETABLE_MODELS:
        deferred.defer(delete_entities, e, 'purging', _queue = 'purging')

class NotEmptyException(Exception): pass

def delete_entities(e, queue):
    try:
        q = e.all(keys_only=True)
        db.delete(q.fetch(200))
        ct = q.count(1)
        if ct > 0:
            raise NotEmptyException('there are still entities to be deleted')
        else:
            logging.info('processing %s completed' % queue)
    except Exception, err:
        deferred.defer(delete_entities, e, then, queue, _queue = queue)
        logging.info('processing %s deferred: %s' % (queue, err))

All this does is queue a request to delete some data (once for each class) and then if the queued process either fails or knows there is still some stuff to delete, it re-queues itself.

This beats the heck out of hitting the refresh on a browser for 10 minutes.

However, I'm having trouble deleting AlphaBeta entities, there are always a few left at the end. I think because it contains Reference Properties:

class AlphaBeta(db.Model):
    alpha = db.ReferenceProperty(Alpha, required=True, collection_name='betas')
    beta = db.ReferenceProperty(Beta, required=True, collection_name='alphas')

I have tried deleting the indexes relating to these entity types, but that did not make any difference.

Any advice would be appreciated please.

© Stack Overflow or respective owner

Related posts about python

Related posts about google-datastore