Try to fill the GAE datastore but the code consumes to much cpu time. How to optimize this?

Posted by Neverland on Stack Overflow See other posts from Stack Overflow or by Neverland
Published on 2010-03-28T12:21:49Z Indexed on 2010/03/28 12:23 UTC
Read the original article Hit count: 308

I try to get the list of images in Amazon EC2 inside the Google datastore. I want to realize this with a cron job inside the GAE.

class AmazonEC2uswest(db.Model):
    ami = db.StringProperty(required=True)
    mani = db.StringProperty()
    typ = db.StringProperty()
    arch = db.StringProperty()
    state = db.StringProperty()
    owner = db.StringProperty()

class CronAMIsAmazonUS_WEST(webapp.RequestHandler):
    def get(self):
      aws_access_key_id_admin = "<secret>"
      aws_secret_access_key_admin = "<secret>"

      conn_us_west = boto.ec2.connect_to_region('us-west-1', aws_access_key_id=aws_access_key_id_admin,
      aws_secret_access_key=aws_secret_access_key_admin, is_secure = False)

      liste_images_us_west = conn_us_west.get_all_images()

      laenge_liste_images_us_west = len(liste_images_us_west)

      for i in range(laenge_liste_images_us_west):
              datastore_uswest_AMIs = AmazonEC2uswest(ami=liste_images_us_west[i].id,
                                                      mani=str(liste_images_us_west[i].location),
                                                      typ=liste_images_us_west[i].type,
                                                      arch=liste_images_us_west[i].architecture,
                                                      state=liste_images_us_west[i].state,
                                                      owner=liste_images_us_west[i].ownerId)
              datastore_uswest_AMIs.put()

The problem: Getting the list with get_all_images() lasts only a few seconds. But writing the data to the Google datastore needs way too much CPU time.

My IBM T42p (P4M with 2GHz) needs for that piece of code approx. 1 Minute!

Is it possible to optimize my code in a way that it needs fewer CPU time?

© Stack Overflow or respective owner

Related posts about google-app-engine

Related posts about optimization