I'm using JDO 2.3 on app engine. I was using the Master/Slave datastore for local testing and recently switched over to using the HRD datastore for local testing, and parts of my app are breaking (which is to be expected). One part of the app that's breaking is where it sends a lot of writes quickly - that is because of the 1-second limit thing, it's failing with a concurrent modification exception.

Okay, so that's also to be expected, so I have the browser retry the writes again later when they fail (maybe not the best hack but I'm just trying to get it working quickly).

But a weird thing is happening. Some of the writes which should be succeeding (the ones that DON'T get the concurrent modification exception) are also failing, even though the commit phase completes and the request returns my success code. I can see from the log that the retried requests are working okay, but these other requests that seem to have committed on the first try are, I guess, never "applied." But from what I read about the Apply phase, writing again to that same entity should force the apply... but it doesn't.

Code follows. Some things to note:

1. I am attempting to use automatic JDO caching. So this is where JDO uses memcache under the covers. This doesn't actually work unless you wrap everything in a transaction.
2. all the requests are doing is reading a string out of an entity, modifying part of the string, and saving that string back to the entity. If these requests weren't in transactions, you'd of course have the "dirty read" problem. But with transactions, isolation is supposed to be at the level of "serializable" so I don't see what's happening here.
3. the entity being modified is a root entity (not in a group)
4. I have cross-group transactions enabled
5. Another weird thing is happening. If the concurrent modification thing happens, and I subsequently edit more than 5 more entities (this is the max for cross-group transactions), then nothing happens right away, but when I stop and restart the server I get "IllegalArgumentException: operating on too many entity groups in a single transaction".

Could it be possible that the PMF is returning the same PersistenceManager every time, or the PM is reusing the same transaction every time? I don't see how I could possibly get the above error otherwise. The code inside the transaction just edits one root entity. I can't think of any other way that GAE would give me the "too many entity groups" error.

The relevant code (this is a simplified version)

PersistenceManager pm = PMF.getManager();
Transaction tx = pm.currentTransaction();
String responsetext = "";
try {
  tx.begin();
  // I have extra calls to "makePersistent" because I found that relying
  // on pm.close didn't always write the objects to cache, maybe that
  // was only a DataNucleus 1.x issue though
  Key userkey = obtainUserKeyFromCookie();
  User u = pm.getObjectById(User.class, userkey);
  pm.makePersistent(u); // to make sure it gets cached for next time
  Key mapkey = obtainMapKeyFromQueryString();
  // this is NOT a java.util.Map, just FYI
  Map currentmap = pm.getObjectById(Map.class, mapkey);
  Text mapData = currentmap.getMapData(); // mapData is JSON stored in the entity
  Text newMapData = parseModifyAndReturn(mapData); // transform the map
  currentmap.setMapData(newMapData); // mutate the Map object
  pm.makePersistent(currentmap); // make sure to persist so there is a cache hit
  tx.commit();
  responsetext = "OK";
} catch (JDOCanRetryException jdoe) {
  // log jdoe
  responsetext = "RETRY";
} catch (Exception e) {
  // log e
  responsetext = "ERROR";
} finally {
  if (tx.isActive()) {
    tx.rollback();
  }
  pm.close();
}
resp.getWriter().println(responsetext);

EDIT: so I have verified that it fails after exactly 5 transactions. Here's what I do: I create a Foo (root entity), do a bunch of concurrent operations on that Foo, and some fail and get retried, and some commit but don't apply (as described above). Then, I start creating more Foos, and do a few operations on those new Foos. If I only create four Foos, stopping and restarting app engine does NOT give me the IllegalArgumentException. However if I create five Foos (which is the limit for cross-group transactions), then when I stop and restart app engine, I do get the exception. So it seems that somehow these new Foos I am creating are counting toward the limit of 5 max entities per transaction, even though they are supposed to be handled by separate transactions. It's as if a transaction is still open and is being reused by the servlet when it handles the new requests for the 2nd through 5th Foos.

EDIT2: it looks like the IllegalArgument thing is independent of the other bug. In other words, it always happens when I create five Foos, even if I don't get the concurrent modification exception. I don't know if it's a symptom of the same problem or if it's unrelated.

EDIT3: I found out what was causing the (unrelated) IllegalArgumentException, it was a dumb mistake on my part. But the other issue is still happening.

EDIT4: added pseudocode for the datastore access

EDIT5: I am pretty sure I know why this is happening, but I will still award the bounty to anyone who can confirm it.

Basically, I think the problem is that transactions are not really implemented in the local version of the datastore. References:

https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-java/gVMS1dFSpcU https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-java/deGasFdIO-M https://groups.google.com/forum/?hl=en&fromgroups=#!msg/google-appengine-java/4YuNb6TVD6I/gSttMmHYwo0J

Because transactions are not implemented, rollback is essentially a no-op. Therefore, I get a dirty read when two transactions try to modify the record at the same time. In other words, A reads the data and B reads the data at the same time. A attempts to modify the data, and B attempts to modify a different part of the data. A writes to the datastore, then B writes, obliterating A's changes. Then B is "rolled back" by app engine, but since rollbacks are a no-op when running on the local datastore, B's changes stay, and A's do not. Meanwhile, since B is the thread that threw the exception, the client retries B, but does not retry A (since A was supposedly the transaction that succeeded).