For a 2D board game I'd like to move and rotate an orthogonal camera in coordinates given in a reference system (window space), but simply can't get it to work.

The idea is that the user can drag the camera over a surface, rotate and scale it. Rotation and scaling should always be around the center of the current viewport.

The camera is set up as:

gl.glMatrixMode(GL2.GL_PROJECTION);
gl.glLoadIdentity();
gl.glOrtho(-width/2, width/2, -height/2, height/2, nearPlane, farPlane);

where width and height are equal to the viewport's width and height, so that 1 unit is one pixel when no zoom is applied.

Since these transformations usually mean (scaling and) translating the world, then rotating it, the implementation is:

gl.glMatrixMode(GL2.GL_MODELVIEW);
gl.glLoadIdentity();

gl.glRotatef(rotation, 0, 0, 1); // e.g. 45°
gl.glTranslatef(x, y, 0); // e.g. +10 for 10px right, -2 for 2px down
gl.glScalef(zoomFactor, zoomFactor, zoomFactor); // e.g. scale by 1.5

That however has the nasty side effect that translations are transformed as well, that is applied in world coordinates. If I rotate around 90° and translate again, X and Y axis are swapped.

If I reorder the transformations so they read

gl.glTranslatef(x, y, 0);
gl.glScalef(zoomFactor, zoomFactor, zoomFactor);
gl.glRotatef(rotation, 0, 0, 1);

the translation will be applied correctly (in reference space, so translation along x always visually moves the camera sideways) but rotation and scaling are now performed around origin.

It shouldn't be too hard, so what is it I'm missing?