Is it possible to zoom to a certain point on screen by modifying the field of view and rotating the view of the camera as to keep that point/object in the same place on screen while zooming ? Changing the camera position is not allowed.

I projected the 3D pos of the object on screen and remembered it.

Then on each frame I calculate the direction to it in camera space and then I construct a rotation matrix to align this direction to Z axis (in cam space).

After this, I calculate the direction from the camera to the object in world space and transform this vector with the matrix I obtained earlier and then use this final vector as the camera's new direction.

And it's actually "kinda working", the problem is that it is more/less off than the camera's rotation before starting to zoom depending on the area you are trying to zoom in (larger error on edges/corners).

It looks acceptable, but I'm not settling for only this. Any suggestions/resources for doing this technique perfectly? If some of you want to explain the math in detail, be my guest, I can understand these things well.