If I have a server A into which I can login with my ssh key and I have the ability to "sudo su - otheruser", I lose key forwarding, because the env variables are removed and the socket is only readable by my original user. Is there a way I can bridge the key forwarding through the "sudo su - otheruser", so I can do stuff on a server B with my forwarded key (git clone and rsync in my case)?

The only way I can think of is adding my key to authorized_keys of otheruser and "ssh otheruser@localhost", but that's cumbersome to do for every user and server combination I may have.