Installing LibreOffice manually on Ubuntu 12.04 works great following the advice in this answer: http://askubuntu.com/a/182602/103893

It basically is a matter of downloading the most recent release from the LibreOffice site and running sudo apt-get remove libreoffice-core and sudo dpkg -i *.deb.

Doing the above I installed LibreOffice 3.6.3 and it works great.

But what is the best way to uninstall this new version of LibreOffice (for when the next release is coming along)?

Trying sudo apt-get remove libreoffice-core will not do anything, and sudo apt-get remove libreoffice3.6 does not seem to remove all packages installed with dpkg...?