I'm sure there's a lot of questions on here similar to this, and I've been reading them, but I still feel this warrants a new question.

I want nightly, incremental backups (full disk images would waste a lot of space - unless compressed somehow.) Preferably rotating or deleting old backups when running out of space or after a fixed number of backups. I want to be able to quickly and painlessly restore my system from these backups. This is my first time running ubuntu as my main development machine and I know from my experience with it as a server and in virtual machines that I regularly manage to make it unbootable or damage it to the point of being unable to rescue it. So how would you recommend I do this? There are so many options out there I really don't know where to start.

There seems to be a vocal school of thought that it's sufficient to backup your home directory and the list of installed packages from the package manager. I've already installed lots of things from source, or outside of the package manager (development tools, ides, compilers, graphics drivers, etc.) So at the very least, if I do not back up the operating system itself I need to grab all config files, all program binaries, all created but required files, etc. I'd rather backup too much than too little - an ubuntu install is tiny anyway. Also this drastically reduces the restore time, which would cost me more in my time than the extra storage space.

I tried using Deja Dup to backup the root partition, excluding some things like /mnt /media /dev /proc etc. Although many websites assured me you can backup a running linux system this way - that seems to be false as it complained that it could not backup the following files:

/boot/System.map-3.0.0-17-generic
/boot/System.map-3.2.0-22-generic
/boot/vmcoreinfo-3.0.0-17-generic
/boot/vmlinuz-3.0.0-17-generic
/boot/vmlinuz-3.2.0-22-generic
/etc/.pwd.lock
/etc/NetworkManager/system-connections/LAN Connection
/etc/apparmor.d/cache/lightdm-guest-session
/etc/apparmor.d/cache/sbin.dhclient
/etc/apparmor.d/cache/usr.bin.evince
/etc/apparmor.d/cache/usr.lib.telepathy
/etc/apparmor.d/cache/usr.sbin.cupsd
/etc/apparmor.d/cache/usr.sbin.tcpdump
/etc/apt/trustdb.gpg
/etc/at.deny
/etc/ati/inst_path_default
/etc/ati/inst_path_override
/etc/chatscripts
/etc/cups/ssl
/etc/cups/subscriptions.conf
/etc/cups/subscriptions.conf.O
/etc/default/cacerts
/etc/fuse.conf
/etc/group-
/etc/gshadow
/etc/gshadow-
/etc/mtab.fuselock
/etc/passwd-
/etc/ppp/chap-secrets
/etc/ppp/pap-secrets
/etc/ppp/peers
/etc/security/opasswd
/etc/shadow
/etc/shadow-
/etc/ssl/private
/etc/sudoers
/etc/sudoers.d/README
/etc/ufw/after.rules
/etc/ufw/after6.rules
/etc/ufw/before.rules
/etc/ufw/before6.rules
/lib/ufw/user.rules
/lib/ufw/user6.rules
/lost+found
/root
/run/crond.reboot
/run/cups/certs
/run/lightdm
/run/lock/whoopsie/lock
/run/udisks
/var/backups/group.bak
/var/backups/gshadow.bak
/var/backups/passwd.bak
/var/backups/shadow.bak
/var/cache/apt/archives/lock
/var/cache/cups/job.cache
/var/cache/cups/job.cache.O
/var/cache/cups/ppds.dat
/var/cache/debconf/passwords.dat
/var/cache/ldconfig
/var/cache/lightdm/dmrc
/var/crash/_usr_lib_x86_64-linux-gnu_colord_colord.102.crash
/var/lib/apt/lists/lock
/var/lib/dpkg/lock
/var/lib/dpkg/triggers/Lock
/var/lib/lightdm
/var/lib/mlocate/mlocate.db
/var/lib/polkit-1
/var/lib/sudo
/var/lib/urandom/random-seed
/var/lib/ureadahead/pack
/var/lib/ureadahead/run.pack
/var/log/btmp
/var/log/installer/casper.log
/var/log/installer/debug
/var/log/installer/partman
/var/log/installer/syslog
/var/log/installer/version
/var/log/lightdm/lightdm.log
/var/log/lightdm/x-0-greeter.log
/var/log/lightdm/x-0.log
/var/log/speech-dispatcher
/var/log/upstart/alsa-restore.log
/var/log/upstart/alsa-restore.log.1.gz
/var/log/upstart/console-setup.log
/var/log/upstart/console-setup.log.1.gz
/var/log/upstart/container-detect.log
/var/log/upstart/container-detect.log.1.gz
/var/log/upstart/hybrid-gfx.log
/var/log/upstart/hybrid-gfx.log.1.gz
/var/log/upstart/modemmanager.log
/var/log/upstart/modemmanager.log.1.gz
/var/log/upstart/module-init-tools.log
/var/log/upstart/module-init-tools.log.1.gz
/var/log/upstart/procps-static-network-up.log
/var/log/upstart/procps-static-network-up.log.1.gz
/var/log/upstart/procps-virtual-filesystems.log
/var/log/upstart/procps-virtual-filesystems.log.1.gz
/var/log/upstart/rsyslog.log
/var/log/upstart/rsyslog.log.1.gz
/var/log/upstart/ureadahead.log
/var/log/upstart/ureadahead.log.1.gz
/var/spool/anacron/cron.daily
/var/spool/anacron/cron.monthly
/var/spool/anacron/cron.weekly
/var/spool/cron/atjobs
/var/spool/cron/atspool
/var/spool/cron/crontabs
/var/spool/cups