I need to move hosting providers for four or five domains and for several reasons I'm considering a Linux VPS rather than staying with my current shared, managed hosting provider. The only thing that's stopping me is email. I have lots of experience running and maintaining Apache, but none with email servers. Based on some research, if I want to keep what I've using now, it looks like I'd be going with Postfix and Dovecot, and probably Exim and SpamAssassin. I have no problem performing regular maintenance and watching for security updates, but I don't want to bite off more than I can chew.

For someone new to email services, how hard is it to set up an email server that is externally accessible (via SMTP and POP3, not IMAP), available over SSL/TLS and reasonably reliable for multiple domains? How much of a time commitment is it to maintain one?