I just completed a complex piece of code. It works to spec, it meets performance requirements etc etc but I feel a bit anxious about it and am considering rewriting and/or refactoring it. Should I do this (spending time that could otherwise be spent on features that users will actually notice)?

The reasons I feel anxious about the code are:

• The class hierarchy is complex and not obvious
• Some classes don't have a well defined purpose (they do a number of unrelated things)
• Some classes use others internals (they're declared as friend classes) to bypass the layers of abstraction for performance, but I feel they break encapsulation by doing this
• Some classes leak implementation details (eg, I changed a map to a hash map earlier and found myself having to modify code in other source files to make the change work)
• My memory management/pooling system is kinda clunky and less-than transparent

They look like excellent reasons to refactor and clean code, aiding future maintenance and extension, but could be quite time consuming. Also, I'll never be perfectly happy with any code I write anyway...

So, what does stackoverflow think? Clean code or work on features?