I have haproxy listening on several ports and pointed at several backend servers.

Ideally, I would like it so that repeated communications to the same port get pointed at the same backend.

"balance source" isn't workable because often requests come from the same source.

Is this doable?

I'm also open to non-haproxy solutions. The protocol being used isn't important but is definitely not http. Just assume its ssh and you shouldn't go wrong.