Let's say you have 2 servers each with 8 CPU cores each.

The servers each run 8 network services that each host an arbitrary number of long-lived TCP/IP client connections.

Clients send messages to the services.

The services do something based on the messages, and potentially notify N>1 of the clients of state changes.

Sure, it sounds like a botnet but it isn't. Consider how IRC works with c2s and s2s connections and s2s message relaying.

• The servers are in the same data center.
• The servers can communicate over a private VLAN @1GigE.
• Messages are < 1KB in size.

How would you coordinate which services on which host should receive and relay messages to connected clients for state change messages?

There's an infinite number of ways to solve this problem efficiently.

• AMQP (RabbitMQ, ZeroMQ, etc.)
• Spread Toolkit
• N^2 connections between allservices (bad)
• Heck, even run IRC!
• ...

I'm looking for a solution that:

• perhaps exploits the fact that there's only a small closed cluster
• is easy to admin
• scales well
• is "dumb" (no weird edge cases)

What are your experiences?

What do you recommend?

Thanks!