I'm developing an Asynchronous Game Server using .Net Socket Asynchronous Model( BeginAccept/EndAccept...etc.)

The problem I'm facing is described like that: When I have only one client connected, the server response time is very fast but once a second client connects, the server response time increases too much.

I've measured the time from a client sends a message to the server until it gets the reply in both cases. I found that the average time in case of one client is about 17ms and in case of 2 clients about 280ms!!!

What I really see is that: When 2 clients are connected and only one of them is moving(i.e. requesting service from the server) it is equivalently equal to the case when only one client is connected(i.e. fast response). However, when the 2 clients move at the same time(i.e. requests service from the server at the same time) their motion becomes very slow (as if the server replies each one of them in order i.e. not simultaneously).

Basically, what I am doing is that:

When a client requests a permission for motion from the server and the server grants him the request, the server then broadcasts the new position of the client to all the players. So if two clients are moving in the same time, the server is eventually trying to broadcast to both clients the new position of each of them at the same time.

EX:

• Client1 asks to go to position (2,2)
• Client2 asks to go to position (5,5)
• Server sends to each of Client1 & Client2 the same two messages:
• message1: "Client1 at (2,2)"
• message2: "Client2 at (5,5)"

I believe that the problem comes from the fact that Socket class is thread safe according MSDN documentation http://msdn.microsoft.com/en-us/library/system.net.sockets.socket.aspx. (NOT SURE THAT IT IS THE PROBLEM)

Below is the code for the server:

/// 
/// This class is responsible for handling packet receiving and sending
/// 
public class NetworkManager
{
    /// 
    /// An integer to hold the server port number to be used for the connections. Its default value is 5000.
    /// 
    private readonly int port = 5000;

    /// 
    /// hashtable contain all the clients connected to the server.
    /// key: player Id
    /// value: socket
    /// 
    private readonly Hashtable connectedClients = new Hashtable();

    /// 
    /// An event to hold the thread to wait for a new client
    /// 
    private readonly ManualResetEvent resetEvent = new ManualResetEvent(false);

    /// 
    /// keeps track of the number of the connected clients
    /// 
    private int clientCount;

    /// 
    /// The socket of the server at which the clients connect
    /// 
    private readonly Socket mainSocket =
        new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);

    /// 
    /// The socket exception that informs that a client is disconnected
    /// 
    private const int ClientDisconnectedErrorCode = 10054;

    /// 
    /// The only instance of this class. 
    /// 
    private static readonly NetworkManager networkManagerInstance = new NetworkManager();

    /// 
    /// A delegate for the new client connected event.
    /// 
    /// the sender object
    /// the event args
    public delegate void NewClientConnected(Object sender, SystemEventArgs e);

    /// 
    /// A delegate for the position update message reception.
    /// 
    /// the sender object
    /// the event args
    public delegate void PositionUpdateMessageRecieved(Object sender, PositionUpdateEventArgs e);

    /// 
    /// The event which fires when a client sends a position message 
    /// 
    public PositionUpdateMessageRecieved PositionUpdateMessageEvent
    {
        get;
        set;
    }

    /// 
    /// keeps track of the number of the connected clients
    /// 
    public int ClientCount
    {
        get
        {
            return clientCount;
        }
    }

    /// 
    /// A getter for this class instance.
    /// 
    ///  only instance.
    public static NetworkManager NetworkManagerInstance
    {
        get
        {
            return networkManagerInstance;
        }
    }

    private NetworkManager()
    {}

    /// Starts the game server and holds this thread alive
    /// 
    public void StartServer()
    {
        //Bind the mainSocket to the server IP address and port
        mainSocket.Bind(new IPEndPoint(IPAddress.Any, port));

        //The server starts to listen on the binded socket with  max connection queue //1024
        mainSocket.Listen(1024);

        //Start accepting clients asynchronously
        mainSocket.BeginAccept(OnClientConnected, null);

        //Wait until there is a client wants to connect
        resetEvent.WaitOne();
    }
    /// 
    /// Receives connections of new clients and fire the NewClientConnected event
    /// 
    private void OnClientConnected(IAsyncResult asyncResult)
    {
        Interlocked.Increment(ref clientCount);

        ClientInfo newClient = new ClientInfo
                               {
                                   WorkerSocket = mainSocket.EndAccept(asyncResult),
                                   PlayerId = clientCount
                               };
        //Add the new client to the hashtable and increment the number of clients
        connectedClients.Add(newClient.PlayerId, newClient);

        //fire the new client event informing that a new client is connected to the server
        if (NewClientEvent != null)
        {
            NewClientEvent(this, System.EventArgs.Empty);
        }

        newClient.WorkerSocket.BeginReceive(newClient.Buffer, 0, BasePacket.GetMaxPacketSize(),
            SocketFlags.None, new AsyncCallback(WaitForData), newClient);

        //Start accepting clients asynchronously again
        mainSocket.BeginAccept(OnClientConnected, null);
    }

    /// Waits for the upcoming messages from different clients and fires the proper event according to the packet type.
    /// 
    /// 
    private void WaitForData(IAsyncResult asyncResult)
    {
        ClientInfo sendingClient = null;
        try
        {
            //Take the client information from the asynchronous result resulting from the BeginReceive
            sendingClient = asyncResult.AsyncState as ClientInfo;

            // If client is disconnected, then throw a socket exception
            // with the correct error code.
            if (!IsConnected(sendingClient.WorkerSocket))
            {
                throw new SocketException(ClientDisconnectedErrorCode);
            }

            //End the pending receive request
            sendingClient.WorkerSocket.EndReceive(asyncResult);
            //Fire the appropriate event
            FireMessageTypeEvent(sendingClient.ConvertBytesToPacket() as BasePacket);

            // Begin receiving data from this client
            sendingClient.WorkerSocket.BeginReceive(sendingClient.Buffer, 0, BasePacket.GetMaxPacketSize(),
                SocketFlags.None, new AsyncCallback(WaitForData), sendingClient);
        }
        catch (SocketException e)
        {
            if (e.ErrorCode == ClientDisconnectedErrorCode)
            {
                // Close the socket.
                if (sendingClient.WorkerSocket != null)
                {
                    sendingClient.WorkerSocket.Close();
                    sendingClient.WorkerSocket = null;
                }

                // Remove it from the hash table.
                connectedClients.Remove(sendingClient.PlayerId);

                if (ClientDisconnectedEvent != null)
                {
                    ClientDisconnectedEvent(this, new ClientDisconnectedEventArgs(sendingClient.PlayerId));
                }
            }
        }
        catch (Exception e)
        {
            // Begin receiving data from this client
            sendingClient.WorkerSocket.BeginReceive(sendingClient.Buffer, 0, BasePacket.GetMaxPacketSize(),
                SocketFlags.None, new AsyncCallback(WaitForData), sendingClient);
        }
    }
    /// 
    /// Broadcasts the input message to all the connected clients
    /// 
    /// 
    public void BroadcastMessage(BasePacket message)
    {
        byte[] bytes = message.ConvertToBytes();
        foreach (ClientInfo client in connectedClients.Values)
        {
            client.WorkerSocket.BeginSend(bytes, 0, bytes.Length, SocketFlags.None, SendAsync, client);
        }
    }

    /// 
    /// Sends the input message to the client specified by his ID.
    /// 
    ///
    /// The message to be sent.
    /// The id of the client to receive the message.
    public void SendToClient(BasePacket message, int id)
    {

        byte[] bytes = message.ConvertToBytes();
        (connectedClients[id] as ClientInfo).WorkerSocket.BeginSend(bytes, 0, bytes.Length,
            SocketFlags.None, SendAsync, connectedClients[id]);
    }

    private void SendAsync(IAsyncResult asyncResult)
    {
        ClientInfo currentClient = (ClientInfo)asyncResult.AsyncState;
        currentClient.WorkerSocket.EndSend(asyncResult);
    }

    /// Fires the event depending on the type of received packet
    /// 
    /// The received packet.
    void FireMessageTypeEvent(BasePacket packet)
    {

        switch (packet.MessageType)
        {
            case MessageType.PositionUpdateMessage:
                if (PositionUpdateMessageEvent != null)
                {
                    PositionUpdateMessageEvent(this, new PositionUpdateEventArgs(packet as PositionUpdatePacket));
                }
                break;
        }

    }
}

The events fired are handled in a different class, here are the event handling code for the PositionUpdateMessage (Other handlers are irrelevant):

    private readonly Hashtable onlinePlayers = new Hashtable();

    /// 
    /// Constructor that creates a new instance of the GameController class.
    /// 
    private GameController()
    {
        //Start the server
        server = new Thread(networkManager.StartServer);
        server.Start();
        //Create an event handler for the NewClientEvent of networkManager

        networkManager.PositionUpdateMessageEvent += OnPositionUpdateMessageReceived;
    }

    /// 
    /// this event handler is called when a client asks for movement.
    /// 
    private void OnPositionUpdateMessageReceived(object sender, PositionUpdateEventArgs e)
    {
        Point currentLocation = ((PlayerData)onlinePlayers[e.PositionUpdatePacket.PlayerId]).Position;
        Point locationRequested = e.PositionUpdatePacket.Position;


       ((PlayerData)onlinePlayers[e.PositionUpdatePacket.PlayerId]).Position = locationRequested;

            // Broadcast the new position
            networkManager.BroadcastMessage(new PositionUpdatePacket
                                            {
                                                Position = locationRequested,
                                                PlayerId = e.PositionUpdatePacket.PlayerId
                                            });


        }