Cloud computing gives us more flexibility on the computing resource, we can provision and deploy an application or service with multiple instances over multiple machines. With the increment of the service instances, how to balance the incoming message and workload would become a new challenge.

Currently there are two approaches we can use to pass the incoming messages to the service instances, I would like call them dispatcher mode and pulling mode.

Dispatcher Mode

The dispatcher mode introduces a role which takes the responsible to find the best service instance to process the request. The image below describes the sharp of this mode.

There are four clients communicate with the service through the underlying transportation. For example, if we are using HTTP the clients might be connecting to the same service URL. On the server side there’s a dispatcher listening on this URL and try to retrieve all messages. When a message came in, the dispatcher will find a proper service instance to process it. There are three mechanism to find the instance:

• Round-robin: Dispatcher will always send the message to the next instance. For example, if the dispatcher sent the message to instance 2, then the next message will be sent to instance 3, regardless if instance 3 is busy or not at that moment.
• Random: Dispatcher will find a service instance randomly, and same as the round-robin mode it regardless if the instance is busy or not.
• Sticky: Dispatcher will send all related messages to the same service instance. This approach always being used if the service methods are state-ful or session-ful.

But as you can see, all of these approaches are not really load balanced. The clients will send messages at any time, and each message might take different process duration on the server side. This means in some cases, some of the service instances are very busy while others are almost idle. For example, if we were using round-robin mode, it could be happened that most of the simple task messages were passed to instance 1 while the complex ones were sent to instance 3, even though instance 1 should be idle.

This brings some problem in our architecture. The first one is that, the response to the clients might be longer than it should be. As it’s shown in the figure above, message 6 and 9 can be processed by instance 1 or instance 2, but in reality they were dispatched to the busy instance 3 since the dispatcher and round-robin mode.

Secondly, if there are many requests came from the clients in a very short period, service instances might be filled by tons of pending tasks and some instances might be crashed.

Third, if we are using some cloud platform to host our service instances, for example the Windows Azure, the computing resource is billed by service deployment period instead of the actual CPU usage. This means if any service instance is idle it is wasting our money!

Last one, the dispatcher would be the bottleneck of our system since all incoming messages must be routed by the dispatcher. If we are using HTTP or TCP as the transport, the dispatcher would be a network load balance. If we wants more capacity, we have to scale-up, or buy a hardware load balance which is very expensive, as well as scaling-out the service instances.

Pulling Mode

Pulling mode doesn’t need a dispatcher to route the messages. All service instances are listening to the same transport and try to retrieve the next proper message to process if they are idle.

Since there is no dispatcher in pulling mode, it requires some features on the transportation.

• The transportation must support multiple client connection and server listening. HTTP and TCP doesn’t allow multiple clients are listening on the same address and port, so it cannot be used in pulling mode directly.
• All messages in the transportation must be FIFO, which means the old message must be received before the new one.
• Message selection would be a plus on the transportation. This means both service and client can specify some selection criteria and just receive some specified kinds of messages. This feature is not mandatory but would be very useful when implementing the request reply and duplex WCF channel modes. Otherwise we must have a memory dictionary to store the reply messages. I will explain more about this in the following articles.

Message bus, or the message queue would be best candidate as the transportation when using the pulling mode. First, it allows multiple application to listen on the same queue, and it’s FIFO. Some of the message bus also support the message selection, such as TIBCO EMS, RabbitMQ. Some others provide in memory dictionary which can store the reply messages, for example the Redis.

The principle of pulling mode is to let the service instances self-managed. This means each instance will try to retrieve the next pending incoming message if they finished the current task. This gives us more benefit and can solve the problems we met with in the dispatcher mode.

• The incoming message will be received to the best instance to process, which means this will be very balanced. And it will not happen that some instances are busy while other are idle, since the idle one will retrieve more tasks to make them busy.
• Since all instances are try their best to be busy we can use less instances than dispatcher mode, which more cost effective.
• Since there’s no dispatcher in the system, there is no bottleneck.
• When we introduced more service instances, in dispatcher mode we have to change something to let the dispatcher know the new instances. But in pulling mode since all service instance are self-managed, there no extra change at all.
• If there are many incoming messages, since the message bus can queue them in the transportation, service instances would not be crashed.

All above are the benefits using the pulling mode, but it will introduce some problem as well.

• The process tracking and debugging become more difficult. Since the service instances are self-managed, we cannot know which instance will process the message. So we need more information to support debug and track.
• Real-time response may not be supported. All service instances will process the next message after the current one has done, if we have some real-time request this may not be a good solution.

Compare with the Pros and Cons above, the pulling mode would a better solution for the distributed system architecture. Because what we need more is the scalability, cost-effect and the self-management.

WCF and WCF Transport Extensibility

Windows Communication Foundation (WCF) is a framework for building service-oriented applications. In the .NET world WCF is the best way to implement the service. In this series I’m going to demonstrate how to implement the pulling mode on top of a message bus by extending the WCF.

I don’t want to deep into every related field in WCF but will highlight its transport extensibility. When we implemented an RPC foundation there are many aspects we need to deal with, for example the message encoding, encryption, authentication and message sending and receiving. In WCF, each aspect is represented by a channel. A message will be passed through all necessary channels and finally send to the underlying transportation. And on the other side the message will be received from the transport and though the same channels until the business logic.

This mode is called “Channel Stack” in WCF, and the last channel in the channel stack must always be a transport channel, which takes the responsible for sending and receiving the messages. As we are going to implement the WCF over message bus and implement the pulling mode scaling-out solution, we need to create our own transport channel so that the client and service can exchange messages over our bus. Before we deep into the transport channel, let’s have a look on the message exchange patterns that WCF defines.

Message exchange pattern (MEP) defines how client and service exchange the messages over the transportation. WCF defines 3 basic MEPs which are datagram, Request-Reply and Duplex.

• Datagram: Also known as one-way, or fire-forgot mode. The message sent from the client to the service, and no need any reply from the service. The client doesn’t care about the message result at all.
• Request-Reply: Very common used pattern. The client send the request message to the service and wait until the reply message comes from the service.
• Duplex: The client sent message to the service, when the service processing the message it can callback to the client. When callback the service would be like a client while the client would be like a service.

In WCF, each MEP represent some channels associated.

And the channels are created by ChannelListener on the server side, and ChannelFactory on the client side. The ChannelListener and ChannelFactory are created by the TransportBindingElement. The TransportBindingElement is created by the Binding, which can be defined as a new binding or from a custom binding.

For more information about the transport channel mode, please refer to the MSDN document.

The figure below shows the transport channel objects when using the request-reply MEP.

And this is the datagram MEP.

And this is the duplex MEP.

After investigated the WCF transport architecture, channel mode and MEP, we finally identified what we should do to extend our message bus based transport layer. They are:

• Binding: (Optional) Defines the channel elements in the channel stack and added our transport binding element at the bottom of the stack. But we can use the build-in CustomBinding as well.
• TransportBindingElement: Defines which MEP is supported in our transport and create the related ChannelListener and ChannelFactory. This also defines the scheme of the endpoint if using this transport.
• ChannelListener: Create the server side channel based on the MEP it’s. We can have one ChannelListener to create channels for all supported MEPs, or we can have ChannelListener for each MEP. In this series I will use the second approach.
• ChannelFactory: Create the client side channel based on the MEP it’s. We can have one ChannelFactory to create channels for all supported MEPs, or we can have ChannelFactory for each MEP. In this series I will use the second approach.
• Channels: Based on the MEPs we want to support, we need to implement the channels accordingly. For example, if we want our transport support Request-Reply mode we should implement IRequestChannel and IReplyChannel. In this series I will implement all 3 MEPs listed above one by one.
• Scaffold: In order to make our transport extension works we also need to implement some scaffold stuff. For example we need some classes to send and receive message though out message bus. We also need some codes to read and write the WCF message, etc.. These are not necessary but would be very useful in our example.

Message Bus

There is only one thing remained before we can begin to implement our scaling-out support WCF transport, which is the message bus. As I mentioned above, the message bus must have some features to fulfill all the WCF MEPs. In my company we will be using TIBCO EMS, which is an enterprise message bus product. And I have said before we can use any message bus production if it’s satisfied with our requests.

Here I would like to introduce an interface to separate the message bus from the WCF. This allows us to implement the bus operations by any kinds bus we are going to use. The interface would be like this.

   1: public interface IBus : IDisposable

   2: {

   3:     string SendRequest(string message, bool fromClient, string from, string to = null);

   4:  

   5:     void SendReply(string message, bool fromClient, string replyTo);

   6:  

   7:     BusMessage Receive(bool fromClient, string replyTo);

   8: }

There are only three methods for the bus interface. Let me explain one by one.

The SendRequest method takes the responsible for sending the request message into the bus. The parameters description are:

• message: The WCF message content.
• fromClient: Indicates if this message was came from the client.
• from: The channel ID that this message was sent from. The channel ID will be generated when any kinds of channel was created, which will be explained in the following articles.
• to: The channel ID that this message should be received. In Request-Reply and Duplex MEP this is necessary since the reply message must be received by the channel which sent the related request message.

The SendReply method takes the responsible for sending the reply message. It’s very similar as the previous one but no “from” parameter. This is because it’s no need to reply a reply message again in any MEPs.

The Receive method takes the responsible for waiting for a incoming message, includes the request message and specified reply message. It returned a BusMessage object, which contains some information about the channel information. The code of the BusMessage class is

   1: public class BusMessage

   2: {

   3:     public string MessageID { get; private set; }

   4:     public string From { get; private set; }

   5:     public string ReplyTo { get; private set; }

   6:     public string Content { get; private set; }

   7:  

   8:     public BusMessage(string messageId, string fromChannelId, string replyToChannelId, string content)

   9:     {

  10:         MessageID = messageId;

  11:         From = fromChannelId;

  12:         ReplyTo = replyToChannelId;

  13:         Content = content;

  14:     }

  15: }

Now let’s implement a message bus based on the IBus interface. Since I don’t want you to buy and install the TIBCO EMS or any other message bus products, I will implement an in process memory bus. This bus is only for test and sample purpose. It can only be used if the service and client are in the same process. Very straightforward.

   1: public class InProcMessageBus : IBus

   2: {

   3:     private readonly ConcurrentDictionary<Guid, InProcMessageEntity> _queue;

   4:     private readonly object _lock;

   5:  

   6:     public InProcMessageBus()

   7:     {

   8:         _queue = new ConcurrentDictionary<Guid, InProcMessageEntity>();

   9:         _lock = new object();

  10:     }

  11:  

  12:     public string SendRequest(string message, bool fromClient, string from, string to = null)

  13:     {

  14:         var entity = new InProcMessageEntity(message, fromClient, from, to);

  15:         _queue.TryAdd(entity.ID, entity);

  16:         return entity.ID.ToString();

  17:     }

  18:  

  19:     public void SendReply(string message, bool fromClient, string replyTo)

  20:     {

  21:         var entity = new InProcMessageEntity(message, fromClient, null, replyTo);

  22:         _queue.TryAdd(entity.ID, entity);

  23:     }

  24:  

  25:     public BusMessage Receive(bool fromClient, string replyTo)

  26:     {

  27:         InProcMessageEntity e = null;

  28:         while (true)

  29:         {

  30:             lock (_lock)

  31:             {

  32:                 var entity = _queue

  33:                     .Where(kvp => kvp.Value.FromClient == fromClient && (kvp.Value.To == replyTo || string.IsNullOrWhiteSpace(kvp.Value.To)))

  34:                     .FirstOrDefault();

  35:                 if (entity.Key != Guid.Empty && entity.Value != null)

  36:                 {

  37:                     _queue.TryRemove(entity.Key, out e);

  38:                 }

  39:             }

  40:             if (e == null)

  41:             {

  42:                 Thread.Sleep(100);

  43:             }

  44:             else

  45:             {

  46:                 return new BusMessage(e.ID.ToString(), e.From, e.To, e.Content);

  47:             }

  48:         }

  49:     }

  50:  

  51:     public void Dispose()

  52:     {

  53:     }

  54: }

The InProcMessageBus stores the messages in the objects of InProcMessageEntity, which can take some extra information beside the WCF message itself.

   1: public class InProcMessageEntity

   2: {

   3:     public Guid ID { get; set; }

   4:     public string Content { get; set; }

   5:     public bool FromClient { get; set; }

   6:     public string From { get; set; }

   7:     public string To { get; set; }

   8:  

   9:     public InProcMessageEntity()

  10:         : this(string.Empty, false, string.Empty, string.Empty)

  11:     {

  12:     }

  13:  

  14:     public InProcMessageEntity(string content, bool fromClient, string from, string to)

  15:     {

  16:         ID = Guid.NewGuid();

  17:         Content = content;

  18:         FromClient = fromClient;

  19:         From = from;

  20:         To = to;

  21:     }

  22: }

Summary

OK, now I have all necessary stuff ready. The next step would be implementing our WCF message bus transport extension.

In this post I described two scaling-out approaches on the service side especially if we are using the cloud platform: dispatcher mode and pulling mode. And I compared the Pros and Cons of them. Then I introduced the WCF channel stack, channel mode and the transport extension part, and identified what we should do to create our own WCF transport extension, to let our WCF services using pulling mode based on a message bus. And finally I provided some classes that need to be used in the future posts that working against an in process memory message bus, for the demonstration purpose only.

In the next post I will begin to implement the transport extension step by step.

Hope this helps,

Shaun