Even distribution through a chain of resources

Posted by ClosetGeek on Programmers See other posts from Programmers or by ClosetGeek
Published on 2014-02-04T23:05:18Z Indexed on 2014/06/10 3:41 UTC
Read the original article Hit count: 181

I'm working on an algorithm which routes tasks through a chain of distributed resources based on a hash (or random number).

For example, say you have 10 gateways into a service which distribute tasks to 1000 handlers through 100 queues. 10,000 connected clients are expected to be connected to gateways at any given time (numbers are very general to keep it simple).

Thats

  • 10,000 clients
  • 10 gateways (producers)
  • 100 queues
  • 1000 workers/handlers (consumers)

The flow of each task is client->gateway->queue->worker

Each client will have it's own hash/number which is used to route each task from the client to the same worker each time, with each task going through the same gateway and queue each time. Yet the algorithm handles distribution evenly, meaning each gateway, queue, and worker will have an even workload.

My question is what exactly would this be called? Does such a thing already exist? This started off as a DHT, but I realized that DHTs can't do exactly what I need, so I started from scratch.

© Programmers or respective owner

Related posts about algorithms

Related posts about scalability