I have an application which retrieves and caches the results of a clients query. The client then requests different chunks of data and the application sends the relevant results and removes them from the cache.

A new requirement for this application is that there needs to be a run-time configurable maximum number of results which may be cached. I've taken the naive approach and implemented this by using a counter under a lock which is incremented every time a result is cached and decremented whenever a result is removed from the cache.

Unfortunately, this has drastically reduced the applications performance when processing a large number of concurrent requests. I have tried both a critical section lock and spin-lock; the performance improves a bit with a spin-lock, but is still unacceptably slow. Is there a better way to solve this problem which may improve performance?

Right now I have a thread pool that services requests and each request is tied to a Request object which stores that cached results for that particular request. Here is a simplified pseudo code version of my current implementation:

void ResultCallback( Result result, Request *request )
{
    lock totalResultsCached 
    lock cachedLimit

    if( totalResultsCached + 1 > cachedLimit )
    {
        unlock cachedLimit
        unlock totalResultsCached 

        //cancel the request
        return;
    } 

    ++totalResultsCached;

    unlock cachedLimit
    unlock totalResultsCached 

    request.add(result)
}

void SendResults( int resultsToSend, Request *request )
{
    while ( resultsToSend > 0 )
    {
        send(request.remove())

        lock totalResultsCached 
        --totalResultsCached 
        unlock totalResultsCached 

        --resultsToSend;
    }
}