I need a data structure that acts like a SortedDictionary<int, double> but is sorted based on the values rather than the keys. I need it to take about 1-2 microseconds to add and remove items when we have about 3000 items in the dictionary.

My first thought was simply to switch the keys and values in my code. This very nearly works. I can add and remove elements in about 1.2 microseconds in my testing by doing this.

But the keys have to be unique in a SortedDictionary so that means that values in my inverse dictionary would have to be unique. And there are some cases where they may not be.

Any ideas of something in the .NET libraries already that would work for me?