I have an image where I want all the pixels with a specific color converted to transparent pixels. The operation should not change the color/alpha value of any pixel which don't match the color exactly. How do I do that?

At first I thought I could use Colors->"Color to Alpha" but that don't work because it changes the color of all pixels(It adds an alpha value to all pixels).

Using Gimp 2.6.11 on Linux