I have this design:

public interface IFactory<T> {
  T Create();
  T CreateWithSensibleDefaults();
}

public class AppleFactory : IFactory<Apple> { ... }
public class BananaFactory : IFactory<Banana> { ... }
// ...

The fictitious Apple and Banana here do not necessarily share any common types (other than object, of course).

I don't want clients to have to depend on specific factories, so instead, you can just ask a FactoryManager for a new type. It has a FactoryForType method:

IFactory<T> FactoryForType<T>();

Now you can invoke the appropriate interface methods with something like FactoryForType<Apple>().Create(). So far, so good.

But there's a problem at the implementation level: how do I store this mapping from types to IFactory<T>s? The naive answer is an IDictionary<Type, IFactory<T>>, but that doesn't work since there's no type covariance on the T (I'm using C# 3.5). Am I just stuck with an IDictionary<Type, object> and doing the casting manually?