I'm interested in building a heatmap with geom_tile in ggplot2 that uses a different gradient high color based on a factor.

The plot below creates the plot where the individual tiles are colored blue or red based on the xy_type, but there is no gradient.

ggplot() + 
  geom_tile(data=mydata, aes(x=factor(myx), y=myy, fill=factor(xy_type))) + 
  scale_fill_manual(values=c("blue", "red"))

The plot below does not use the xy_type factor to choose the color, but I get a single group gradient based on the xy_avg_value.

ggplot() + 
  geom_tile(data=mydata, aes(x=factor(myx), y=myy, fill=xy_avg_value))

Is there a technique to blend these two plots? I can use a facet_grid(xy_type ~ .) to create separate plots of this data, with the gradient. As this is ultimately going to be a map (x~y coordinates), I'd like to find a way to display the different gradient together in a single geom_tile map.