I'd like to remove a layer (in this case the results of geom_ribbon) from a ggplot2 created grid object. Is there a way I can remove it once it's already part of the object?

library(ggplot2)
dat <- data.frame(x=1:3, y=1:3, ymin=0:2, ymax=2:4)
p <- ggplot(dat, aes(x=x, y=y)) + geom_ribbon(aes(ymin=ymin, ymax=ymax), alpha=0.3) 
     + geom_line()

# This has the geom_ribbon
p

# This overlays another ribbon on top
p + geom_ribbon(aes(ymin=ymin, ymax=ymax, fill=NA))