Local adaptation to rare habitats is difficult due to gene flow, but can occur if the habitat has higher productivity. Differences in offspring phenotypes have attracted little attention in this context. We model a scenario where the rarer habitat improves offspring's later competitive ability – a carryover effect that operates on top of local adaptation to one or the other habitat type. Assuming localised dispersal, so the offspring tend to settle in similar habitat to the natal type, the superior competitive ability of offspring remaining in the rarer habitat hampers immigration from the majority habitat. This initiates a positive feedback between local adaptation and trait divergence, which can thereafter be reinforced by coevolution with dispersal traits that match ecotype to habitat type. Rarity strengthens selection on dispersal traits and promotes linkage disequilibrium between locally adapted traits and ecotype-habitat matching dispersal. We propose that carryover effects may initiate isolation by ecology.