Attentional templates are stored representations of target features that guide visual search. While transiently active templates are as efficient as templates held in a sustained fashion, their simultaneous activation generates costs for the sustained template. Here, we investigated whether the quality of the memory representation determines these costs. Two possible target colors were cued before search display onset. In blocked conditions, the two colors either changed on every trial or were fixed throughout. In the mixed condition, one color was fixed, while the other varied from trial to trial. In Experiment 1, participants also reproduced one of the two target colors on a memory wheel after each search episode. The analysis of search performance replicated longer RTs to sustained than transient targets when template types were mixed, but no difference when they were blocked. Critically, analysis of memory judgments showed that random guesses for sustained templates increased from the blocked to the mixed condition, mirroring RTs. This suggests that newly activated transient templates retroactively interfered with already activated sustained templates, impairing their efficiency to guide attention and their stability in memory. Increasing the priority of sustained templates through maintenance constraints (Experiments 1 vs. 2) or retro-cueing (Experiment 3) reduced the associated costs. Finally, these costs were unaffected by different retention intervals (Experiment 4). We argue that retroactive interference affects the control of visual search and memory maintenance alike, but critically depends on the respective priority of representations in visual working memory.