Prominent theories of attention claim that visual search is guided through attentional templates stored in working memory. Recently, the contralateral delay activity (CDA), an electrophysiological index of working memory storage, has been found to rapidly decrease when participants repeatedly search for the same target, suggesting that, with learning, the template moves out of working memory. However, this has only been investigated with pop-out search for distinct targets, for which a strong attentional template may not be necessary. More effortful search tasks might rely more on an active attentional template in working memory, leading to a slower handoff to long-term memory and thus a slower decline of the CDA. Using ERPs, we compared the rate of learning of attentional templates in pop-out and effortful search tasks. In two experiments, the rate of decrease in the CDA was the same for both search tasks. Similar results were found for a second component indexing working memory effort, the late positive complex. However, the late positive complex was also sensitive to anticipated search difficulty, as was expressed in a greater amplitude before the harder search task. We conclude that the amount of working memory effort invested in maintaining an attentional template, but not the rate of learning, depends on search difficulty.