We examined the attentional modulation of semantic priming and the N400 effect for spoken words. The aim was to find out how the semantics of spoken language is processed when attention is directed to another modality (passive task), to the phonetics of spoken words (phonological task), or to the semantics of spoken words (word task). Equally strong behavioral priming effects were obtained in the phonological and the word tasks. A significant N400 effect was found in all tasks. The effect was stronger in the word and the phonological tasks than in the passive task, but there was no difference in the magnitude of the effect between the phonological and the word tasks. The latency of the N400 effect did not differ between the tasks. Although the N400 effect had a centroparietal maximum in the phonological and the word tasks, it was largest at the parietal recording sites in the passive task. The effect was more pronounced at the left than right recording sites in the phonological task, but there was no laterality effect in the other tasks. The N400 effect in the passive task indicates that semantic priming occurs even when spoken words are not actively attended. However, stronger N400 effect in the phonological and the word tasks than in the passive task suggests that controlled processes modulate the N400 effect. The finding that there were no differences in the N400 effect between the phonological and the word tasks indicates that the semantics of attended spoken words is processed regardless of whether semantic processing is relevant for task performance.