Speech is the most important signal in our daily auditory environment and its processing is highly context dependent. However it is unknown how contextual demands influence the encoding of the speech signal. We found with the use of fMRI that the performance of different tasks on the very same speech sounds affects sound encoding in the earliest levels of auditory cortical processing in a manner that is largely consistent with the acoustic features in the stimuli that are critically relevant for task performance. This modulation is possible because of the multi-resolution analysis of sounds by the auditory cortex, allowing for a stable and robust auditory experience despite the need for simultaneous preferential processing of specific features depending on the environmental demands. Our work provides important insights into the mechanisms that enable stable auditory perception within our dynamic auditory environment, and may generalize to perceptual mechanisms in other sensory modalities.