Several studies have shown that maintaining in memory some attributes of speech, such as the content or pitch of an interlocutor's message, is markedly reduced in the presence of background sounds made of spectrotemporal variations. However, experimental paradigms showing this interference have only focused on one attribute of speech at a time, and thus differ from real-life situations in which several attributes have to be memorized and maintained simultaneously. It is possible that the interference is even greater in such a case and can occur for a broader range of background sounds. We developed a paradigm in which participants had to maintain the content, pitch and speaker size of auditorily presented speech information and used various auditory distractors to generate interference. We found that only distractors with spectrotemporal variations impaired the detection, which shows that similar interference mechanisms occur whether there are one or more speech attributes to maintain in memory. A high percentage of false alarms was observed with these distractors, suggesting that spectrotemporal variations not only weaken but also modify the information maintained in memory. Lastly, we found that participants were unaware of the interference. These results are similar to those observed in the visual modality.