We report evidence for a context- and not stimulus-dependent functional asymmetry in the left and right human auditory midbrain, thalamus, and cortex in response to monaural sounds. Neural activity elicited by left- and right-ear stimulation was measured simultaneously in the cochlear nuclei, inferior colliculi (ICs), medial geniculate bodies (MGBs), and auditory cortices (ACs) in 2 functional magnetic resonance imaging experiments. In experiment 1, pulsed noise was presented monaurally to either ear, or binaurally, simulating a moving sound source. In experiment 2, only monaural sounds were presented. The results show a modulation of the neural responses to monaural sounds by the presence of binaural sounds at a time scale of tens of seconds: In the absence of binaural stimulation, the left and right ICs, MGBs, and ACs responded stronger to stimulation of the contralateral ear. When blocks of binaural stimuli were interspersed in the sound sequence, the contralateral preference vanished in those structures in the right hemisphere. The resulting hemispheric asymmetry was similar to the asymmetry demonstrated for spatial sound processing. Taken together, the data demonstrate that functional asymmetries in auditory processing are modulated by context. The observed long time constant suggests that this effect results from a "top-down" mechanism.