Congruency tasks have provided support for an amodal magnitude system for magnitudes that have a “spatial” character, but conflicting results have been obtained for magnitudes that do not (e.g., luminance). In this study, we extricated the factors that underlie these number–luminance congruency effects and tested alternative explanations: (unsigned) luminance contrast and saliency. When luminance had to be compared under specific task conditions, we revealed, for the first time, a true influence of number on luminanc e judgments: Darker stimuli were consistently associated with numerically larger stimuli. However, when number had to be compared, luminance contrast, not luminance, influenced number judgments. Apparently, associations exist between number and luminance, as well as luminance contrast, of which the latter is probably stronger. Therefore, similar tasks, comprising exactly the same stimuli, can lead to distinct interference effects.