Visual working memory (WM) enables the use of past sensory experience in guiding behavior. Yet, laboratory tasks commonly evaluate WM in a way that separates it from its sensory bottleneck. To understand how perception interacts with visual memory, we used a delayed shape recognition task to probe how WM may differ for stimuli that bias processing toward different visual pathways. Luminance compared with chromatic signals are more efficient in driving the processing of shapes and may thus also lead to better WM encoding, maintenance, and memory recognition. To evaluate this prediction, we conducted two experiments. In the first psychophysical experiment, we measured contrast thresholds for different WM loads. Luminance contrast was encoded into WM more efficiently than chromatic contrast, even when both sets of stimuli were equated for discriminability. In the second experiment, which also equated stimuli for discriminability, early sensory responses in the EEG that are specific to luminance pathways were modulated by WM load and thus likely reflect the neural substrate of the increased efficiency. Our results cannot be accounted for by simple saliency differences between luminance and color. Rather, they provide evidence for a direct connection between low-level perceptual mechanisms and WM by showing a crucial role of luminance for forming WM representations of shape.