Given its severe capacity limitations, visual working memory (VWM) can process only a tiny fraction of the complex visual world. While selection of relevant information from cluttered scenes is a main topic of research on visual attention, it has not received much research efforts in the VWM community. Based on knowledge from visual-attention research, we developed a task which approaches the complexity of real-world scenes while maintaining tight experimental control over stimulation. Participants were presented with an array of 33 vertical bars and 3 tilted target bars (12°, 28° and 45°). After presentation, one of the targets was probed and participant had to recall its color (continuous report). In a first experiment, we provide evidence that the distribution of a limited VWM resource is parametrically influenced by saliency, F(2, 18) = 42.77, p < .001. In a second experiment, by assigning the same tilt to targets within a trial, we showed that the distribution of resources is not only influenced by the saliency difference between targets, but also by their absolute saliency. The second experiment also replicated results of the first, F(2, 60) = 102.90, p < .001. Our third experiment used typical displays (three squares) and demonstrated that – in contrast to the real world – saliency is virtually maxed out for relevant objects in typical laboratory studies of VWM, likely yielding a pronounced underestimation of this major influence on VWM. A fourth experiment examined the influence of saliency across 7 different encoding times and showed that encoding time has an influence on the effect of saliency, F(12, 180) = 6.72, p < .001, but also that, even after encoding for 2 seconds this effect remains present, t(15) = 3.52, p = .003, showing that our manipulation of saliency is very strong.