Quantum theory allows for randomness generation in a device-independent setting, where no detailed description of the experimental device is required. Here we derive a general upper bound on the amount of randomness that can be generated in such a setting. Our bound applies to any black-box scenario, thus covering a wide range of scenarios from partially characterised to completely uncharacterised devices. Specifically, we prove that the number of random bits that can be generated is limited by the number of different input states that enter the measurement device. We show explicitly that our bound is tight in the simplest case. More generally, our work indicates that the prospects of generating a large amount of randomness by using high-dimensional (or even continuous variable) systems will be extremely challenging in practice.