The discrimination power of well-known model selection criteria is analyzed when the R-squared is low as in typical asset return predictability studies. It turns out that the discrimination power is low in this situation and this may explain, already in a simple i.i.d. setup, why often in-sample predictability, but no out-of-sample predictability is found. In particular it is possible to give another interpretation to the results of the well-cited Bossaerts and Hillion (Rev. Financial Stud. 12 (1999) 405–428) study. As a consequence, model selection criteria are put in a testing framework and a bootstrap-based procedure is proposed as a diagnostic tool to construct the class of models which are statistically indistinguishable from the best model chosen by a model selection criterion. In an empirical illustration the Pesaran and Timmerman (J. Finance 50 (1995) 1201–1228) results are reanalyzed and it turns out that in this case this class of models can be large. Finally it is shown that similar problems arise in a more hidden way in the context of recent model uncertainty studies using Bayesian model selection criteria.