Modeled direct normal irradiance (DNI) can be either derived from satellite data or from numerical weather prediction models. Such modeled datasets are available at continental scale and provide continuous long-term time series, but are known to fall short in quality over some areas, compared to high-quality ground measurements. The uncertainty in DNI may be locally so high that CSP projects cannot be financed. The CSP industry would obviously benefit from a comprehensive and large-scale benchmarking of the existing modeled DNI datasets. This would help CSP developers select the most appropriate dataset for a given region, and would also provide due-diligence or financial analysts with the desired information on the expected accuracy of the data. This contribution investigates how the benchmarking study should be conducted, and evaluates the difficulties that can be encountered. A large set of ground stations with anticipated good- to high-quality measurements has been identified, most of which reporting public-domain data. Various criteria that such measured datasets must fulfill to be usable in the benchmarking study are discussed. Automatized quality control methods are also proposed for quality assessment purposes, and are described in some detail. The most important statistics to be used for the benchmarking process are discussed, with an emphasis on those that appear of particular importace to the CSP industry. The full-scale benchmarking study should now follow, assuming proper funding is secured.