Botanic gardens and arboreta, particularly in regions where iconic relict trees naturally occur, play a vital role in the conservation of these species. Maintaining well-managed living ex situ collections of rare and threatened relict tree species provides an immediate insurance policy for the future species conservation. The aim of this research was to investigate the origin, representativeness and genetic diversity of relict trees kept in botanic gardens and arboreta. We used as a model two ecologically and biogeographically distinct members of the prominent relict genus Zelkova (Ulmaceae), which survived the last glaciation in disjunct and isolated refugial regions: Z. carpinifolia in Transcaucasia and Z. abelicea endemic to Crete (Greece) in the Mediterranean. Our study revealed substantial differences in the genetic diversity and the origin of living ex situ collections of the two investigated taxa. The living ex situ collections of Z. carpinifolia have relatively high representativeness compared with the global genetic variability of natural stands identified in a previous study. In contrast, Z. abelicea, which possesses an extraordinarily high genetic variability in natural populations, is clearly underrepresented in botanic garden collections. Moreover, all Z. abelicea trees investigated in this study most probably originated from a single region, the Levka Ori in western Crete. Thus, the ex situ conservation of Z. abelicea requires major planning and coordination efforts, including the establishment of well-documented collections in botanic gardens in Greece and especially on Crete. New living ex situ collections should be created using plant material collected from all of the mountain regions where Z. abelicea still occurs. Our study highlights the need for re-evaluating the existing living ex situ collections of trees and the development of new strategies for future conservation efforts in botanic gardens and arboreta. The coordination of conservation efforts between gardens must be enhanced to prioritize actions for the most threatened relict tree species.