Lithium borohydride is a promising lithium ion conductor for all-solid-state batteries. However, the compound only exhibits high ionic conductivity at elevated temperatures, typically above 110 °C. It was shown that the addition of oxides such as silica or alumina increases the room temperature ionic conductivity by 3 orders of magnitude. The origin of this remarkable effect is not yet well understood. Here, we investigate the influence of oxide surface groups on the ionic conductivity of LiBH4/SiO2 nanocomposites. We systematically varied the density and nature of the surface groups of mesoporous silica by heat treatment at different temperatures, or surface functionalization, and subsequently prepared LiBH4/SiO2 nanocomposites by melt infiltration. The ionic conductivity is strongly influenced by the heat treatment temperature, hence the density of the free surface silanol groups. Replacing some of the silanol groups with hydrophobic surface groups resulted in an order of magnitude reduction of the room temperature ionic conductivity, suggesting that their presence is crucial to obtain high ionic conductivity in the nanocomposites. This systematic study and insight provide a basis for further exploration of the impact of surface groups, and for the rational design of novel solid-state nanocomposite electrolytes via interface engineering.