We propose a new family of matrix models whose 1/N expansion captures the all-genus topological string on toric Calabi-Yau threefolds. These matrix models are constructed from the trace class operators appearing in the quantization of the corresponding mirror curves. The fact that they provide a non-perturbative realization of the (standard) topological string follows from a recent conjecture connecting the spectral properties of these operators, to the enumerative invariants of the underlying Calabi-Yau threefolds. We study in detail the resulting matrix models for some simple geometries, like local P^2 and local F_2, and we verify that their weak 't Hooft coupling expansion reproduces the topological string free energies near the conifold singularity. These matrix models are formally similar to those appearing in the Fermi-gas formulation of Chern-Simons-matter theories, and their 1/N expansion receives non-perturbative corrections determined by the Nekrasov-Shatashvili limit of the refined topological string.