Recently, a correspondence has been proposed between spectral theory and topological strings on toric Calabi-Yau manifolds. In this paper we develop in detail this correspondence for mirror curves of higher genus, which display many new features as compared to the genus one case studied so far. Given a curve of genus g, our quantization scheme leads to g different trace class operators. Their spectral properties are encoded in a generalized spectral determinant, which is an entire function on the Calabi-Yau moduli space. We conjecture an exact expression for this spectral determinant in terms of the standard and refined topological string amplitudes. This conjecture provides a non-perturbative definition of the topological string on these geometries, in which the genus expansion emerges in a suitable 't Hooft limit of the spectral traces of the operators. In contrast to what happens in quantum integrable systems, our quantization scheme leads to a single quantization condition, which is elegantly encoded by the vanishing of a quantum-deformed theta function on the mirror curve. We illustrate our general theory by analyzing in detail the resolved C3/Z5 orbifold, which is the simplest toric Calabi-Yau manifold with a genus two mirror curve. By applying our conjecture to this example, we find new quantization conditions for quantum mechanical operators, in terms of genus two theta functions, as well as new number-theoretic properties for the periods of this Calabi-Yau.