The partition function of ABJ(M) theories on the three-sphere can be regarded as the canonical partition function of an ideal Fermi gas with a non-trivial Hamiltonian. We propose an exact expression for the spectral determinant of this Hamiltonian, which generalizes recent results obtained in the maximally supersymmetric case. As a consequence, we find an exact WKB quantization condition determining the spectrum which is in agreement with numerical results. In addition, we investigate the factorization properties and functional equations for our conjectured spectral determinants. These functional equations relate the spectral determinants of ABJ theories with consecutive ranks of gauge groups but the same Chern-Simons coupling.