The heme enzyme indoleamine 2,3-dioxygenase 1 (IDO1) plays an essential role in immunity, neuronal function, and aging through catalysis of the rate-limiting step in the kynurenine pathway of tryptophan metabolism. Many IDO1 inhibitors with different chemotypes have been developed, mainly targeted for use in anti-cancer immunotherapy. Lead optimization of direct heme iron-binding inhibitors has proven difficult due to the remarkable selectivity and sensitivity of the heme-ligand interactions. Here, we present experimental data for a set of closely related small azole compounds with more than 4 orders of magnitude differences in their inhibitory activities, ranging from millimolar to nanomolar levels. We investigate and rationalize their activities based on structural data, molecular dynamics simulations, and density functional theory calculations. Our results not only expand the presently known four confirmed chemotypes of sub-micromolar heme binding IDO1 inhibitors by two additional scaffolds but also provide a model to predict the activities of novel scaffolds.