Energy system optimization models (ESOMs) are widely used to generate insight that informs energy and environmental policy. Using ESOMs to produce policy-relevant insight requires significant modeler judgement, yet little formal guidance exists on how to conduct analysis with ESOMs. To address this shortcoming, we draw on our collective modelling experience and conduct an extensive literature review to formalize best practice for energy system optimization modelling. We begin by articulating a set of overarching principles that can be used to guide ESOM-based analysis. To help operationalize the guiding principles, we outline and explain critical steps in the modelling process, including how to formulate research questions, set spatio-temporal boundaries, consider appropriate model features, conduct and refine the analysis, quantify uncertainty, and communicate insights. We highlight the need to develop and refine formal guidance on ESOM application, which comes at a critical time as ESOMs are being used to inform national climate targets.