Modelling a reward able to convey the right incentive to the agent is fairly tedious in terms of engineering required. Imitation learning bypasses this time-consuming hurdle by enticing the agent to mimic an expert instead of trying to maximize a potentially ill-designed reward signal. We first build on the strengths of off-policy learning to design a novel adversarial imitation approach that addresses the high sample complexity suffered by Generative Adversarial Imitation Learning, the state-of-the-art imitation approach. Second, we show that forcing the adversarially learned reward function to be local Lipschitz-continuous is a sine qua non condition for the method to perform well. We complement this empirical evidence with several theoretical guarantees. Finally, we introduce the concept of dataset-grounded optimality inductive bias for offline agents. By carefully orchestrating such priors in our generalization of importance-weighted regression, we can achieve better results, while remaining agnostic with respect to the quality the dataset.