In this thesis, we investigate various regularization approaches for deep learning models. As the deep learning models can learn very complex functions using enormous numbers of parameters, the need for appropriate regularization becomes even more crucial. In this work, we are particularly interested in the setting where we have access to some domain knowledge that can be used to design various constraints on the learned model to improve models generalization performance.Throughout the thesis, we consider two types of domain knowledge: information about features’ intrinsic properties, which we refer to as feature side-information, and a black-box software that can measure the properties of any instances sampled from the underlying domain.In the first half of the manuscript, we focus on regularizing the predictive models using the feature side-information. Feature side-information is most often ignored or used only for feature selection prior to model fitting. In this work, we propose to incorporate the feature side-information into the learning of the predictive models where we assume that similar features should have a similar effect on the learned model. We present a regularizer that forces the learned model to be invariant/symmetric to relative changes in the values of similar features where the feature similarity is defined based on the feature side information. In the second half of the thesis, we focus on tackling the inverse problem: generating discrete structures that exhibit a fixed set of properties in the presence of black-box software which can evaluate the property of any discrete structures from the underlying domain. Even though unconditional generation of discrete structures has been tackled very successfully, the conditional generation remains challenging due to the discrete nature of the data. Existing methods are mostly limited to conditioning on binary classes. When conditioning on continuous properties, it is formulated as property optimization where the algorithms look for discrete structures that have their properties enhanced.