The application of deep learning to medical imaging tasks has led to exceptional results in several contexts, including the analysis of human tissue samples. Convolutional neural networks (CNNs) constitute a highly performant model, that can almost perfectly detect even the smallest tumor cells in tissue biopsies. These models may have a great potential to support physicians if introduced in the clinical routines. Despite their impeccable performance on the test sets, CNNs fail in the real-world settings of the clinical workflow, lacking generalization capabilities to unseen data coming from diverse domains. New approaches shall be researched to evaluate whether a model has learned to detect correct patterns and can provide a reliable outcome. Particularly in the medical domain, understanding what are the limitations of a model is a compelling task, to ensure physicians that the model predictions are in line with the standards of clinical practice and can thus be considered in clinical routines. This thesis investigates this task by developing new interpretability techniques, with the aim of making the inner working of deep learning classifiers understandable to physicians and applicable to new inputs. By narrowing the focus onto microscopy images of breast cancer, my work starts by demonstrating that prior knowledge is a valuable source of input for explaining the model behavior. I introduce information about where the nuclei are located in the images to generate visual explanations that demonstrate that the model predictions are based on the pixels inside the nuclei contours. I then propose a method called Regression Concept Vectors (RCVs) to produce explanations based on the representation of arbitrary concepts that can be obtained as measures directly extracted from the images or annotated by experts. This approach demonstrates that variations of the texture in the images are relevant to the model. Going beyond the purpose of generating explanations, I directly tackle the generalization deficiencies of existing models. I propose a pruning system that uses RCVs to remove from the model's learning process the undesired behavior of capturing content about unwanted features. As an example, I analyze the removal of the implicitly learned invariance to object scale in models that are pre-trained on natural images, since scale is instead a relevant measure for the analysis of medical images. I then guide the training of CNNs to learn morphology features while discarding the confounding information about data provenance, demonstrating that the resulting model has increased generalization capabilities.