Rectal cancer is a heterogeneous disease at the genetic and molecular levels, both aspects having major repercussions on the tumor immune contexture. Whilst microsatellite status and tumor mutational load have been associated with response to immunotherapy, presence of tumor-infiltrating lymphocytes is one of the most powerful prognostic and predictive biomarkers. Yet, the majority of rectal cancers are characterized by microsatellite stability, low tumor mutational burden and poor T cell infiltration. Consequently, these tumors do not respond to immunotherapy and treatment largely relies on radiotherapy alone or in combination with chemotherapy followed by radical surgery. Importantly, pre-clinical and clinical studies suggest that radiotherapy can induce a complete reprograming of the tumor microenvironment, potentially sensitizing it for immune checkpoint inhibition. Nonetheless, growing evidence suggest that this synergistic effect strongly depends on radiotherapy dosing, fractionation and timing. Despite ongoing work, information about the radiotherapy regimen required to yield optimal clinical outcome when combined to checkpoint blockade remains largely unavailable. In this review, we describe the molecular and immune heterogeneity of rectal cancer and outline its prognostic value. In addition, we discuss the effect of radiotherapy on the tumor microenvironment, focusing on the mechanisms and benefits of its combination with immune checkpoint inhibitors.