Colostomized colorectal cancer survivors present physical, psychological, and social difficulties after treatment, which complicate their return to normal life. Nevertheless, we lack insight into their lived transition experience after treatment in Switzerland. We led nine semi-structured interviews with participants who had a colostomy for a mean of 16.4 years. The participants’ mean age was 73.1 years. Through an inductive thematic analysis, we highlighted several adaptation difficulties. Their body image was poor, and they behaved differently than before the surgery. For fear of being rejected, they concealed their colostomy. However, they felt isolated and often wished for more support. They may be in a vicious circle where their difficulties are maintained. We advocate that supporting survivors psychologically beyond the end of their treatment is essential to facilitate adjustment and overcome cancer.