The SARS-CoV-2 pandemic has overwhelmed health care systems worldwide since its first wave. Intensive care units have been under a significant amount of pressure as patients with the most severe form of the disease presented with acute respiratory distress syndrome (ARDS). A proportion of them experienced refractory acute respiratory failure and had to be supported with veno-venous extracorporeal membrane oxygenation (VV-ECMO). The present retrospective study reports the experiences of our ECMO center in the management of COVID-19 patients with refractory ARDS. Patient characteristics and outcomes are presented through the different waves of the pandemic. A cohort study was conducted on patients with refractory ARDS due to COVID-19 infection who were admitted to the intensive care unit (ICU) at the Geneva University Hospital and supported with VV-ECMO between 14 March 2020 and January 2022. The VV-ECMO implementation criteria were defined according to an institutional algorithm validated by the local crisis unit of the hospital and the Swiss Society of Intensive Care Medicine. Among the 500 ARDS patients admitted to our ICU, 41 patients with a median age of 57 (52−63) years, a body mass index (BMI) of 28 (26−32) kg/m2, and a SAPS II score of 57 (47−67), and 27 (66%) of whom were men required VV-ECMO. None of the patients were vaccinated. The time of ventilation, including noninvasive ventilation (NIV) and mechanical ventilation (MV), and the time of MV before ECMO were 7 (4−11) days and 4 (1−7) days, respectively. The time under ECMO was 20 (10−27) days. The ICU and hospital lengths of stay were 36 (21−45) days and 45 (33−69) days, respectively. The survival rate for patients on ECMO was 59%. Comparative analysis between survivors and non-survivors highlighted that survivors had a significantly shorter ventilation duration before ECMO (NIV + MV: 5.5 (1.3−9) vs. 9 (6.5−13.5) days, p = 0.0026 and MV alone: 1.6 (0.4−5.5) vs. 5.8 (5−8) days, p < 0.0001). The management of patients on ECMO by an experienced ECMO team dedicated to this activity was associated with improved survival (78% vs. 28%, p = 0.0012). Between the first wave and the following waves, patients presented with a higher incidence of ventilator-associated pneumonia (100% vs. 82%, p = 0.0325) but had better survival rates (74% vs. 35%, p = 0.024). The present study suggests that both the prompt insertion of VV-ECMO to control refractory hypoxemia and the involvement of an ECMO team improve the survival of COVID-19 patients.