Emerging SARS-CoV-2 variants raise questions about escape from previous immunity. As the population immunity to SARS-CoV-2 has become more complex due to prior infections with different variants, vaccinations or the combination of both, understanding the antigenic relationship between variants is needed. Here, we have assessed neutralizing capacity of 120 blood specimens from convalescent individuals infected with ancestral SARS-CoV-2, Alpha, Beta, Gamma or Delta, double vaccinated individuals and patients after breakthrough infections with Delta or Omicron-BA.1. Neutralization against seven authentic SARS-CoV-2 isolates (B.1, Alpha, Beta, Gamma, Delta, Zeta and Omicron-BA.1) determined by plaque-reduction neutralization assay allowed us to map the antigenic relationship of SARS-CoV-2 variants. Highest neutralization titers were observed against the homologous variant. Antigenic cartography identified Zeta and Omicron-BA.1 as separate antigenic clusters. Substantial immune escape in vaccinated individuals was detected for Omicron-BA.1 but not Zeta. Combined infection/vaccination derived immunity results in less Omicron-BA.1 immune escape. Last, breakthrough infections with Omicron-BA.1 lead to broadly neutralizing sera.