The new Neural Machine Translation (NMT) paradigm could reveal itself as an efficient tool to facilitate access to Easy Language (EL) texts to larger segments of the population, particularly when resources are limited. One could hypothesize that, if the source text complies with EL best practices, the MT output would be comparable from an accessibility perspective. Results from recent work, however, suggest otherwise (Kaplan et al., 2019; Rodríguez Vázquez et al., forthcoming). Still, the potential improvement achieved after a post editing stage remains unexplored. Similarly, a key question needs to be asked: do all NMT errors have the same impact on text comprehensibility? With a view to gaining insight into this question, a reception study was conducted with adults with reading difficulties. Participants, who regularly collaborate with a Swiss institution delivering certified EL documents, assessed the human translation, as well as the raw and post-edited versions of a French administrative text translated from Easy German with DeepL. The raw NMT output had previously undergone manual error annotation in terms of translation quality and violation of EL guidelines. Based on our knowledge about the target group, mistranslations and non comprehensible passages were discarded for ethical reasons. The final set of test sentences (human, raw and post-edited) were counterbalanced but presented to participants in context. Text comprehensibility was measured through content-related questions in dialogue, on a one-to-one basis. In addition, the study facilitator kept a checklist-like diary where problematic passages and specific text elements were recorded. Findings from the error annotation will be compared against the data collected during the reception study. We anticipate that there will not be major differences between the two MT versions in terms of cognitive complexity (Hansen-Schirra et al., 2020), but that text comprehension will be more strongly affected by certain EL rule violations than others.