The crystalline and defect structure of epitaxial hexagonal RExMnyO3 (RE = Er, Dy) films with varying cationic composition was investigated by X-ray diffraction and transmission electron microscopy. The films are composed of a strained layer at the interface with the substrate and of a relaxed layer on top of it. The critical thickness is of similar to 10 to 25 nm. For Mn-rich films (or RE deficient), an off-stoichiometric composition maintaining the hexagonal LuMnO3-type structure is stabilized over a large range of the RE/Mn ratio (0.72-1.00), with no Mn-rich secondary phases observed. A linear dependence of the out-of-plane lattice parameter with RE/Mn is observed in this range. Out-of-phase boundary (OPB) extended defects are observed in all films and exhibit a local change in stoichiometry. Such a large solubility limit in the RE deficient region points toward the formation of vacancies on the RE site (RExMnO3-delta, with 0.72 <= x < 1), a phenomenon that is encountered in perovskite manganites such as LaxMnO3-delta (x < 1) and that may strongly impact the physical properties of hexagonal manganites.