We consider an atomic frequency comb based quantum memory inside an asymmetric optical cavity. In this configuration it is possible to absorb the input light completely in a system with an effective optical depth of one, provided that the absorption per cavity round trip exactly matches the transmission of the coupling mirror (“impedance matching”). We show that the impedance matching results in a readout efficiency only limited by irreversible atomic dephasing, whose effect can be made very small in systems with large inhomogeneous broadening. Our proposal opens up an attractive route toward quantum memories with close to unit efficiency.