We investigate the two-terminal nonlinear conductance of a Coulomb-blockaded quantum dot attached to chiral edge states. Reversal of the applied magnetic field inverts the system chirality and leads to a different polarization charge. As a result, the current--voltage characteristic is not an even function of the magnetic field. We show that the corresponding magnetic-field asymmetry arises from single-charge effects and vanishes in the limit of high temperature.