Axion dark matter differentiates the phase velocities of the circular-polarized photons. In this Letter, a scheme to measure the phase difference by using a linear optical cavity is proposed. If the scheme is applied to the Fabry-Pérot arm of Advanced-LIGO-like (Cosmic-Explorer-Like) gravitational wave detector, the potential sensitivity to the axion-photon coupling constant, gaγ, reaches gaγ≃8×10−13 GeV−1(4×10−14 GeV−1) at the axion mass m≃3×10−13 eV (2×10−15 eV) and remains at around this sensitivity for three orders of magnitude in mass. Furthermore, its sensitivity has a sharp peak reaching gaγ≃10−14 GeV−1(8×10−17 GeV−1) at m=1.563×10−10 eV (1.563×10−11 eV). This sensitivity can be achieved without losing any sensitivity to gravitational waves.