We present measurements of cross-sections for production of a leptonically decaying $Z$ boson in association with a large-radius jet in 13 TeV proton-proton collisions at the LHC, using $36~\mathrm{fb}^{-1}$ of data from the ATLAS detector. Integrated and differential cross-sections are measured at particle-level in both a flavor-inclusive and a doubly $b$-tagged fiducial phase-space. The large-radius jet mass and transverse momentum, its kinematic relationship to the $Z$ boson, and the angular separation of $b$-tagged small-radius track-jets within the large-radius jet are measured. This measurement constitutes an important test of perturbative quantum chromodynamics in kinematic and flavor configurations relevant to several Higgs boson and beyond-Standard-Model physics analyses. The results highlight issues with modeling of additional hadronic activity in the flavor-inclusive selection, and a distinction between flavor-number schemes in the $b$-tagged phase-space.