A measurement of the associated production of a top-quark pair (tt¯) with a vector boson (W, Z) in proton-proton collisions at a center-of-mass energy of 13 TeV is presented, using 36.1 fb−1 of integrated luminosity collected by the ATLAS detector at the Large Hadron Collider. Events are selected in channels with two same- or opposite-sign leptons (electrons or muons), three leptons or four leptons, and each channel is further divided into multiple regions to maximize the sensitivity of the measurement. The tt¯Z and tt¯W production cross sections are simultaneously measured using a combined fit to all regions. The best-fit values of the production cross sections are σtt¯Z=0.95±0.08stat±0.10syst pb and σtt¯W=0.87±0.13stat±0.14syst pb in agreement with the Standard Model predictions. The measurement of the tt¯Z cross section is used to set constraints on effective field theory operators which modify the tt¯Z vertex.