A search for the nonresonant production of Higgs boson pairs in the HH→bb¯τ+τ- channel is performed using 140  fb-1 of proton-proton collisions at a center-of-mass energy of 13 TeV recorded by the ATLAS detector at the CERN Large Hadron Collider. The analysis strategy is optimized to probe anomalous values of the Higgs boson self-coupling modifier κλ and of the quartic HHVV (V=W,Z) coupling modifier κ2V. No significant excess above the expected background from Standard Model processes is observed. An observed (expected) upper limit μHH<5.9(3.3) is set at 95% confidence-level on the Higgs boson pair production cross section normalized to its Standard Model prediction. The coupling modifiers are constrained to an observed (expected) 95% confidence interval of -3.1<κλ<9.0 (-2.5<κλ<9.3) and -0.5<κ2V<2.7 (-0.2<κ2V<2.4), assuming all other Higgs boson couplings are fixed to the Standard Model prediction. The results are also interpreted in the context of effective field theories via constraints on anomalous Higgs boson couplings and Higgs boson pair production cross sections assuming different kinematic benchmark scenarios.