This Letter presents a search for exotic decays of the Higgs boson to a pair of new (pseudo)scalar particles, H→aa , where the a particle has a mass in the range 20–60 GeV, and where one of the a bosons decays into a pair of photons and the other to a pair of gluons. The search is performed in event samples enhanced in vector-boson fusion Higgs boson production by requiring two jets with large invariant mass in addition to the Higgs boson candidate decay products. The analysis is based on the full dataset of pp collisions at s=13TeV recorded in 2015 and 2016 with the ATLAS detector at the CERN Large Hadron Collider, corresponding to an integrated luminosity of 36.7 fb −1 . The data are in agreement with the Standard Model predictions and an upper limit at the 95% confidence level is placed on the production cross section times the branching ratio for the decay H→aa→γγgg . This limit ranges from 3.1 pb to 9.0 pb depending on the mass of the a boson.