Motion mitigation strategies are crucial for scanned particle therapy of mobile tumours in order to prevent geometrical target miss and interplay effects. We developed a patient-specific respiratory motion model based on simultaneously acquired time-resolved volumetric MRI and 2D abdominal ultrasound images. We present its effects on 4D pencil beam scanned treatment planning and simulated dose distributions. Given an ultrasound image of the liver and the diaphragm, principal component analysis and Gaussian process regression were applied to infer dense motion information of the lungs. 4D dose calculations for scanned proton therapy were performed using the estimated and the corresponding ground truth respiratory motion; the differences were compared by dose difference volume metrics. We performed this simulation study on 10 combined CT and 4DMRI data sets where the motion characteristics were extracted from 5 healthy volunteers and fused with the anatomical CT data of two lung cancer patients. Median geometrical estimation errors below 2 mm for all data sets and maximum dose differences of = 43.2% and = 16.3% were found. Moreover, it was shown that abdominal ultrasound imaging allows to monitor organ drift. This study demonstrated the feasibility of the proposed ultrasound-based motion modelling approach for its application in scanned proton therapy of lung tumours.