We present an ultrasound-driven 4D magnetic resonance imaging (US-4DMRI) method for respiratory motion imaging in the thorax and abdomen. The proposed US-4DMRI comes along with a high temporal resolution, and allows for organ motion imaging beyond a single respiratory cycle. With the availability of the US surrogate both inside and outside the MR bore, 4D MR images can be reconstructed for 4D treatment planning and online respiratory motion prediction during radiotherapy. US-4DMRI relies on simultaneously acquired 2D liver US images and abdominal 2D MR multi-slice scans under free respiration. MR volumes are retrospectively composed by grouping the MR slices corresponding to the most similar US images. We present two different US similarity metrics: an intensity-based approach, and a similarity measure relying on predefined fiducials which are being tracked over time. The proposed method is demonstrated on MR liver scans of eight volunteers acquired over a duration of 5.5 min each at a temporal resolution of 2.6 Hz with synchronous US imaging at 14 Hz-17 Hz. Visual inspection of the reconstructed MR volumes revealed satisfactory results in terms of continuity in organ boundaries and blood vessels. In quantitative leave-one-out experiments, both US similarity metrics reach the performance level of state-of-the-art navigator-based approaches.