The brain continuously maintains a remarkably high level of intrinsic activity. This activity is non-stationary and its dynamics reveal highly structured patterns across several spatial scales, from fine-grained functional architecture in sensory cortices to large-scale networks. The mechanistic function of this activity is only poorly understood. The central goal of the current review is to provide an integrated summary of recent studies on structure, dynamics and behavioral consequences of spontaneous brain activity. In light of these empirical observations we propose that the structure of ongoing activity and its itinerant nature can be understood as an indispensible memory system modeling the statistical structure of the world. We review the dynamic properties of ongoing activity, and how they are malleable over short to long temporal scales that permit adapting over a range of short- to long-term cognitive challenges. We conclude by reviewing how the functional significance of ongoing activity manifests in its impact on human action, perception, and higher cognitive function.