Dairy products are included in dietary guidelines worldwide, as milk, yoghurt, and cheese are good sources of calcium and protein, vital nutrients for bones and muscle mass maintenance. Bone growth and mineralization occur during infancy and childhood, peak bone mass being attained after early adulthood. A low peak bone mass has consequences later in life, including increased risk of osteoporosis and fractures. Currently, more than 200 million people worldwide suffer from osteoporosis, with approximately 9 million fractures yearly. This poses a tremendous economic burden on health care. Between 5% and 10% of the elderly suffer from sarcopenia, the loss of muscle mass and strength, further increasing the risk of fractures due to falls. Evidence from interventional and observational studies support that fermented dairy products in particular exert beneficial effects on bone growth and mineralization, attenuation of bone loss, and reduce fracture risk. The effect cannot be explained by single nutrients in dairy, which suggests that a combined or matrix effect may be responsible similar to the matrix effects of foods on cardiometabolic health. Recently, several plant-based beverages and products have become available and marketed as substitutes for dairy products, even though their nutrient content differs substantially from dairy. Some of these products have been fortified, in efforts to mimic the nutritional profile of milk, but it is unknown whether the additives have the same bioavailability and beneficial effect as dairy. We conclude that the dairy matrix exerts an effect on bone and muscle health that is more than the sum of its nutrients, and we suggest that whole foods, not only single nutrients, need to be assessed in future observational and intervention studies of health outcomes. Furthermore, the importance of the matrix effect on health outcomes argues in favor of making future dietary guidelines food based.