Promoting calcium intake is a cornerstone for osteoporosis management. Some individuals limit dairy product consumption, a major calcium source, due to their high content in saturated fats and their perceived negative impact on lipid profiles. This study explored the associations of calcium from various sources with blood lipids in community-dwelling elderly (n = 717) from the GERICO cohort. Dietary calcium intake was assessed at several timepoints using a validated food frequency questionnaire (FFQ) and calcium supplement use was recorded. Blood lipids were treated as categorical variables to distinguish those with normal and abnormal levels. Increasing total calcium intake was associated with lower risks for high total cholesterol (p = 0.038) and triglycerides (p = 0.007), and low HDL-cholesterol (p = 0.010). Dairy calcium (p = 0.031), especially calcium from milk (p = 0.044) and milk-based desserts (p = 0.039), i.e., low-fat (p = 0.022) and non-fermented (p = 0.005) dairy products, were associated with a lower risk of high total cholesterol. Greater calcium intakes from total dairies (p = 0.020), milk (p = 0.020) and non-fermented dairies (p = 0.027) were associated with a lower risk of hypertriglyceridemia. No association was observed between calcium from non-dairy sources, cheese or high-fat dairies and blood lipids. Increasing calcium through supplements was associated with lower risks for hypertriglyceridemia (p = 0.022) and low HDL-cholesterol (p = 0.001), but not after adjustments. Our results suggest that higher calcium intakes from dietary sources or supplements are not adversely associated with blood lipids in the elderly, whilst total, and particularly low-fat, dairy products are valuable calcium sources potentially related to favorable lipid profiles.