Detection of calcified plaques in coronary arteries is helpful in cardiovascular disease risk assessment. This is often performed by radiologists on computed tomography (CT) images. We work towards an automatic solution for calcium detection in CT images. Most of previous work in this area combines CT and CTA for this purpose to facilitate the localization of the coronary arteries. Given the cost and dose advantages of using only CT scan compared to using both CT and CTA, we propose a solution for automatic calcium assessment in CT. We model the whole chest including all heart chambers and main arteries. Instead of localizing calcium candidates with respect to the coronary artery alone, we assess their position with respect to eight other anatomies, segmented from CT images using joint atlas label fusion methodology. This comprehensive spatial information together with other types of features such as shape, size and texture of each calcium candidate is used with a random forest classifier trained on 104 patients to detect coronary calcification. The results show that our method has a precision of 95.1% and a recall of 89.0% in classifying calcium candidates found based on thresholding. In the patient level, using this method, all the test patients with true calcification were detected as positive, yielding a patient level sensitivity of 100%. Among the test patients without calcification, 44 out of 56 patients resulted in no calcium finding, yielding a patient level specificity of 78.6%. We quantified the whole heart Agatston score for the manual and the automatically detected calcium on the 22 diseased test cases, and found a Pearson correlation coefficient of 0.98. These results show that our proposed framework can reliably detect calcification using CT data.