The fovea is one of the most important anatomical landmarks in the eye and its localization is required in automated analysis of retinal diseases due to its role in sharp central vision. In this paper, we propose a two-stage deep learning framework for accurate segmentation of the fovea in retinal colour fundus images. In the first stage, coarse segmentation is performed to localize the fovea in the fundus image. The location information from the first stage is then used to perform fine-grained segmentation of the fovea region in the second stage. The proposed method performs end-to-end pixelwise segmentation by creating a deep learning model based on fully convolutional neural networks, which does not require the prior knowledge of the location of other retinal structures such as optic disc (OD) and vasculature geometry. We demonstrate the effectiveness of our method on a dataset with 400 retinal images with average localization error of 14 ± 7 pixels.