Citrus producers need to monitor orchards frequently, and would benefit greatly from having automated tools to analyze aerial images acquired by drones over the plantations. However, analysing large aerial data sets to enable producers to take management decisions that would optimize productivity and sustainability over time and space remains challenging. Motivated by the success of deep learning in computer vision, this work proposes a novel approach based on Fully Convolutional Regression Networks and Multi-Task Learning to detect individual full-grown trees, tree seedlings, and tree gaps in citrus orchards for inventory tracking. We show that the proposal can identify eight-year-old orange trees with accuracy between 95-99% in high-density commercial plantations where adjacent crowns overlap. This quality of detection was achieved on RGB orthomosaics with a pixel size of about 9.5 cm and requires the nominal spacing between adjacent trees as a priori information. Our results also highlight that detecting tree seedlings and tree gaps remains a challenge. For these two categories, classification sensitivity (recall) was between 59-100% and 63-94%, respectively.