In the paper, we present an approach to efficiently summarizing UAV video data. Our approach is based on first detecting and tracking moving objects. Significant camera motion usually present in UAV video data is successfully handled by a robust feature-based frame registration technique. We then devise a saliency-based scoring method to score and rank detected object tracks. Object tracks are then grouped into video segments. The final step is to generate a concise summarization and visualization. Experimental results on the VIRAT UAV dataset show that we can accomplish a data reduction rate in excess of 1000 without significantly missing any activities of interest. © 2012 ICPR Org Committee.