Estimation of cloud motion from all-sky image sequences is a very big challenge in the meteorological area that requires extensively study. In this paper, we propose a novel local features-based method to track clouds and to forecast short-term cloudiness up to fifteen minutes ahead using all-sky image sequences. This method allows us to estimate the cloud displacement as well as the change of size scale with high accuracy and robustness, leading to a significant improvement in the precision of cloud image prediction. The proposed Short-term All-sky Image Prediction System (SAIPS) includes three key steps: detection of clouds based on clustering, matching of the clouds from captured sequences, and analysis of cloud tracks to predict their locations. The effectiveness of the proposed system is verified using a database of captured images of real clouds. Experimental results demonstrate a better performance of the proposed approach compared to other algorithms.