Solar energy penetration both at utility scale and residential scale has been increasing at an exponential rate. However, its stochastic nature poses great challenge to power grid operation. Knowing how much solar energy generation in advance is vital for power grid balancing, planning and optimization. Therefore, solar energy generation forecast is essential for the stability and operation efficiency of today's smart grid. Although the sun path and energy can be computed with physical laws, the prediction of solar energy generation and production remains very challenging problem both in the field of physical simulation and artificial intelligence. The main reason lies in the fact that the actual solar production are impacted by many factors including the sun position, weather condition and the characteristics of photovoltaic panel, curtailment, etc. Especially in cloudy day, where the cloud movement becomes the main factor in solar energy production. However, predicting the movement of cloud is extremely difficult. In this paper, we present several deep convolutional neural networks utilizing high resolution weather forecast data exploring various temporal and spatial connectivities to capture the cloud movement pattern and its effect on forecasting solar energy generation for solar farms. Comparing with state-of-the-art forecast error rate, we have been able to reduce the error rate from about 21% in the persistent model, to 15.1% from the SVR model, and to 11.8% from the convolutional neural networks. These improvements have significant impact on the healthy growth of the solar energy industry, will save billions of dollars for the US utilities and most importantly reduce dependency on fossil fuel and reduction in CO2 emission.