With the growth of cloud computing, resiliency of cloud is critical for enterprises’ business. However, the continuous-changing of cloud makes evaluation of cloud resiliency more difficult. In this study, we design a methodology for automatic and continuous evaluation of cloud resiliency and implement it in a tool called CRGauge. The Continuous Evaluation Model methodology leverages fault injection techniques to inject faults and an open-source library to set up synthetic workloads for the test campaign. Our experiment results on OpenStack cloud platform show that resiliency of OpenStack is needed to be improved especially in heavy workloads.