Can analog in-memory accelerators provide sufficient accuracy for AI applications under ambient temperature variations? Here, we answer this question by focusing on phase-change memory (PCM)-based deep learning acceleration. We investigate for the first time the impact of temperature on multi-level PCM conductance states used to store the synaptic weights. First, we characterize the temperature and drift behavior of 10,000 doped Ge2Sb2Te5 (GST)-based mushroom PCM. Next, we present a model which can capture this behavior and faithfully reproduce the complete time-temperature dependence of the conductance states. Finally, we experimentally study the sensitivity of various network architectures to ambient temperature variations. For this, we employ a multi-layer perceptron, a convolutional neural network and a recurrent neural network, with more than 1.1M PCM weights. We demonstrate that a simple array-level scaling could correct for the conductance shift due to temperature and drift and prevent any significant accuracy drop for all the studied networks during inference.