Cavity and circuit QED study light-matter interaction at its most fundamental level. Yet, this interaction is most often neglected when considering the coupling of this system with an environment. In this paper, we show how this simplification, which leads to the standard quantum optics master equation, is at the root of unphysical effects. Including qubit relaxation and dephasing, and cavity relaxation, we derive a master equation that takes into account the qubit-resonator coupling. Special attention is given to the ultrastrong coupling regime, where the failure of the quantum optical master equation is manifest. In this situation, our model predicts an asymmetry in the vacuum Rabi splitting that could be used to probe dephasing noise at unexplored frequencies. We also show how fluctuations in the qubit frequency can cause sideband transitions, squeezing, and Casimir-like photon generation. © 2011 American Physical Society.