In high temperature superconductors, although some phenomena such as the Mott transition (MT) at low doping are clearly driven by electron correlations, recent experimental data imply that anharmonic oxygen degrees of freedom - characteristic of perovskite materials - are playing a significant role. A key test of the role of anharmonic oxygen is to reproduce the complex cuprate phase diagram from a simple model. Here, we show that a field theory based on nonlinear coupling to anharmonic oxygens, parametrized from ab initio calculations, quantitatively reproduces the cuprate phase diagram for dopings above the MT. Pairing is mediated by renormalized oxygen vibrations transmuted into excitations of the pseudogap. The observed strong dependence of gap to transition temperature ratio on Tc also emerges from this field theory. This work suggests that including vibrational degrees of freedom is key to developing a complete understanding of the cuprates.