Since the very first experiments, superconducting circuits have suffered from strong coupling to environmental noise, destroying quantum coherence and degrading performance. In state-of-the-art experiments, it is found that the relaxation time of superconducting qubits fluctuates as a function of time. We present measurements of such fluctuations in a 3D-transmon circuit and develop a qualitative model based on interactions within a bath of background two-level systems (TLS) which emerge from defects in the device material. In our model, the time-dependent noise density acting on the qubit emerges from its near-resonant coupling to high-frequency TLS which experience energy fluctuations due to their interaction with thermally fluctuating TLS at low frequencies. We support the model by providing experimental evidence of such energy fluctuations observed in a single TLS in a phase qubit circuit.