In this work, I investigate the effect of the swapping time frequency in parallel tempering (or replica exchange molecular dynamics, REMD) on the sampled equilibrium distributions at the different temperatures. By means of a simple deterministic Ising model originally introduced by M. Creutz (Phys. Rev. Lett., 1983, 50, 1411) I show that high frequency swaps can induce a systematic bias on the sampled REMD equilibrium distributions. The origin of this failure is ascribed to correlation effects among close temperature configurations used to evaluate REMD swapping probabilities. These results should serve as a monitor for using too frequent swapping attempts in parallel tempering simulations of generic Hamiltonians, including the ones used in atomistic simulations.