The field- and current-induced magnetization reversal of a Co-Fe-B layer in a perpendicular magnetic tunnel junction (pMTJ) is studied at room temperature. The magnetization switching probability from the parallel (P) state to the antiparallel (AP) state and from AP to P is found to be asymmetric. We observe that this asymmetry depends on the magnetic configuration of the synthetic antiferromagnetic (SAF). The state diagram and the energy landscape are compared for two SAF configurations. We conclude that the asymmetry is due to the inhomogeneity of the fringe field generated by the SAF. Our study highlights the role of the reference-layer fringe field on the free layer's energy barrier height, which has to be carefully controlled for memory applications.