Hardware efficient transpilation of quantum circuits is essential for the execution of quantum algorithms on noisy quantum computers. However, typical devices limit their gate set to a single Clifford two-qubit gate, e.g. the CNOT gate, which is easy to characterize via randomized benchmarking. Yet, for some applications, access to a non-Clifford two-qubit gate can result in more optimal circuit decompositions and also allows more flexibility in optimizing over noise. Furthermore, recent methods have been proposed for benchmarking gates in the CNOT-Dihedral group, which include gates such as the control pi/2 gate (CS). To that goal, here we demonstrate calibration of a high fidelity CS gate on an IBM cloud device. To measure the gate error of the calibrated CS gate we perform the first experimental demonstration of non-Clifford CNOT-Dihedral interleaved randomized benchmarking. We are able to obtain a low gate error close to the coherence limit of the associated qubits, and lower error than the backends standard calibrated CNOT gate. The experiments were implemented entirely using open source software tools available in the Qiskit software library, and run on a cloud accessible quantum computer. Hence the presented techniques can be readily employed by other researchers.