In this arXiv pre-print, we prescribe a procedure for measuring Circuit Layer Operations Per Second (CLOPS) and use it to characterize the performance of some IBM Quantum systems.
The IBM Quantum team has accomplished a great deal this past year. We’ve made strides progressing along our development roadmap, and plan to achieve frictionless quantum computing on a 1,000+ system by the end of 2023, which we expect to be an inflection point in the field’s development.
Our guiding principle and goal as we advance our quantum computing systems is to increase the amount of useful work that these systems can accomplish—simply put, quantum computing performance. Performance requires three critical attributes: Scale, Quality, and Speed. Bringing quantum computers into organizations’ computing workflows requires pushing on all three areas, which our superconducting quantum systems at IBM Quantum are poised to lead.
For scale, we measure our progress by the number of qubits in our systems. We released Hummingbird with 65 qubits last year and are on track to deliver Eagle with 127 qubits this year, thanks to continuous innovations in scaling technology. With quality, we use Quantum Volume1 as we defined previously, and offer multiple systems with QV 128 to users today. Speed is an often-discussed property with real implications on application workloads, but an appropriate system-agnostic metric which captures the full dependencies across hardware and software of circuit executions has yet to emerge.
What Is Quantum Volume, anyway? Quantum Volume tests how well a quantum computer can run a circuit consisting of random two-qubit gates acting in parallel on a subset of the device’s qubits. Read more.
Today, we would like to propose a metric we call the Circuit Layer Operations Per Second, or CLOPS.2
The three key metrics for measuring quantum computing performance: Scale. Quality. Speed.
Bringing practicality to quantum computing with Circuit Layer Operations Per Second
As quantum computing evolves and begins to tackle practical problems, we must pay greater attention to how much work quantum computing systems can do in a given unit of time. We expect real workloads to involve quantum-classical interactions—a full program will invoke a quantum processor as an accelerator for certain tasks, or an algorithm will require multiple calls to a quantum processor. Consequently, the runtime system that allows for efficient quantum-classical communication will be critical to achieving high performance. We have embedded this runtime interaction in our proposal for the CLOPS benchmark.
CLOPS is a metric correlated with how fast a quantum processor can execute circuits.
CLOPS is a metric correlated with how fast a quantum processor can execute circuits—specifically, the metric measures the speed the processor can execute layers of a parameterized model circuit of the same sort used to measure Quantum Volume.
Increased quantum processor speed is critical to support near-term algorithms based on the variational method, which requires thousands of iterations. The improved qubit gate times have enabled us to greatly expand the reach of current quantum systems and bring us closer to outpacing classical computing hardware. —Pranav Gokhale, Founder and CEO, Super.tech
Runtime architecture and phases of compilation. The circuit pattern of a Quantum Volume benchmark is shown, as well as its offline compilation. Circuit parameters in the Circuit Layer Operations per Second benchmark are updated during runtime, making the metric heavily dependent on the runtime architecture and runtime compilation.
Quantum circuits are the basic unit of computation for quantum computers, much like logic circuits for classical computing. The benchmark requires execution of many instances of this model circuit with different parameters generated at runtime. Various parts of this hardware-software stack contribute to CLOPS, including the repetition rate of the quantum processor, the speed at which gates run, the runtime compilation, the amount of time it takes to generate the classical control instructions, and finally, the data transfer rate among all units.
Delivering the highest performance quantum computing systems required us to entirely rethink the architecture governing how we run a quantum computing program—and hence, why we introduced Qiskit Runtime.
In May of 2021, we demonstrated a 120x speedup in simulating molecules thanks to a host of improvements, including the ability to run quantum programs entirely on the cloud with Qiskit Runtime.
Qiskit Runtime is a portable, secure, containerized architecture that runs quantum programs on a classical computing unit tightly integrated with the quantum processor. Qiskit Runtime allows the quantum computer to become a part of any computing environment to accelerate computation—similar to a GPU—and handles the job orchestration and data transfer to the quantum processing unit, maximizing efficiency.
Today, our fastest systems can run up to 1,400 circuit layer operations per second.
While our systems already benefit from the advances in runtime architecture we made this year with the Qiskit Runtime, initial timing measurements using the benchmark have also identified several new speed bottlenecks. Improvements in quantum hardware will reduce circuit delay times—idle times between consecutive circuits—and further advances in the runtime architecture will reduce initialization times for data loading as well as improved runtime compilation.
Superconducting qubits are the natural choice for high-performance quantum computation.
Our goal is to develop practical quantum computing, and we believe that our superconducting qubit systems provide the best chance to drive quantum computing ubiquity. Other quantum architectures can achieve high performance in some, but not all, of the attributes of scale, quality, and speed. For example, trapped ions have shown the ability to achieve high Quantum Volume, but face challenges while tackling speed, while spin qubits can achieve high speeds but thus far face challenges in their ability to push quality or scale.
We expect that when it comes to continued achievement of performance gains across scalability, quality, and speed, superconducting qubits provide the most opportunities for continued growth in all three.
In fact, we're already seeing the benefits of speed play out with real scientific demonstrations. Back in 2017, our work modeling the lithium hydride molecule3 required running 4.8 billion quantum circuits would require months to years on our previous commercial stack. But now, we can perform this calculation 120 times faster with Qiskit Runtime and other improvements.
If we hope to accelerate the adoption of quantum computers, we need to focus on the useful work that quantum computers can do, and we need to relentlessly improve across all three key performance areas:
- We are committed to our roadmap on scaling, and Falcon (27), Hummingbird (65) and soon Eagle (127) demonstrate that.
- Quality, we are actively pushing through core research improving the underlying coherence and gate errors of the superconducting qubits.
- We have now introduced CLOPS to round out this triumvirate of performance measures.
We’re confident that through our choice of the superconducting qubit architecture and the introduction of the Qiskit Runtime we’re going to see even greater performance gains in the near future as we continue to execute our development roadmap.
