Quantum Volume Testing: Setting the Steady Pace to Higher Performing Devices
When it comes to completing the statistical tests and other steps necessary for calculating quantum volume, few people have as much as experience as Dr. Charlie Baldwin.
Baldwin, a lead physicist at ҹɫֱ, and his team have performed the tests numerous times on three different H-Series quantum computers, which have set six industry records for measured quantum volume since 2020.
Quantum volume is a benchmark developed by IBM in 2019 to measure the overall performance of a quantum computer regardless of the hardware technology. (ҹɫֱ builds trapped ion systems).
Baldwin’s experience with quantum volume prompted him to share what he’s learned and suggest ways to improve the benchmark in a peer-reviewed paper published this week in .
“We’ve learned a lot by running these tests and believe there are ways to make quantum volume an even stronger benchmark,” Baldwin said.
We sat down with Baldwin to discuss quantum volume, the paper, and the team’s findings.
How is quantum volume measured? What tests do you run?
Quantum volume is measured by running many randomly constructed circuits on a quantum computer and comparing the outputs to a classical simulation. The circuits are chosen to require random gates and random connectivity to not favor any one architecture. We follow the construction proposed by IBM to build the circuits.
What does quantum volume measure? Why is it important?
In some sense, quantum volume only measures your ability to run the specific set of random quantum volume circuits. That probably doesn’t sound very useful if you have some other application in mind for a quantum computer, but quantum volume is sensitive to many aspects that we believe are key to building more powerful devices.
Quantum computers are often built from the ground up. Different parts—for example, single- and two-qubit gates—have been developed independently over decades of academic research. When these parts are put together in a large quantum circuit, there’re often other errors that creep in and can degrade the overall performance. That’s what makes full-system tests like quantum volume so important; they’re sensitive to these errors.
Increasing quantum volume requires adding more qubits while simultaneously decreasing errors. Our quantum volume results demonstrate all the amazing progress ҹɫֱ has made at upgrading our trapped-ion systems to include more qubits and identifying and mitigating errors so that users can expect high-fidelity performance on many other algorithms.
You’ve been running quantum volume tests since 2020. What is your biggest takeaway?
I think there’re a couple of things I’ve learned. First, quantum volume isn’t an easy test to run on current machines. While it doesn’t necessarily require a lot of qubits, it does have fairly demanding error requirements. That’s also clear when comparing progress in quantum volume tests across different platforms, .
Second, I’m always impressed by the continuous and sustained performance progress that our hardware team achieves. And that the progress is actually measurable by using the quantum volume benchmark.
The hardware team has been able to push down many different error sources in the last year while also running customer jobs. This is proven by the quantum volume measurement. For example, H1-2 launched in Fall 2021 with QV=128. But since then, the team has implemented many performance upgrades, recently achieving QV=4096 in about 8 months while also running commercial jobs.
What are the key findings from your paper?
The paper is about four small findings that when put together, we believe, give a clearer view of the quantum volume test.
First, we explored how compiling the quantum volume circuits scales with qubit number and, also proposed using arbitrary angle gates to improve performance—an optimization that many companies are currently exploring.
Second, we studied how quantum volume circuits behave without errors to better relate circuit results to ideal performance.
Third, we ran many numerical simulations to see how the quantum volume test behaved with errors and constructed a method to efficiently estimate performance in larger future systems.
Finally, and I think most importantly, we explored what it takes to meet the quantum volume threshold and what passing it implies about the ability of the quantum computer, especially compared to the requirements for quantum error correction.
What does it take to “pass” the quantum volume threshold?
Passing the threshold for quantum volume is defined by the results of a statistical test on the output of the circuits called the heavy output test. The result of the heavy output test—called the heavy output probability or HOP—must have an uncertainty bar that clears a threshold (2/3).
Originally, IBM constructed a method to estimate that uncertainty based on some assumptions about the distribution and number of samples. They acknowledged that this construction was likely too conservative, meaning it made much larger uncertainty estimates than necessary.
We were able to verify this with simulations and proposed a different method that constructed much tighter uncertainty estimates. We’ve verified the method with numerical simulations. The method allows us to run the test with many fewer circuits while still having the same confidence in the returned estimate.
How do you think the quantum volume test can be improved?
Quantum volume has been criticized for a variety of reasons, but I think there’s still a lot to like about the test. Unlike some other full-system tests, quantum volume has a well-defined procedure, requires challenging circuits, and sets reasonable fidelity requirements.
However, it still has some room for improvement. As machines start to scale up, runtime will become an important dimension to probe. IBM has proposed a metric for measuring run time of quantum volume tests (CLOPS). We also agree that the duration of the computation is important but that there should also be tests that balance run time with fidelity, sometimes called ‘time-to-solution.”
Another aspect that could be improved is filling the gap between when quantum volume is no longer feasible to run—at around 30 qubits—and larger machines. There’s recent work in this area that will be interesting to compare to quantum volume tests.
You presented these findings to IBM researchers who first proposed the benchmark. How was that experience?
It was great to talk to the experts at IBM. They have so much knowledge and experience on running and testing quantum computers. I’ve learned a lot from their previous work and publications.
There is a lot of debate about quantum volume and how long it will be a useful benchmark. What are your thoughts?
The current iteration of quantum volume definitely has an expiration date. It’s limited by our ability to classically simulate the system, so being unable to run quantum volume actually is a goal for quantum computing development. Similarly, quantum volume is a good measuring stick for early development.
Building a large-scale quantum computer is an incredibly challenging task. Like any large project, you break the task up into milestones that you can reach in a reasonable amount of time.
It's like if you want to run a marathon. You wouldn’t start your training by trying to run a marathon on Day 1. You’d build up the distance you run every day at a steady pace. The quantum volume test has been setting our pace of development to steadily reach our goal of building ever higher performing devices.
About ҹɫֱ
ҹɫֱ, the world’s largest integrated quantum company, pioneers powerful quantum computers and advanced software solutions. ҹɫֱ’s technology drives breakthroughs in materials discovery, cybersecurity, and next-gen quantum AI. With over 500 employees, including 370+ scientists and engineers, ҹɫֱ leads the quantum computing revolution across continents.
Back in 2020, we to increase our Quantum Volume (QV), a measure of computational power, by 10x per year for 5 years.
Today, we’re pleased to share that we’ve followed through on our commitment: Our System Model H2 has reached a Quantum Volume of 2²³ = 8,388,608, proving not just that we always do what we say, but that our quantum computers are leading the world forward.
The QV benchmark was developed by IBM to represent a machine’s performance, accounting for things like qubit count, coherence times, qubit connectivity, and error rates. :
“the higher the Quantum Volume, the higher the potential for exploring solutions to real world problems across industry, government, and research."
Our announcement today is precisely what sets us apart from the competition. No one else has been bold enough to make a similar promise on such a challenging metric – and no one else has ever completed a five-year goal like this.
We chose QV because we believe it’s a great metric. For starters, it’s not gameable, like other metrics in the ecosystem. Also, it brings together all the relevant metrics in the NISQ era for moving towards fault tolerance, such as gate fidelity and connectivity.
Our path to achieve a QV of over 8 million was led in part by Dr. Charlie Baldwin, who studied under the legendary Ivan H. Deutsch. Dr. Baldwin has made his name as a globally renowned expert in quantum hardware performance over the past decade, and it is because of his leadership that we don’t just claim to be the best, but that we can prove we are the best.
Sources: [][][][][]
Alongside the world’s biggest quantum volume, we have the industry’s . To that point, the table below breaks down the leading commercial specs for each quantum computing architecture.
We’ve never shied away from benchmarking our machines, because we know the results will be impressive. It is our provably world-leading performance that has enabled us to demonstrate:
As we look ahead to our next generation system, Helios, ҹɫֱ’s Senior Director of Engineering, Dr. Brian Neyenhuis, reflects: “We finished our five-year commitment to Quantum Volume ahead of schedule, showing that we can do more than just maintain performance while increasing system size. We can improve performance while scaling.”
Helios’ performance will exceed that of our previous machines, meaning that ҹɫֱ will continue to lead in performance while following through on our promises.
As the undisputed industry leader, we’re racing against no one other than ourselves to deliver higher performance and to better serve our customers.
At the heart of quantum computing’s promise lies the ability to solve problems that are fundamentally out of reach for classical computers. One of the most powerful ways to unlock that promise is through a novel approach we call Generative Quantum AI, or GenQAI. A key element of this approach is the (GQE).
GenQAI is based on a simple but powerful idea: combine the unique capabilities of quantum hardware with the flexibility and intelligence of AI. By using quantum systems to generate data, and then using AI to learn from and guide the generation of more data, we can create a powerful feedback loop that enables breakthroughs in diverse fields.
Unlike classical systems, our quantum processing unit (QPU) produces data that is extremely difficult, if not impossible, to generate classically. That gives us a unique edge: we’re not just feeding an AI more text from the internet; we’re giving it new and valuable data that can’t be obtained anywhere else.
The Search for Ground State Energy
One of the most compelling challenges in quantum chemistry and materials science is computing the properties of a molecule’s ground state. For any given molecule or material, the ground state is its lowest energy configuration. Understanding this state is essential for understanding molecular behavior and designing new drugs or materials.
The problem is that accurately computing this state for anything but the simplest systems is incredibly complicated. You cannot even do it by brute force—testing every possible state and measuring its energy—because the number of quantum states grows as a double-exponential, making this an ineffective solution. This illustrates the need for an intelligent way to search for the ground state energy and other molecular properties.
That’s where GQE comes in. GQE is a methodology that uses data from our quantum computers to train a transformer. The transformer then proposes promising trial quantum circuits; ones likely to prepare states with low energy. You can think of it as an AI-guided search engine for ground states. The novelty is in how our transformer is trained from scratch using data generated on our hardware.
Here's how it works:
- We start with a batch of trial quantum circuits, which are run on our QPU.
- Each circuit prepares a quantum state, and we measure the energy of that state with respect to the Hamiltonian for each one.
- Those measurements are then fed back into a transformer model (the same architecture behind models like GPT-2) to improve its outputs.
- The transformer generates a new distribution of circuits, biased toward ones that are more likely to find lower energy states.
- We sample a new batch from the distribution, run them on the QPU, and repeat.
- The system learns over time, narrowing in on the true ground state.
To test our system, we tackled a benchmark problem: finding the ground state energy of the hydrogen molecule (H₂). This is a problem with a known solution, which allows us to verify that our setup works as intended. As a result, our GQE system successfully found the ground state to within chemical accuracy.
To our knowledge, we’re the first to solve this problem using a combination of a QPU and a transformer, marking the beginning of a new era in computational chemistry.
The Future of Quantum Chemistry
The idea of using a generative model guided by quantum measurements can be extended to a whole class of problems—from to materials discovery, and potentially, even drug design.
By combining the power of quantum computing and AI we can unlock their unified full power. Our quantum processors can generate rich data that was previously unobtainable. Then, an AI can learn from that data. Together, they can tackle problems neither could solve alone.
This is just the beginning. We’re already looking at applying GQE to more complex molecules—ones that can’t currently be solved with existing methods, and we’re exploring how this methodology could be extended to real-world use cases. This opens many new doors in chemistry, and we are excited to see what comes next.
Last year, we joined forces with RIKEN, Japan's largest comprehensive research institution, to install our hardware at RIKEN’s campus in Wako, Saitama. This deployment is part of RIKEN’s project to build a quantum-HPC hybrid platform consisting of high-performance computing systems, such as the supercomputer Fugaku and ҹɫֱ Systems.
Today, marks the first of many breakthroughs coming from this international supercomputing partnership. The team from RIKEN and ҹɫֱ joined up with researchers from Keio University to show that quantum information can be delocalized (scrambled) using a quantum circuit modeled after periodically driven systems.
"Scrambling" of quantum information happens in many quantum systems, from those found in complex materials to black holes. Understanding information scrambling will help researchers better understand things like thermalization and chaos, both of which have wide reaching implications.
To visualize scrambling, imagine a set of particles (say bits in a memory), where one particle holds specific information that you want to know. As time marches on, the quantum information will spread out across the other bits, making it harder and harder to recover the original information from local (few-bit) measurements.
While many classical techniques exist for studying complex scrambling dynamics, quantum computing has been known as a promising tool for these types of studies, due to its inherently quantum nature and ease with implementing quantum elements like entanglement. The joint team proved that to be true with their latest result, which shows that not only can scrambling states be generated on a quantum computer, but that they behave as expected and are ripe for further study.
Thanks to this new understanding, we now know that the preparation, verification, and application of a scrambling state, a key quantum information state, can be consistently realized using currently available quantum computers. Read the paper , and read more about our partnership with RIKEN here.