QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation

Li, Ang; Stein, Samuel; Krishnamoorthy, Sriram; Ang, James

doi:10.1145/3550488

QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation

Journal Article · Fri Feb 24 00:00:00 EST 2023 · ACM Transactions on Quantum Computing

DOI:https://doi.org/10.1145/3550488· OSTI ID:1969005

^[1]; ^[1]; ^[1]; ^[1]

Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

The rapid development of quantum computing (QC) in the NISQ era urgently demands a low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers and assemblers, and the capability of quantum system simulators in a classical computer. In this work, we fill this gap by proposing a low-level, easy-to-use benchmark suite called QASMBench based on the OpenQASM assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, and so on, trading-off between generality and usability. To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. Applications in QASMBench can be launched and verified on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum. For evaluation, we measure the execution fidelity of a subset of QASMBench applications on 12 IBM-Q machines through density matrix state tomography, comprising 25K circuit evaluations. In addition we also compare the fidelity of executions among the IBM-Q machines, the IonQ QPU and the Rigetti Aspen M-1 system.

View Accepted Manuscript (DOE)

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC05-76RL01830

OSTI ID:: 1969005

Report Number(s):: PNNL-SA-162867

Journal Information:: ACM Transactions on Quantum Computing, Journal Name: ACM Transactions on Quantum Computing Journal Issue: 2 Vol. 4; ISSN 2643-6809

Publisher:: Association for Computing MachineryCopyright Statement

Country of Publication:: United States

Language:: English

References (78)

The Physical Implementation of Quantum Computation DiVincenzo, David P. Fortschritte der Physik, Vol. 48, Issue 9-11 https://doi.org/10.1002/1521-3978(200009)48:9/11<771::AID-PROP771>3.0.CO;2-E	journal	September 2000
Reversible computing Toffoli, Tommaso Automata, Languages and Programming https://doi.org/10.1007/3-540-10003-2_104	book	January 1980
Quantum Counterfeit Coin Problems Iwama, Kazuo; Nishimura, Harumichi; Raymond, Rudy Algorithms and Computation https://doi.org/10.1007/978-3-642-17517-6_10	book	January 2010
Going Beyond Bell’s Theorem Greenberger, Daniel M.; Horne, Michael A.; Zeilinger, Anton Bell’s Theorem, Quantum Theory and Conceptions of the Universe https://doi.org/10.1007/978-94-017-0849-4_10	book	January 1989
The computer as a physical system: A microscopic quantum mechanical Hamiltonian model of computers as represented by Turing machines Benioff, Paul Journal of Statistical Physics, Vol. 22, Issue 5 https://doi.org/10.1007/BF01011339	journal	May 1980
Efficient Quantum Algorithms for Simulating Sparse Hamiltonians Berry, Dominic W.; Ahokas, Graeme; Cleve, Richard Communications in Mathematical Physics, Vol. 270, Issue 2 https://doi.org/10.1007/s00220-006-0150-x	journal	December 2006
Quantum walks: a comprehensive review Venegas-Andraca, Salvador Elías Quantum Information Processing, Vol. 11, Issue 5 https://doi.org/10.1007/s11128-012-0432-5	journal	July 2012
Cloud based QC with Amazon Braket Gonzalez, Constantin Digitale Welt, Vol. 5, Issue 2 https://doi.org/10.1007/s42354-021-0330-z	journal	March 2021
Benchmarking gate-based quantum computers Michielsen, Kristel; Nocon, Madita; Willsch, Dennis Computer Physics Communications, Vol. 220 https://doi.org/10.1016/j.cpc.2017.06.011	journal	November 2017
Swift: A language for distributed parallel scripting Wilde, Michael; Hategan, Mihael; Wozniak, Justin M. Parallel Computing, Vol. 37, Issue 9 https://doi.org/10.1016/j.parco.2011.05.005	journal	September 2011
A language and hardware independent approach to quantum–classical computing McCaskey, A. J.; Dumitrescu, E. F.; Liakh, D. SoftwareX, Vol. 7 https://doi.org/10.1016/j.softx.2018.07.007	journal	January 2018
Quantum counterfeit coin problems Iwama, Kazuo; Nishimura, Harumichi; Raymond, Rudy Theoretical Computer Science, Vol. 456 https://doi.org/10.1016/j.tcs.2012.05.039	journal	October 2012
Creation of a six-atom ‘Schrödinger cat’ state Leibfried, D.; Knill, E.; Seidelin, S. Nature, Vol. 438, Issue 7068 https://doi.org/10.1038/nature04251	journal	December 2005
The quantum internet Kimble, H. J. Nature, Vol. 453, Issue 7198 https://doi.org/10.1038/nature07127	journal	June 2008
Superconducting quantum bits Clarke, John; Wilhelm, Frank K. Nature, Vol. 453, Issue 7198 https://doi.org/10.1038/nature07128	journal	June 2008
A single-atom electron spin qubit in silicon Pla, Jarryd J.; Tan, Kuan Y.; Dehollain, Juan P. Nature, Vol. 489, Issue 7417 https://doi.org/10.1038/nature11449	journal	September 2012
Tunable two-dimensional arrays of single Rydberg atoms for realizing quantum Ising models Labuhn, Henning; Barredo, Daniel; Ravets, Sylvain Nature, Vol. 534, Issue 7609 https://doi.org/10.1038/nature18274	journal	June 2016
Quantum machine learning Biamonte, Jacob; Wittek, Peter; Pancotti, Nicola Nature, Vol. 549, Issue 7671 https://doi.org/10.1038/nature23474	journal	September 2017
Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets Kandala, Abhinav; Mezzacapo, Antonio; Temme, Kristan Nature, Vol. 549, Issue 7671 https://doi.org/10.1038/nature23879	journal	September 2017
A CMOS silicon spin qubit Maurand, R.; Jehl, X.; Kotekar-Patil, D. Nature Communications, Vol. 7, Issue 1 https://doi.org/10.1038/ncomms13575	journal	November 2016
Photonic quantum technologies O'Brien, Jeremy L.; Furusawa, Akira; Vučković, Jelena Nature Photonics, Vol. 3, Issue 12 https://doi.org/10.1038/nphoton.2009.229	journal	December 2009
From quantum multiplexing to high-performance quantum networking Munro, W. J.; Harrison, K. A.; Stephens, A. M. Nature Photonics, Vol. 4, Issue 11 https://doi.org/10.1038/nphoton.2010.213	journal	August 2010
Photonic quantum simulators Aspuru-Guzik, Alán; Walther, Philip Nature Physics, Vol. 8, Issue 4 https://doi.org/10.1038/nphys2253	journal	April 2012
Quantum supremacy using a programmable superconducting processor Arute, Frank; Arya, Kunal; Babbush, Ryan Nature, Vol. 574, Issue 7779 https://doi.org/10.1038/s41586-019-1666-5	journal	October 2019
Google in a Quantum Network Paparo, G. D.; Martin-Delgado, M. A. Scientific Reports, Vol. 2, Issue 1 https://doi.org/10.1038/srep00444	journal	June 2012
An introduction to quantum machine learning Schuld, Maria; Sinayskiy, Ilya; Petruccione, Francesco Contemporary Physics, Vol. 56, Issue 2 https://doi.org/10.1080/00107514.2014.964942	journal	October 2014
Simulation of electronic structure Hamiltonians using quantum computers Whitfield, James D.; Biamonte, Jacob; Aspuru-Guzik, Alán Molecular Physics, Vol. 109, Issue 5 https://doi.org/10.1080/00268976.2011.552441	journal	March 2011
The theory of variational hybrid quantum-classical algorithms McClean, Jarrod R.; Romero, Jonathan; Babbush, Ryan New Journal of Physics, Vol. 18, Issue 2 https://doi.org/10.1088/1367-2630/18/2/023023	journal	February 2016
Learning the quantum algorithm for state overlap Cincio, Lukasz; Subaşı, Yiğit; Sornborger, Andrew T. New Journal of Physics, Vol. 20, Issue 11 https://doi.org/10.1088/1367-2630/aae94a	journal	November 2018
OpenFermion: the electronic structure package for quantum computers McClean, Jarrod R.; Rubin, Nicholas C.; Sung, Kevin J. Quantum Science and Technology, Vol. 5, Issue 3 https://doi.org/10.1088/2058-9565/ab8ebc	journal	June 2020
Validating quantum computers using randomized model circuits Cross, Andrew W.; Bishop, Lev S.; Sheldon, Sarah Physical Review A, Vol. 100, Issue 3 https://doi.org/10.1103/PhysRevA.100.032328	journal	September 2019
Quantum networks for elementary arithmetic operations Vedral, Vlatko; Barenco, Adriano; Ekert, Artur Physical Review A, Vol. 54, Issue 1 https://doi.org/10.1103/PhysRevA.54.147	journal	July 1996
Quantum computation and decision trees Farhi, Edward; Gutmann, Sam Physical Review A, Vol. 58, Issue 2 https://doi.org/10.1103/PhysRevA.58.915	journal	August 1998
Three qubits can be entangled in two inequivalent ways Dür, W.; Vidal, G.; Cirac, J. I. Physical Review A, Vol. 62, Issue 6 https://doi.org/10.1103/PhysRevA.62.062314	journal	November 2000
Quantum memory for photons: Dark-state polaritons Fleischhauer, M.; Lukin, M. D. Physical Review A, Vol. 65, Issue 2 https://doi.org/10.1103/PhysRevA.65.022314	journal	January 2002
Security of two quantum cryptography protocols using the same four qubit states Branciard, Cyril; Gisin, Nicolas; Kraus, Barbara Physical Review A, Vol. 72, Issue 3 https://doi.org/10.1103/PhysRevA.72.032301	journal	September 2005
Arbitrary accuracy iterative quantum phase estimation algorithm using a single ancillary qubit: A two-qubit benchmark Dobšíček, Miroslav; Johansson, Göran; Shumeiko, Vitaly Physical Review A, Vol. 76, Issue 3 https://doi.org/10.1103/PhysRevA.76.030306	journal	September 2007
Two-particle quantum walks applied to the graph isomorphism problem Gamble, John King; Friesen, Mark; Zhou, Dong Physical Review A, Vol. 81, Issue 5 https://doi.org/10.1103/PhysRevA.81.052313	journal	May 2010
Characterizing quantum gates via randomized benchmarking Magesan, Easwar; Gambetta, Jay M.; Emerson, Joseph Physical Review A, Vol. 85, Issue 4 https://doi.org/10.1103/PhysRevA.85.042311	journal	April 2012
Surface codes: Towards practical large-scale quantum computation Fowler, Austin G.; Mariantoni, Matteo; Martinis, John M. Physical Review A, Vol. 86, Issue 3 https://doi.org/10.1103/PhysRevA.86.032324	journal	September 2012
Quantum learning robust against noise Cross, Andrew W.; Smith, Graeme; Smolin, John A. Physical Review A, Vol. 92, Issue 1 https://doi.org/10.1103/PhysRevA.92.012327	journal	July 2015
Noise-tolerant parity learning with one quantum bit Park, Daniel K.; Rhee, June-Koo K.; Lee, Soonchil Physical Review A, Vol. 97, Issue 3 https://doi.org/10.1103/PhysRevA.97.032327	journal	March 2018
Quantum computational finance: Monte Carlo pricing of financial derivatives Rebentrost, Patrick; Gupt, Brajesh; Bromley, Thomas R. Physical Review A, Vol. 98, Issue 2 https://doi.org/10.1103/PhysRevA.98.022321	journal	August 2018
Variational quantum algorithms for discovering Hamiltonian spectra Jones, Tyson; Endo, Suguru; McArdle, Sam Physical Review A, Vol. 99, Issue 6 https://doi.org/10.1103/PhysRevA.99.062304	journal	June 2019
Superconducting qubit in a waveguide cavity with a coherence time approaching 0.1 ms Rigetti, Chad; Gambetta, Jay M.; Poletto, Stefano Physical Review B, Vol. 86, Issue 10 https://doi.org/10.1103/PhysRevB.86.100506	journal	September 2012
Quantum Algorithm for Linear Systems of Equations Harrow, Aram W.; Hassidim, Avinatan; Lloyd, Seth Physical Review Letters, Vol. 103, Issue 15 https://doi.org/10.1103/PhysRevLett.103.150502	journal	October 2009
Scalable and Robust Randomized Benchmarking of Quantum Processes Magesan, Easwar; Gambetta, J. M.; Emerson, Joseph Physical Review Letters, Vol. 106, Issue 18 https://doi.org/10.1103/PhysRevLett.106.180504	journal	May 2011
Preconditioned Quantum Linear System Algorithm Clader, B. D.; Jacobs, B. C.; Sprouse, C. R. Physical Review Letters, Vol. 110, Issue 25 https://doi.org/10.1103/PhysRevLett.110.250504	journal	June 2013
Cloud Quantum Computing of an Atomic Nucleus Dumitrescu, E. F.; McCaskey, A. J.; Hagen, G. Physical Review Letters, Vol. 120, Issue 21 https://doi.org/10.1103/PhysRevLett.120.210501	journal	May 2018
Direct Randomized Benchmarking for Multiqubit Devices Proctor, Timothy J.; Carignan-Dugas, Arnaud; Rudinger, Kenneth Physical Review Letters, Vol. 123, Issue 3 https://doi.org/10.1103/PhysRevLett.123.030503	journal	July 2019
Quantum Computations with Cold Trapped Ions Cirac, J. I.; Zoller, P. Physical Review Letters, Vol. 74, Issue 20 https://doi.org/10.1103/PhysRevLett.74.4091	journal	May 1995
Quantum Computers Can Search Rapidly by Using Almost Any Transformation Grover, Lov K. Physical Review Letters, Vol. 80, Issue 19 https://doi.org/10.1103/PhysRevLett.80.4329	journal	May 1998
Quantum cryptography Gisin, Nicolas; Ribordy, Grégoire; Tittel, Wolfgang Reviews of Modern Physics, Vol. 74, Issue 1 https://doi.org/10.1103/RevModPhys.74.145	journal	March 2002
Quantum dynamics of single trapped ions Leibfried, D.; Blatt, R.; Monroe, C. Reviews of Modern Physics, Vol. 75, Issue 1 https://doi.org/10.1103/RevModPhys.75.281	journal	March 2003
Quantum simulation Georgescu, I. M.; Ashhab, S.; Nori, Franco Reviews of Modern Physics, Vol. 86, Issue 1 https://doi.org/10.1103/RevModPhys.86.153	journal	March 2014
LLVM: A compilation framework for lifelong program analysis & transformation Lattner, C.; Adve, V. International Symposium on Code Generation and Optimization, 2004. CGO 2004. https://doi.org/10.1109/CGO.2004.1281665	conference	January 2004
Algorithms for quantum computation: discrete logarithms and factoring Shor, P. W. Proceedings 35th Annual Symposium on Foundations of Computer Science https://doi.org/10.1109/SFCS.1994.365700	conference	January 1994
Toward a MILP Modeling Framework for Distribution System Restoration Chen, Bo; Ye, Zhigang; Chen, Chen IEEE Transactions on Power Systems, Vol. 34, Issue 3 https://doi.org/10.1109/TPWRS.2018.2885322	journal	May 2019
Quantum Computation and Quantum Information Nielsen, Michael A.; Chuang, Isaac; Grover, Lov K. American Journal of Physics, Vol. 70, Issue 5 https://doi.org/10.1119/1.1463744	journal	May 2002
A quantum Fredkin gate Patel, Raj B.; Ho, Joseph; Ferreyrol, Franck Science Advances, Vol. 2, Issue 3 https://doi.org/10.1126/sciadv.1501531	journal	March 2016
Universal Quantum Simulators Lloyd, S. Science, Vol. 273, Issue 5278 https://doi.org/10.1126/science.273.5278.1073	journal	August 1996
Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer Shor, Peter W. SIAM Review, Vol. 41, Issue 2 https://doi.org/10.1137/S0036144598347011	journal	January 1999
On the Power of Quantum Computation Simon, Daniel R. SIAM Journal on Computing, Vol. 26, Issue 5 https://doi.org/10.1137/S0097539796298637	journal	October 1997
Quantum Complexity Theory Bernstein, Ethan; Vazirani, Umesh SIAM Journal on Computing, Vol. 26, Issue 5 https://doi.org/10.1137/S0097539796300921	journal	October 1997
A fast quantum mechanical algorithm for database search Grover, Lov K. Proceedings of the twenty-eighth annual ACM symposium on Theory of computing - STOC '96 https://doi.org/10.1145/237814.237866	conference	January 1996
Quantum Computing is Getting Real Chong, Frederic T. Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems https://doi.org/10.1145/3173162.3177152	conference	March 2018
Not All Qubits Are Created Equal: A Case for Variability-Aware Policies for NISQ-Era Quantum Computers Tannu, Swamit S.; Qureshi, Moinuddin K. ASPLOS '19: Architectural Support for Programming Languages and Operating Systems, Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems https://doi.org/10.1145/3297858.3304007	conference	April 2019
Noise-Adaptive Compiler Mappings for Noisy Intermediate-Scale Quantum Computers Murali, Prakash; Baker, Jonathan M.; Javadi-Abhari, Ali ASPLOS '19: Architectural Support for Programming Languages and Operating Systems, Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems https://doi.org/10.1145/3297858.3304075	conference	April 2019
A quantum computational compiler and design tool for technology-specific targets Smith, Kaitlin N.; Thornton, Mitchell A. Proceedings of the 46th International Symposium on Computer Architecture https://doi.org/10.1145/3307650.3322262	conference	June 2019
Full-stack, real-system quantum computer studies: architectural comparisons and design insights Murali, Prakash; Linke, Norbert Matthias; Martonosi, Margaret ISCA '19: The 46th Annual International Symposium on Computer Architecture, Proceedings of the 46th International Symposium on Computer Architecture https://doi.org/10.1145/3307650.3322273	conference	June 2019
Ensemble of Diverse Mappings: Improving Reliability of Quantum Computers by Orchestrating Dissimilar Mistakes Tannu, Swamit S.; Qureshi, Moinuddin MICRO '52: The 52nd Annual IEEE/ACM International Symposium on Microarchitecture, Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture https://doi.org/10.1145/3352460.3358257	conference	October 2019
Mitigating Measurement Errors in Quantum Computers by Exploiting State-Dependent Bias Tannu, Swamit S.; Qureshi, Moinuddin K. MICRO '52: The 52nd Annual IEEE/ACM International Symposium on Microarchitecture, Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture https://doi.org/10.1145/3352460.3358265	conference	October 2019
Eqc Stein, Samuel; Wiebe, Nathan; Ding, Yufei Proceedings of the 49th Annual International Symposium on Computer Architecture https://doi.org/10.1145/3470496.3527434	conference	June 2022
Quantum walks on graphs Aharonov, Dorit; Ambainis, Andris; Kempe, Julia Proceedings of the thirty-third annual ACM symposium on Theory of computing - STOC '01 https://doi.org/10.1145/380752.380758	conference	January 2001
Experimental Bell-State Analysis Weinfurter, H. Europhysics Letters (EPL), Vol. 25, Issue 8 https://doi.org/10.1209/0295-5075/25/8/001	journal	March 1994
ProjectQ: an open source software framework for quantum computing Steiger, Damian S.; Häner, Thomas; Troyer, Matthias Quantum, Vol. 2 https://doi.org/10.22331/q-2018-01-31-49	journal	January 2018
Quantum Computing in the NISQ era and beyond Preskill, John Quantum, Vol. 2 https://doi.org/10.22331/q-2018-08-06-79	journal	August 2018
Overview and Comparison of Gate Level Quantum Software Platforms LaRose, Ryan Quantum, Vol. 3 https://doi.org/10.22331/q-2019-03-25-130	journal	March 2019

Similar Records

pnnl/QASMBench

Software · Sun Jun 07 20:00:00 EDT 2020 · OSTI ID:code-38355

NISQ Benchmarking

Software · Thu Jul 14 20:00:00 EDT 2022 · OSTI ID:code-76077

Related Subjects

97 MATHEMATICS AND COMPUTING
CCS concepts
NISQ
benchmark
computer systems organization
hardware → quantum computation
openQASM
quantum computing
quantum metrics
software and its engineering
software libraries and repositories

QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation

Citation Formats

References (78)

Similar Records

Related Subjects