Quantum logic gate synthesis as a Markov decision process

Alam, Muhammad Sohaib; Berthusen, Noah F.; Orth, Peter P.

doi:10.1038/s41534-023-00766-w

Quantum logic gate synthesis as a Markov decision process

Journal Article · Wed Oct 25 00:00:00 EDT 2023 · npj Quantum Information

DOI:https://doi.org/10.1038/s41534-023-00766-w· OSTI ID:2216915

^[1]; Berthusen, Noah F. ^[2]; ^[3]

Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States); Rigetti Computing, Berkeley, CA (United States); NASA Ames Research Center (ARC), Moffett Field, Mountain View, CA (United States)
Univ. of Maryland, College Park, MD (United States); Ames Lab., and Iowa State Univ., Ames, IA (United States)
Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States); Ames Lab., and Iowa State Univ., Ames, IA (United States)

Reinforcement learning has witnessed recent applications to a variety of tasks in quantum programming. The underlying assumption is that those tasks could be modeled as Markov decision processes (MDPs). Here, we investigate the feasibility of this assumption by exploring its consequences for single-qubit quantum state preparation and gate compilation. By forming discrete MDPs, we solve for the optimal policy exactly through policy iteration. We find optimal paths that correspond to the shortest possible sequence of gates to prepare a state or compile a gate, up to some target accuracy. Our method works in both the absence and presence of noise and compares favorably to other quantum compilation methods, such as the Ross–Selinger algorithm. This work provides theoretical insight into why reinforcement learning may be successfully used to find optimally short gate sequences in quantum programming.

View Accepted Manuscript (DOE)

Research Organization:: Ames Laboratory (AMES), Ames, IA (United States)

Sponsoring Organization:: USDOE Office of Science (SC); National Aeronautics and Space Administration (NASA)

Grant/Contract Number:: AC02-07CH11358; AC02-07CH11359

OSTI ID:: 2216915

Alternate ID(s):: OSTI ID: 2287671

Report Number(s):: IS-J--11179

Journal Information:: npj Quantum Information, Journal Name: npj Quantum Information Journal Issue: 1 Vol. 9; ISSN 2056-6387

Publisher:: Nature Partner JournalsCopyright Statement

Country of Publication:: United States

Language:: English

References (41)

Quantum Machine Learning Implementations: Proposals and Experiments Lamata, Lucas Advanced Quantum Technologies, Vol. 6, Issue 7 https://doi.org/10.1002/qute.202300059	journal	May 2023
Taking Gradients Through Experiments: LSTMs and Memory Proximal Policy Optimization for Black-Box Quantum Control August, Moritz; Hernández-Lobato, José Miguel Lecture Notes in Computer Science https://doi.org/10.1007/978-3-030-02465-9_43	book	January 2018
Experimentally realizing efficient quantum control with reinforcement learning Ai, Ming-Zhong; Ding, Yongcheng; Ban, Yue Science China Physics, Mechanics & Astronomy, Vol. 65, Issue 5 https://doi.org/10.1007/s11433-021-1841-2	journal	March 2022
A reinforcement learning approach for quantum state engineering Mackeprang, Jelena; Dasari, Durga B. Rao; Wrachtrup, Jörg Quantum Machine Intelligence, Vol. 2, Issue 1 https://doi.org/10.1007/s42484-020-00016-8	journal	May 2020
A tutorial on optimal control and reinforcement learning methods for quantum technologies Giannelli, Luigi; Sgroi, Pierpaolo; Brown, Jonathon Physics Letters A, Vol. 434 https://doi.org/10.1016/j.physleta.2022.128054	journal	May 2022
Understanding Machine Learning Shalev-Shwartz, Shai; Ben-David, Shai https://doi.org/10.1017/CBO9781107298019	book	July 2014
Universal quantum control through deep reinforcement learning Niu, Murphy Yuezhen; Boixo, Sergio; Smelyanskiy, Vadim N. npj Quantum Information, Vol. 5, Issue 1 https://doi.org/10.1038/s41534-019-0141-3	journal	April 2019
When does reinforcement learning stand out in quantum control? A comparative study on state preparation Zhang, Xiao-Ming; Wei, Zezhu; Asad, Raza npj Quantum Information, Vol. 5, Issue 1 https://doi.org/10.1038/s41534-019-0201-8	journal	October 2019
Coherent transport of quantum states by deep reinforcement learning Porotti, Riccardo; Tamascelli, Dario; Restelli, Marcello Communications Physics, Vol. 2, Issue 1 https://doi.org/10.1038/s42005-019-0169-x	journal	June 2019
Quantum compiling by deep reinforcement learning Moro, Lorenzo; Paris, Matteo G. A.; Restelli, Marcello Communications Physics, Vol. 4, Issue 1 https://doi.org/10.1038/s42005-021-00684-3	journal	August 2021
On the Theory of Dynamic Programming Bellman, R. Proceedings of the National Academy of Sciences, Vol. 38, Issue 8 https://doi.org/10.1073/pnas.38.8.716	journal	August 1952
Quantum machine learning and quantum biomimetics: A perspective Lamata, Lucas Machine Learning: Science and Technology, Vol. 1, Issue 3 https://doi.org/10.1088/2632-2153/ab9803	journal	July 2020
Machine Learning of Noise-Resilient Quantum Circuits Cincio, Lukasz; Rudinger, Kenneth; Sarovar, Mohan PRX Quantum, Vol. 2, Issue 1 https://doi.org/10.1103/PRXQuantum.2.010324	journal	February 2021
Extending quantum probabilistic error cancellation by noise scaling Mari, Andrea; Shammah, Nathan; Zeng, William J. Physical Review A, Vol. 104, Issue 5 https://doi.org/10.1103/PhysRevA.104.052607	journal	November 2021
Optimal quantum circuits for general two-qubit gates Vatan, Farrokh; Williams, Colin Physical Review A, Vol. 69, Issue 3 https://doi.org/10.1103/PhysRevA.69.032315	journal	March 2004
Measurement-based adaptation protocol with quantum reinforcement learning Albarrán-Arriagada, F.; Retamal, J. C.; Solano, E. Physical Review A, Vol. 98, Issue 4 https://doi.org/10.1103/PhysRevA.98.042315	journal	October 2018
Gradient-based optimal control of open quantum systems using quantum trajectories and automatic differentiation Abdelhafez, Mohamed; Schuster, David I.; Koch, Jens Physical Review A, Vol. 99, Issue 5 https://doi.org/10.1103/PhysRevA.99.052327	journal	May 2019
Reinforcement learning for autonomous preparation of Floquet-engineered states: Inverting the quantum Kapitza oscillator Bukov, Marin Physical Review B, Vol. 98, Issue 22 https://doi.org/10.1103/PhysRevB.98.224305	journal	December 2018
Dynamically Error-Corrected Gates for Universal Quantum Computation Khodjasteh, Kaveh; Viola, Lorenza Physical Review Letters, Vol. 102, Issue 8 https://doi.org/10.1103/PhysRevLett.102.080501	journal	February 2009
Error Mitigation for Short-Depth Quantum Circuits Temme, Kristan; Bravyi, Sergey; Gambetta, Jay M. Physical Review Letters, Vol. 119, Issue 18 https://doi.org/10.1103/PhysRevLett.119.180509	journal	November 2017
Setting Up Experimental Bell Tests with Reinforcement Learning Melnikov, Alexey A.; Sekatski, Pavel; Sangouard, Nicolas Physical Review Letters, Vol. 125, Issue 16 https://doi.org/10.1103/PhysRevLett.125.160401	journal	October 2020
Random Decoupling Schemes for Quantum Dynamical Control and Error Suppression Viola, Lorenza; Knill, Emanuel Physical Review Letters, Vol. 94, Issue 6 https://doi.org/10.1103/PhysRevLett.94.060502	journal	February 2005
Reinforcement-learning-assisted quantum optimization Wauters, Matteo M.; Panizon, Emanuele; Mbeng, Glen B. Physical Review Research, Vol. 2, Issue 3 https://doi.org/10.1103/PhysRevResearch.2.033446	journal	September 2020
Unified approach to data-driven quantum error mitigation Lowe, Angus; Gordon, Max Hunter; Czarnik, Piotr Physical Review Research, Vol. 3, Issue 3 https://doi.org/10.1103/PhysRevResearch.3.033098	journal	July 2021
Model-Free Quantum Control with Reinforcement Learning Sivak, V. V.; Eickbusch, A.; Liu, H. Physical Review X, Vol. 12, Issue 1 https://doi.org/10.1103/PhysRevX.12.011059	journal	March 2022
Efficient Variational Quantum Simulator Incorporating Active Error Minimization Li, Ying; Benjamin, Simon C. Physical Review X, Vol. 7, Issue 2 https://doi.org/10.1103/PhysRevX.7.021050	journal	June 2017
Synthesis of quantum-logic circuits Shende, V. V.; Bullock, S. S.; Markov, I. L. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol. 25, Issue 6 https://doi.org/10.1109/TCAD.2005.855930	journal	June 2006
Using Reinforcement Learning to Perform Qubit Routing in Quantum Compilers Pozzi, Matteo G.; Herbert, Steven J.; Sengupta, Akash ACM Transactions on Quantum Computing, Vol. 3, Issue 2 https://doi.org/10.1145/3520434	journal	May 2022
Machine Learning Optimization of Quantum Circuit Layouts Paler, Alexandru; Sasu, Lucian; Florea, Adrian-Cătălin ACM Transactions on Quantum Computing, Vol. 4, Issue 2 https://doi.org/10.1145/3565271	journal	February 2023
Deep reinforcement learning for quantum gate control An, Zheng; Zhou, D. L. EPL (Europhysics Letters), Vol. 126, Issue 6 https://doi.org/10.1209/0295-5075/126/60002	journal	July 2019
The Simplex and Policy-Iteration Methods Are Strongly Polynomial for the Markov Decision Problem with a Fixed Discount Rate Ye, Yinyu Mathematics of Operations Research, Vol. 36, Issue 4 https://doi.org/10.1287/moor.1110.0516	journal	November 2011
Improved and Generalized Upper Bounds on the Complexity of Policy Iteration Scherrer, Bruno Mathematics of Operations Research, Vol. 41, Issue 3 https://doi.org/10.1287/moor.2015.0753	journal	August 2016
An Introduction to Deep Reinforcement Learning François-Lavet, Vincent; Henderson, Peter; Islam, Riashat Foundations and Trends® in Machine Learning, Vol. 11, Issue 3-4 https://doi.org/10.1561/2200000071	journal	January 2018
Quantum Exploration Algorithms for Multi-Armed Bandits Wang, Daochen; You, Xuchen; Li, Tongyang Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, Issue 11 https://doi.org/10.1609/aaai.v35i11.17212	journal	May 2021
Machine learning and quantum devices Marquardt, Florian SciPost Physics Lecture Notes https://doi.org/10.21468/SciPostPhysLectNotes.29	journal	May 2021
Two-Qubit Circuit Depth and the Monodromy Polytope Peterson, Eric C.; Crooks, Gavin E.; Smith, Robert S. Quantum, Vol. 4 https://doi.org/10.22331/q-2020-03-26-247	journal	March 2020
Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning Skolik, Andrea; Jerbi, Sofiene; Dunjko, Vedran Quantum, Vol. 6 https://doi.org/10.22331/q-2022-05-24-720	journal	May 2022
Multi-armed quantum bandits: Exploration versus exploitation when learning properties of quantum states Lumbreras, Josep; Haapasalo, Erkka; Tomamichel, Marco Quantum, Vol. 6 https://doi.org/10.22331/q-2022-06-29-749	journal	June 2022
Quantum speedups for convex dynamic programming Sutter, David; Nannicini, Giacomo; Sutter, Tobias arXiv https://doi.org/10.48550/ARXIV.2011.11654	preprint	January 2020
Parametrized quantum policies for reinforcement learning Jerbi, Sofiene; Gyurik, Casper; Marshall, Simon C. arXiv https://doi.org/10.48550/ARXIV.2103.05577	preprint	January 2021
Optimizing Quantum Variational Circuits with Deep Reinforcement Learning Lockwood, Owen arXiv https://doi.org/10.48550/ARXIV.2109.03188	preprint	January 2021

Similar Records

Mean-Variance Problems for Finite Horizon Semi-Markov Decision Processes

Journal Article · Thu Oct 15 00:00:00 EDT 2015 · Applied Mathematics and Optimization · OSTI ID:22722847

K-Spin Hamiltonian for Quantum-Resolvable Markov Decision Processes

Journal Article · Fri Oct 30 00:00:00 EDT 2020 · Quantum Machine Intelligence · OSTI ID:1885979

Related Subjects

71 CLASSICAL AND QUANTUM MECHANICS
GENERAL PHYSICS
computer science
information theory and computation
quantum physics

Quantum logic gate synthesis as a Markov decision process

Citation Formats

References (41)

Similar Records

Related Subjects