Approximating Nash Equilibrium in Day-ahead Electricity Market Bidding with Multi-agent Deep Reinforcement Learning

Du, Yan; Li, Fangxing; Zandi, Helia; Xue, Yaosuo

doi:10.35833/mpce.2020.000502

Title: Approximating Nash Equilibrium in Day-ahead Electricity Market Bidding with Multi-agent Deep Reinforcement Learning

Full Record
Other Related Research

Abstract

In this paper, a day-ahead electricity market bidding problem with multiple strategic generation company (GEN-CO) bidders is studied. The problem is formulated as a Markov game model, where GENCO bidders interact with each other todevelop their optimal day-ahead bidding strategies. Considering unobservable information in the problem, a model-free and data-driven approach, known as multi-agent deep deterministic policy gradient (MADDPG), is applied for approximating the Nash equilibrium (NE) in the above Markov game. The MADDPG algorithm has the advantage of generalization due to the automatic feature extraction ability of the deep neural networks. The algorithm is tested on an IEEE 30-bus system with three competitive GENCO bidders in both an uncongested caseand a congested case. Comparisons with a truthful bidding strategy and state-of-the-art deep reinforcement learning methods including deep Q network and deep deterministic policy gradient (DDPG) demonstrate that the applied MADDPG algorithm can find a superior bidding strategy for all the market participants with increased profit gains. In addition, the comparison with a conventional model-based method shows that the MADDPG algorithm has higher computational efficiency, which is feasible for real-world applications.

Authors:

Du, Yan ^[1]; Li, Fangxing ^[1];

^[2];

^[2]

Univ. of Tennessee, Knoxville, TN (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Publication Date:: Mon Apr 19 00:00:00 EDT 2021

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE Office of Electricity (OE)

OSTI Identifier:: 1782068

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Accepted Manuscript

Journal Name:: Journal of Modern Power Systems and Clean Energy

Additional Journal Information:: Journal Volume: 9; Journal Issue: 3; Journal ID: ISSN 2196-5625

Publisher:: Springer

Country of Publication:: United States

Language:: English

Subject:: 24 POWER TRANSMISSION AND DISTRIBUTION; Games; electricity supply industry; Markov processes; computational modeling; training; Nash equilibrium; mathematical model; bidding strategy; day-ahead electricity market; deep reinforcement learning; Markov game; multi-agent deterministic policy gradient (MADDPG)

Citation Formats


                    Du, Yan, Li, Fangxing, Zandi, Helia, and Xue, Yaosuo. Approximating Nash Equilibrium in Day-ahead Electricity Market Bidding with Multi-agent Deep Reinforcement Learning.  United States: N. p., 2021. 
Web.  doi:10.35833/mpce.2020.000502.

Copy to clipboard


                    Du, Yan, Li, Fangxing, Zandi, Helia, & Xue, Yaosuo. Approximating Nash Equilibrium in Day-ahead Electricity Market Bidding with Multi-agent Deep Reinforcement Learning.  United States.  https://doi.org/10.35833/mpce.2020.000502

Copy to clipboard


                    Du, Yan, Li, Fangxing, Zandi, Helia, and Xue, Yaosuo. Mon .  
"Approximating Nash Equilibrium in Day-ahead Electricity Market Bidding with Multi-agent Deep Reinforcement Learning".  United States.  https://doi.org/10.35833/mpce.2020.000502.  https://www.osti.gov/servlets/purl/1782068.

Copy to clipboard


                    
@article{osti_1782068,

  title        = {Approximating Nash Equilibrium in Day-ahead Electricity Market Bidding with Multi-agent Deep Reinforcement Learning},

  author       = {Du, Yan and Li, Fangxing and Zandi, Helia and Xue, Yaosuo},

  abstractNote = {In this paper, a day-ahead electricity market bidding problem with multiple strategic generation company (GEN-CO) bidders is studied. The problem is formulated as a Markov game model, where GENCO bidders interact with each other todevelop their optimal day-ahead bidding strategies. Considering unobservable information in the problem, a model-free and data-driven approach, known as multi-agent deep deterministic policy gradient (MADDPG), is applied for approximating the Nash equilibrium (NE) in the above Markov game. The MADDPG algorithm has the advantage of generalization due to the automatic feature extraction ability of the deep neural networks. The algorithm is tested on an IEEE 30-bus system with three competitive GENCO bidders in both an uncongested caseand a congested case. Comparisons with a truthful bidding strategy and state-of-the-art deep reinforcement learning methods including deep Q network and deep deterministic policy gradient (DDPG) demonstrate that the applied MADDPG algorithm can find a superior bidding strategy for all the market participants with increased profit gains. In addition, the comparison with a conventional model-based method shows that the MADDPG algorithm has higher computational efficiency, which is feasible for real-world applications.},

  doi          = {10.35833/mpce.2020.000502},

  journal      = {Journal of Modern Power Systems and Clean Energy},

  number       = 3,

  volume       = 9,

  place        = {United States},

  year         = {Mon Apr 19 00:00:00 EDT 2021},

  month        = {Mon Apr 19 00:00:00 EDT 2021}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.35833/mpce.2020.000502

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

Automated Control of Transactive HVACs in Energy Distribution Systems

Journal Article Liu, Boming ; Akcakaya, Murat ; McDermott, Thomas E. - IEEE Transactions on Smart Grid

Heating, Ventilation, and Air Conditioning (HVAC) systems contribute significantly to a building’s energy consumption. In the recent years, there is an increased interest in developing transactive approaches which could enable automated and flexible scheduling of HVAC systems based on the customer demand and the electricity prices decided by the suppliers. Flexible and automated scheduling of the HVAC systems make it a prime source for participation in residential demand response or transactive energy systems. Therefore, it is of significant interest to identify an optimal strategy to control the HVAC systems. Here, reducing the energy cost while keeping the comfort level acceptablemore »« less
https://doi.org/10.1109/tsg.2020.3042498

Full Text Available
Optimizing and Extending the Functionality of EXARL for Scalable Reinforcement Learning [Slides]

Technical Report Chenna, Sai Prabhakarrao ; Cosburn, Katherine Saara Birgitte ; Ezeobi, Uchenna Mark ; ...

The main goal of the Co-Design Summer School 2021 is to provide algorithmic improvements to EXARL framework by improving performance and by adding functionalities. This presentation includes an introduction to reinforcement learning and to EXARL. The researchers expanded the capability of EXARL by including additional agents like (Asynchronized) Advantage Actor Critic (A2C/A3C) and Twin Delayed Deep Deterministic Policy Gradient (TD3). They also explored algorithmic improvements such as v-trace and Prioritized Experience Replay. They found that A2C/A3C performed best with v-trace and outperformed Deep Q-Network (DQN) on both the CartPole game and the ExaBooster scientific environment. Additionally, they found that TD3more »« less
https://doi.org/10.2172/1812639

Full Text Available
Bidding strategy for microgrid in day-ahead market based on hybrid stochastic/robust optimization

Journal Article Liu, Guodong ; Xu, Yan ; Tomsovic, Kevin - IEEE Transactions on Smart Grid

In this paper, we propose an optimal bidding strategy in the day-ahead market of a microgrid consisting of intermittent distributed generation (DG), storage, dispatchable DG and price responsive loads. The microgrid coordinates the energy consumption or production of its components and trades electricity in both the day-ahead and real-time markets to minimize its operating cost as a single entity. The bidding problem is challenging due to a variety of uncertainties, including power output of intermittent DG, load variation, day-ahead and real-time market prices. A hybrid stochastic/robust optimization model is proposed to minimize the expected net cost, i.e., expected total costmore »« less
Cited by 300
https://doi.org/10.1109/TSG.2015.2476669

Full Text Available
Bidding strategies that minimize risk with options and futures contracts

Conference Richter, Jr, C W ; Sheble, G B

This research builds on earlier research in developing bidding strategies through the inclusion of options and future contracts. In the competitive environment, electric traders` profits depends on the implementation of a successful bidding strategy. Bidding strategies are studied in an environment in which distribution companies (DISTCOs) and generation companies (GENCOs), buy and sell power via double auctions in regional commodity exchanges. The market framework being used was proposed by Kumar and Sheble and allows participants to trade in the spot, future, planning and swap markets, and also gives rise to the use of option contracts. Bid-strategy research previously published bymore »« less
A Novel Multi-Agent Deep Reinforcement Learning-enabled Distributed Power Allocation Scheme for mmWave Cellular Networks

Conference Zhang, Xiang ; Kasera, Sneha Kumar ; Ji, Mingyue ; ...

We consider the power allocation problem over shared spectrum for millimeter-Wave (mmWave) cellular downlink. Existing approaches usually find sub-optimal solutions by solving a non-convex optimization which leads to scalability issues due to centralized control. Therefore, distributed and adaptive approaches are desirable. Recently, model-free Deep Reinforcement Learning (DRL) has achieved success in such wireless resource management tasks. By modeling the radio environment as a Markov Decision Process (MDP) with the base stations (BSs) being the agents, power allocation can be automated at the agent level with comparable throughput performance to conventional centralized schemes. The multi-agent setting presents new challenges as themore »« less
Full Text Available

Similar Records