skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Reinforcement learning based schemes to manage client activities in large distributed control systems

Journal Article · · Physical Review Accelerators and Beams
; ; ;

Large distributed control systems typically can be modeled by a hierarchical structure with two physical layers: console level computers (CLCs) layer and front end computers (FECs) layer. The control system of the Relativistic Heavy Ion Collider (RHIC) at Brookhaven National Laboratory (BNL) consists of more than 500 FECs, each acting as a server providing services to a large number of clients. Hence the interactions between the server and its clients become crucial to the overall system performance. There are different scenarios of the interactions. For instance, there are cases where the server has a limited processing ability and is queried by a large number of clients. Such cases can put a bottleneck in the system, as heavy traffic can slow down or even crash a system, making it momentarily unresponsive. Also, there are cases where the server has adequate ability to process all the traffic from its clients. We pursue different goals in those cases. For the first case, we would like to manage clients’ activities so that their requests are processed by the server as much as possible and the server remains operational. For the second case, we would like to explore an operation point at which the server’s resources get utilized efficiently. Moreover, we consider a real-world time constraint to the above case. The time constraint states that clients expect the responses from the server within a time window. In this work, we analyze those cases from a game theory perspective. We model the underlying interactions as a repeated game between clients, which is carried out in discrete time slots. For clients’ activity management, we apply a reinforcement learning procedure as a baseline to regulate clients’ behaviors. Then we propose a memory scheme to improve its performance. Next, depending on different scenarios, we design corresponding reward functions to stimulate clients in a proper way so that they can learn to optimize different goals. Through extensive simulations, we show that first, the memory structure improves the learning ability of the baseline procedure significantly. Second, by applying appropriate reward functions, clients’ activities can be effectively managed to achieve different optimization goals.

Research Organization:
Brookhaven National Lab. (BNL), Upton, NY (United States); Stony Brook Univ., NY (United States)
Sponsoring Organization:
USDOE; National Science Foundation (NSF)
Grant/Contract Number:
SC0012704; 1553385
OSTI ID:
1489305
Alternate ID(s):
OSTI ID: 1491685
Report Number(s):
BNL-210905-2019-JAAM; PRABCJ; 014601
Journal Information:
Physical Review Accelerators and Beams, Journal Name: Physical Review Accelerators and Beams Vol. 22 Journal Issue: 1; ISSN 2469-9888
Publisher:
American Physical SocietyCopyright Statement
Country of Publication:
United States
Language:
English
Citation Metrics:
Cited by: 3 works
Citation information provided by
Web of Science

References (6)

Multiagent learning using a variable learning rate journal April 2002
Learning, hypothesis testing, and Nash equilibrium journal October 2003
Conditional Universal Consistency journal October 1999
A Simple Adaptive Procedure Leading to Correlated Equilibrium journal September 2000
Calibrated Learning and Correlated Equilibrium journal October 1997
RHIC control system
  • Barton, D. S.; Binello, S.; Buxton, W.
  • Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 499, Issue 2-3 https://doi.org/10.1016/S0168-9002(02)01943-5
journal March 2003

Figures / Tables (19)

FIG. 1(p. 3)figure FIG. 1
FIG. 2(p. 4)figure FIG. 2
Table I(p. 5)table Table I
FIG. 3(p. 8)figure FIG. 3
Table II(p. 13)table Table II
Table III(p. 13)table Table III
FIG. 4(p. 14)figure FIG. 4
FIG. 5(p. 14)figure FIG. 5
FIG. 6(p. 15)figure FIG. 6
Table IV(p. 15)table Table IV
Table V(p. 15)table Table V
FIG. 7(p. 16)figure FIG. 7
FIG. 8(p. 16)figure FIG. 8
FIG. 9(p. 16)figure FIG. 9
FIG. 10(p. 17)figure FIG. 10
FIG. 11(p. 17)figure FIG. 11
Table VI(p. 17)table Table VI
FIG. 12(p. 18)figure FIG. 12
Table VII(p. 18)table Table VII

Similar Records

Related Subjects

97 MATHEMATICS AND COMPUTING
collective behavior in networks
computational complexity
evolving networks
network formation & growth
coherent structures
collective dynamics
high dimensional systems
nonlinear time-delay systems