TY - GEN
T1 - A Meta Reinforcement Learning-based Approach for Self-Adaptive System
AU - Zhang, Mingyue
AU - Li, Jialong
AU - Zhao, Haiyan
AU - Tei, Kenji
AU - Honiden, Shinichi
AU - Jin, Zhi
N1 - Funding Information:
The authors would like to thank Dr. Nianyu Li and Kun Liu for comments on an early version of the paper. Research partially supported by the National Natural Science Foundation of China under Grant Nos. 61620106007, 61751210, and JSPS KAKENHI.
Publisher Copyright:
© 2021 IEEE.
PY - 2021
Y1 - 2021
N2 - A self-learning adaptive system (SLAS) uses machine learning to enable and enhance its adaptability. Such systems are expected to perform well in dynamic environments. For learning high-performance adaptation policy, some assumptions must be made on the environment-system dynamics when information about the real situation is incomplete. However, these assumptions cannot be expected to be always correct, and yet it is difficult to enumerate all possible assumptions. This leads to the problem of incomplete-information learning. We consider this problem as multiple model problem in terms of finding the adaptation policy that can cope with multiple models of environment-system dynamics. This paper proposes a novel approach to engineering the online adaptation of SLAS. It separates three concerns that are related to the adaptation policy and presents the modeling and synthesis process, with the goal of achieving higher model construction efficiency. In addition, it designs a meta-reinforcement learning algorithm for learning the meta policy over the multiple models, so that the meta policy can quickly adapt to the real environment-system dynamics. At last, it reports the case study on a robotic system to evaluate the adaptability of the approach.
AB - A self-learning adaptive system (SLAS) uses machine learning to enable and enhance its adaptability. Such systems are expected to perform well in dynamic environments. For learning high-performance adaptation policy, some assumptions must be made on the environment-system dynamics when information about the real situation is incomplete. However, these assumptions cannot be expected to be always correct, and yet it is difficult to enumerate all possible assumptions. This leads to the problem of incomplete-information learning. We consider this problem as multiple model problem in terms of finding the adaptation policy that can cope with multiple models of environment-system dynamics. This paper proposes a novel approach to engineering the online adaptation of SLAS. It separates three concerns that are related to the adaptation policy and presents the modeling and synthesis process, with the goal of achieving higher model construction efficiency. In addition, it designs a meta-reinforcement learning algorithm for learning the meta policy over the multiple models, so that the meta policy can quickly adapt to the real environment-system dynamics. At last, it reports the case study on a robotic system to evaluate the adaptability of the approach.
KW - Meta Learning
KW - Reinforcement Learning
KW - Self-adaptation
KW - Separation of Concerns
UR - http://www.scopus.com/inward/record.url?scp=85124798114&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85124798114&partnerID=8YFLogxK
U2 - 10.1109/ACSOS52086.2021.00024
DO - 10.1109/ACSOS52086.2021.00024
M3 - Conference contribution
AN - SCOPUS:85124798114
T3 - Proceedings - 2021 IEEE International Conference on Autonomic Computing and Self-Organizing Systems, ACSOS 2021
SP - 1
EP - 10
BT - Proceedings - 2021 IEEE International Conference on Autonomic Computing and Self-Organizing Systems, ACSOS 2021
A2 - El-Araby, Esam
A2 - Kalogeraki, Vana
A2 - Pianini, Danilo
A2 - Lassabe, Frederic
A2 - Porter, Barry
A2 - Ghahremani, Sona
A2 - Nunes, Ingrid
A2 - Bakhouya, Mohamed
A2 - Tomforde, Sven
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2nd IEEE International Conference on Autonomic Computing and Self-Organizing Systems, ACSOS 2021
Y2 - 27 September 2021 through 1 October 2021
ER -