COMBINATIONS OF MICRO-MACRO STATES AND SUBGOALS DISCOVERY IN HIERARCHICAL REINFORCEMENT LEARNING FOR PATH FINDING

Gembong Edhi Setyawan, Hideyuki Sawada, Pitoyo Hartono

研究成果: Article査読

抄録

While Reinforcement Learning (RL) is one of the strongest unsupervised learning algorithms, it often faces difficulties dealing with complex environments. These difficulties correlate with the curse of dimensionality in which an excessively large number of states causes the process of RL prohibitively difficult. Hierarchical Reinforcement Learning (HRL) is proposed to overcome the weaknesses of RL by hierarchically decomposing a complex problem into more manageable sub-problems. This paper proposes Micro-Macro States Combination (MMSC) as a new approach for HRL by formulating the task into two layers. The lower layer depicts the task in their microstates, which represent the original states, while the upper layer depicts macrostates, some collections of a number of the microstates. The macrostates can be considered the higher abstractions of the original states that allow the RL to perceive the problem differently. Here, the proposed MMSC is allowed to operate not only on the microstates but also on their higher-level abstractions, and thus enabling the RL to flexibly change its perspective during the problem solving, each time choosing a perspective that leads it to the solution faster. In this paper, the algorithm for the Micro-Macro States combination is formulated and tested on path-finding problems in grid worlds. Here, the novelty of the proposed algorithm in hierarchically decomposing the given problems and in automatic goal-reaching in the sub-problem is tested against traditional RL and other hierarchical RL, and quantitatively analyzed.

本文言語English
ページ(範囲)447-462
ページ数16
ジャーナルInternational Journal of Innovative Computing, Information and Control
18
2
DOI
出版ステータスPublished - 2022 4月

ASJC Scopus subject areas

  • ソフトウェア
  • 理論的コンピュータサイエンス
  • 情報システム
  • 計算理論と計算数学

フィンガープリント

「COMBINATIONS OF MICRO-MACRO STATES AND SUBGOALS DISCOVERY IN HIERARCHICAL REINFORCEMENT LEARNING FOR PATH FINDING」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル