COMBINATIONS OF MICRO-MACRO STATES AND SUBGOALS DISCOVERY IN HIERARCHICAL REINFORCEMENT LEARNING FOR PATH FINDING

Gembong Edhi Setyawan, Hideyuki Sawada, Pitoyo Hartono

Research output: Contribution to journalArticlepeer-review

Abstract

While Reinforcement Learning (RL) is one of the strongest unsupervised learning algorithms, it often faces difficulties dealing with complex environments. These difficulties correlate with the curse of dimensionality in which an excessively large number of states causes the process of RL prohibitively difficult. Hierarchical Reinforcement Learning (HRL) is proposed to overcome the weaknesses of RL by hierarchically decomposing a complex problem into more manageable sub-problems. This paper proposes Micro-Macro States Combination (MMSC) as a new approach for HRL by formulating the task into two layers. The lower layer depicts the task in their microstates, which represent the original states, while the upper layer depicts macrostates, some collections of a number of the microstates. The macrostates can be considered the higher abstractions of the original states that allow the RL to perceive the problem differently. Here, the proposed MMSC is allowed to operate not only on the microstates but also on their higher-level abstractions, and thus enabling the RL to flexibly change its perspective during the problem solving, each time choosing a perspective that leads it to the solution faster. In this paper, the algorithm for the Micro-Macro States combination is formulated and tested on path-finding problems in grid worlds. Here, the novelty of the proposed algorithm in hierarchically decomposing the given problems and in automatic goal-reaching in the sub-problem is tested against traditional RL and other hierarchical RL, and quantitatively analyzed.

Original languageEnglish
Pages (from-to)447-462
Number of pages16
JournalInternational Journal of Innovative Computing, Information and Control
Volume18
Issue number2
DOIs
Publication statusPublished - 2022 Apr

Keywords

  • Hierarchical abstraction
  • Hierarchical reinforcement learning
  • Reinforcement learning
  • Task decom-position

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Information Systems
  • Computational Theory and Mathematics

Fingerprint

Dive into the research topics of 'COMBINATIONS OF MICRO-MACRO STATES AND SUBGOALS DISCOVERY IN HIERARCHICAL REINFORCEMENT LEARNING FOR PATH FINDING'. Together they form a unique fingerprint.

Cite this