Use advantage-better action imitation for policy constraint

Weipeng Liu; Maolin Hou

doi:10.1117/12.3011988

7 December 2023 Use advantage-better action imitation for policy constraint

Weipeng Liu, Maolin Hou

Proceedings Volume 12941, International Conference on Algorithms, High Performance Computing, and Artificial Intelligence (AHPCAI 2023); 129412F (2023) https://doi.org/10.1117/12.3011988
Event: Third International Conference on Algorithms, High Performance Computing, and Artificial Intelligence (AHPCAI 203), 2023, Yinchuan, China

Abstract

Offline Reinforcement Learning (RL) aims to learn an optimal policy from a fixed dataset previously collected. Unlike in the online training process, the errors in value estimation from out-of-distribution actions (OOD actions) could not be corrected by interacting with the environment, which makes offline RL difficult to train. Prior policy constraint methods mitigate the errors by minimizing the deviation from the behavior policy, which in fact makes a trade-off between RL and imitation learning. However, while using the constraint term to avoid choosing OOD actions, the learned policy may be enslaved to the unfavorable actions from the dataset. In this paper, we propose a simple solution to this problem. Our method uses an imitation learning term introduced by TD3PlusBc as the constraint and re-weights it with a function of actions' advantage value to mitigate the influence of the unfavorable actions. To make the learning process stable, we also decouple the policy evaluation and policy improvement by using implicit q-learning, which modifies the loss function in a SARSA-style TD backup. Our method (ABAI) is easy to implement, fast to train, and computationally efficient. ABAI achieves state-of-the-art performance on the D4RL dataset, a standard benchmark for offline reinforcement learning, and shows a high ability to learn robustly from various datasets.

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Weipeng Liu and Maolin Hou "Use advantage-better action imitation for policy constraint", Proc. SPIE 12941, International Conference on Algorithms, High Performance Computing, and Artificial Intelligence (AHPCAI 2023), 129412F (7 December 2023); https://doi.org/10.1117/12.3011988

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
6 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Education and training

Machine learning

Tunable filters

Error analysis

Online learning

Robots

Mixtures

Show All Keywords

Keywords/Phrases

Search In:

Publication Years