Advancing towards Safe Reinforcement Learning over Sparse Environments with Out-of-Distribution Observations: Detection and Adaptation Strategies

Aitor Martinez-Seras*, Alain Andres, Javier Del Ser

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Safety in AI-based systems is among the highest research priorities, particularly when such systems are deployed in real-world scenarios subject to uncertainties and unpredictable inputs. Among them, the presence of long-tailed stimuli (Out-of-Distribution data, OoD) has captured much interest in recent times, giving rise to many proposals over the years to detect unfamiliar inputs to the model and adapt its knowledge accordingly. This work analyzes several OoD detection and adaptation strategies for Reinforcement Learning agents over environments with sparse reward signals. The sparsity of rewards and the impact of OoD objects on the state transition distribution learned by the agent are shown to be crucial when it comes to the design of effective knowledge transfer methods once OoD objects are detected. Furthermore, different approaches to detect OoD elements within the observation of the agent are also outlined, stressing on their benefits and potential downsides. Experiments with procedurally generated environments are performed to assess the performance of the considered OoD detection techniques, and to gauge the impact of the adaptation strategies on the generalization capability of the RL agent. The results pave the way towards further research around the provision of safety guarantees in sparse open-world Reinforcement Learning environments.

Original languageEnglish
Title of host publication2024 International Joint Conference on Neural Networks, IJCNN 2024 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9798350359312
DOIs
Publication statusPublished - 2024
Event2024 International Joint Conference on Neural Networks, IJCNN 2024 - Yokohama, Japan
Duration: 30 Jun 20245 Jul 2024

Publication series

NameProceedings of the International Joint Conference on Neural Networks

Conference

Conference2024 International Joint Conference on Neural Networks, IJCNN 2024
Country/TerritoryJapan
CityYokohama
Period30/06/245/07/24

Keywords

  • Open-World Learning
  • Out-of-Distribution (OoD)
  • Reinforcement Learning
  • Sparse Rewards

Fingerprint

Dive into the research topics of 'Advancing towards Safe Reinforcement Learning over Sparse Environments with Out-of-Distribution Observations: Detection and Adaptation Strategies'. Together they form a unique fingerprint.

Cite this