Skip to search boxSkip to navigationSkip to main content

Advancing towards Safe Reinforcement Learning over Sparse Environments with Out-of-Distribution Observations: Detection and Adaptation Strategies

*Corresponding author for this work
Research Output:
Chapter in Book/Report/Conference proceeding
Conference contribution
Research
Peer-review

Abstract

Safety in AI-based systems is among the highest research priorities, particularly when such systems are deployed in real-world scenarios subject to uncertainties and unpredictable inputs. Among them, the presence of long-tailed stimuli (Out-of-Distribution data, OoD) has captured much interest in recent times, giving rise to many proposals over the years to detect unfamiliar inputs to the model and adapt its knowledge accordingly. This work analyzes several OoD detection and adaptation strategies for Reinforcement Learning agents over environments with sparse reward signals. The sparsity of rewards and the impact of OoD objects on the state transition distribution learned by the agent are shown to be crucial when it comes to the design of effective knowledge transfer methods once OoD objects are detected. Furthermore, different approaches to detect OoD elements within the observation of the agent are also outlined, stressing on their benefits and potential downsides. Experiments with procedurally generated environments are performed to assess the performance of the considered OoD detection techniques, and to gauge the impact of the adaptation strategies on the generalization capability of the RL agent. The results pave the way towards further research around the provision of safety guarantees in sparse open-world Reinforcement Learning environments.

Research Output Information

Output type

Research Output:
Chapter in Book/Report/Conference proceeding
Conference contribution
Research
Peer-review

Original language

English

Publication milestones

  • Published - 2024

Publication status

Published - 2024

Publisher

Institute of Electrical and Electronics Engineers Inc., United States

Publication series

  • Publication series name: Proceedings of the International Joint Conference on Neural Networks

ISBN (Electronic)

9798350359312

Publication IDs

  • Scopus: 85205005846

Host publication title

2024 International Joint Conference on Neural Networks, IJCNN 2024 - Proceedings

Publication metrics

Metrics

SciVal
Author count
3
SciVal
Paper percentile
25
Scopus
citations