← Research

Generative OOD-regularized Model-based Policy Optimization

Aysin Tumay, Jiahe Huang, Elise Jortberg, Rose Yu

We study sequential decision-making with offline reinforcement learning (RL). Traditional offline RL policies may result in out-of-distribution (OOD) actions when training relies only on sparse offline representations. To ensure safe offline policies in a sparse state-action space, we explore how density estimation models can be integrated into model-based RL methods to avoid OOD regions. Generative models are capable of explicitly modeling the density in sparse state-action spaces. Building on this, we introduce Generative OOD-regularized Model-based Policy Optimization (GORMPO), a density-regularized offline RL algorithm that uses generative density modeling to restrict policy updates to high-density areas of the dataset. Furthermore, we examine whether better OOD detection corresponds to better model-based offline policies. We compare (1) the OOD detection capabilities of various density estimators and (2) their performance within the GORMPO framework on a real-world medical dataset and sparse offline RL datasets. We provide theoretical guarantees bounding GORMPO's performance gap in terms of density estimation error, ensuring near-optimal policies under imperfect density estimates. Empirically, GORMPO remains competitive in reward and improves clinical metrics compared to state-of-the-art clinical RL models, and outperforms state-of-the-art offline RL models in offline RL benchmarks. Our empirical findings show that better OOD detection generally results in improved policies in environments with stable dynamics, while conservative penalties with poor density estimation are favored when dynamics are uncertain. Code, checkpoints, sparse, and OOD D4RL datasets are available at https://github.com/Rose-STL-Lab/GORMPO.

PDF ↗

Models

Loading
EVIDENCE / COMPARISON

Benchmarks

Explore every benchmark

Loading

These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

Install TensorTrace