Offline & Model-Based RL
Cross-Domain Evaluation and Robustness of Offline Model-Based RL Across State-Action Representations
Generated automatically from the limitations stated in 8 papers (ICML, ICLR, AISTATS), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Offline and model-based reinforcement learning algorithms are almost universally developed, theoretically analyzed, and benchmarked exclusively on either discrete (tabular/gridworld) or continuous (continuous control) domains. Because individual methods are rarely evaluated across alternative state-action representations or stochasticity regimes, it is currently unknown whether standard conservatism mechanisms, planning rollouts, and uncertainty bounds remain robust outside their design scope. As a result, practitioners facing mixed, hybrid, or converted state-action spaces cannot anticipate failure modes or know whether existing offline RL algorithms transfer reliably.
Why it matters
Provides the first empirical characterization of how offline and model-based RL methods behave across distinct action and state formulations. Enables principled algorithm selection and reveals where algorithmic adaptations are strictly necessary versus where standard methods transfer seamlessly.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Cross-representation benchmarking: Implement standard offline model-based and model-free algorithms (e.g., MOPO, COMBO, CQL) and evaluate them across paired discrete, binned, and continuous versions of standard offline environments (e.g., D4RL and MinAtar), measuring returns, value overestimation, and out-of-distribution rollout error.
- 2
Stochastic and hybrid robustness analysis: Test algorithm sensitivity to transition stochasticity and hybrid action spaces by applying continuous relaxations to discrete benchmarks and action-space discretizations to continuous environments, measuring policy degradation and model error propagation under varying discretization granularities.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
The evaluation could find that standard algorithms transfer across discretizations with trivial hyperparameter tuning, reducing the findings to a predictable verification of existing continuous control heuristics.
Evidence
Each paper's own statement of the limitation, verbatim.
- Flow-Based Single-Step Completion for Efficient and Expressive Policy LearningICLR 2026
Formulation is restricted to continuous action spaces and cannot directly operate in discrete or hybrid action domains.
- The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low RegretICML 2025
Finite state and action spaces with all states reachable are assumed; results are not directly stated for continuous or very-large-scale settings without further argument
- Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement LearningNeurIPS 2025
Theoretical analysis assumes discrete finite state-action spaces with deterministic transitions and linear function approximation, with extension to continuous domains argued but not proven
- Distributional Off-policy Evaluation with Bellman Residual MinimizationAISTATS 2025
Main detailed theory is limited to tabular settings plus continuous state-action spaces with deterministic transitions; the general continuous stochastic case is not fully developed
- Replay Memory as An Empirical MDP: Combining Conservative Estimation with Experience ReplayICLR 2023
Restricted to discrete action spaces; extension to continuous actions is left as future work.
- Identifiability and Generalizability in Constrained Inverse Reinforcement LearningICML 2023
Results restricted to discrete state and action spaces; continuous settings not addressed.
- Conformal Off-Policy PredictionAISTATS 2023
Restricted to discrete action spaces; continuous actions are left to future work.
- Provably Efficient Learning of Transferable RewardsICML 2021
Restricted to finite state and action spaces
Nearest existing work
- RL Unplugged: A Suite of Benchmarks for Offline Reinforcement LearningNeurIPS 2020
- A Closer Look at Offline RL AgentsNeurIPS 2022
- Offline RL Without Off-Policy EvaluationNeurIPS 2021
- Offline Model-based Adaptable Policy LearningNeurIPS 2021
- MOReL: Model-Based Offline Reinforcement LearningNeurIPS 2020
- A Clean Slate for Offline Reinforcement LearningNeurIPS 2025
- VIPO: Value Function Inconsistency Penalized Offline Reinforcement LearningICML 2026
- A Minimalist Approach to Offline Reinforcement LearningNeurIPS 2021
- RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement LearningNeurIPS 2022
- COMBO: Conservative Offline Model-Based Policy OptimizationNeurIPS 2021
- A Unified Framework for Alternating Offline Model Training and Policy LearningNeurIPS 2022
- Action-Free Offline-To-Online RL via Discretised State PoliciesICLR 2026
- Video-Enhanced Offline Reinforcement Learning: A Model-Based ApproachICML 2025
- Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based ImaginationNeurIPS 2022
- Offline RL Policies Should Be Trained to be AdaptiveICML 2022
Related open problems
Offline & Model-Based RL
Cross-Modality and Sim-to-Real Robustness Evaluation of Offline and Model-Based Reinforcement Learning
Current offline and model-based reinforcement learning algorithms are almost exclusively benchmarked on state-observation continuous-control locomotion tasks in simulation (e.g., standard MuJoCo suites). Because papers routinely restrict evaluations to low-dimensional proprioceptive states, the performance, sample efficiency, and stability of these algorithms remains unknown when applied to visual observations, discrete action spaces, manipulation tasks, or physical platforms. Consequently, practitioners cannot determine whether published algorithmic gains reflect genuine advancements in decision-making or overfitting to low-dimensional simulation dynamics.
Offline & Model-Based RL
Stabilizing Value Learning and Policy Extraction in Sparse-Reward Offline Reinforcement Learning
Current offline reinforcement learning algorithms suffer from catastrophic training instability, late-training policy collapse, and extreme variance on sparse-reward, long-horizon tasks such as AntMaze. To achieve reported benchmark numbers, practitioners routinely resort to unprincipled workarounds, including online pre-training interactions, domain-specific hyperparameter schedules, and checkpoint selection via test-environment rollouts. Without stable training dynamics under sparse feedback, offline RL cannot be deployed in high-stakes or real-world settings where offline checkpoint selection and online environment querying are strictly prohibited.
Offline & Model-Based RL
Empirical Robustness of Model-Based RL Across Simulator Differentiability, Parallelism, and Task Horizons
Current model-based RL and planning algorithms are developed under disjoint simulation assumptions, variously requiring fully differentiable GPU physics engines, massive parallel batch environments, or short-horizon dense rewards. Because these methods are rarely tested outside their native simulator regimes, practitioners cannot determine whether algorithms relying on analytic dynamics or extreme parallel throughput transfer to standard non-differentiable environments or long-horizon agentic tasks. This leaves the empirical boundaries and failure modes of existing model-based approaches unknown across differing simulator capabilities.