Offline & Model-Based RL
Cross-Modality and Sim-to-Real Robustness Evaluation of Offline and Model-Based Reinforcement Learning
Generated automatically from the limitations stated in 10 papers (ICML, ICLR, NeurIPS), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Current offline and model-based reinforcement learning algorithms are almost exclusively benchmarked on state-observation continuous-control locomotion tasks in simulation (e.g., standard MuJoCo suites). Because papers routinely restrict evaluations to low-dimensional proprioceptive states, the performance, sample efficiency, and stability of these algorithms remains unknown when applied to visual observations, discrete action spaces, manipulation tasks, or physical platforms. Consequently, practitioners cannot determine whether published algorithmic gains reflect genuine advancements in decision-making or overfitting to low-dimensional simulation dynamics.
Why it matters
Provides an empirical map of where standard model-based and offline RL algorithms generalize or fail across observation and action modalities. This allows researchers to focus algorithmic innovations on demonstrated points of failure rather than over-optimizing for standard state-based locomotion benchmarks.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Standardized Multi-Domain Empirical Suite: Implement leading model-based and offline RL baselines within a unified codebase and evaluate them across state vs. pixel inputs, continuous vs. discrete action formulations, and locomotion vs. contact-rich manipulation tasks, measuring sample efficiency, normalized return, and runtime variance across matched compute budgets.
- 2
Sim-to-Real Perturbation and Transfer Suite: Evaluate the sensitivity of policies and world models trained on simulated environments to visual distractors, sensor latency, and dynamic parameter mismatches, measuring zero-shot and fine-tuned transfer degradation.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
The primary risk is that running extensive cross-setting evaluations with adequate random seeds across multiple algorithm classes requires significant computational resources. Alternatively, community interest may shift towards foundation-model-based policies before an evaluation study on classical offline/model-based RL algorithms gains traction.
Evidence
Each paper's own statement of the limitation, verbatim.
- Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RLICML 2026
Evaluated only on 10 model-free continuous-control benchmarks (MuJoCo-style locomotion, manipulation, CarRacing); no discrete-action, offline RL, or vision-based manipulation beyond one driving task
- Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few DemonstrationsNeurIPS 2025
Evaluation is simulation-only (MuJoCo Fetch/Shadow Hand plus dense-reward locomotion in appendix); no sim-to-real validation
- Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement LearningICML 2025
Evaluated only on simulation benchmarks (DM Control locomotion and single-task Meta-World); no real-robot or discrete-action results reported
- Efficient Online Reinforcement Learning for Diffusion PolicyICML 2025
Evaluation limited to MuJoCo continuous-control benchmarks; no discrete-action or high-dimensional real-robot settings tested
- Behavior-agnostic Task Inference for Robust Offline In-context Reinforcement LearningICML 2025
Evaluated only on state-based MuJoCo locomotion with 20 training and 20 test tasks per environment; no vision-based or real-robot settings
- Robust Adversarial Reinforcement Learning via Bounded Rationality CurriculaICLR 2024
Evaluated only on MuJoCo locomotion and navigation benchmarks with environment-specific force perturbations; no real-robot or vision-based results
- ReDiffuser: Reliable Decision-Making Using a Diffuser with Confidence EstimationICML 2024
Validated only on simple navigation and locomotion tasks (Maze2D, MuJoCo, KUKA); no real-robot or sim-to-real results
- RVI-SAC: Average Reward Off-Policy Deep Reinforcement LearningICML 2024
Evaluation limited to MuJoCo locomotion benchmarks; no comparison against average-reward on-policy methods such as APO or ATRPO in the main experiments
Show all 10 papers
- Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement LearningICLR 2024
Evaluated only on four Brax/MuJoCo locomotion tasks with leg-contact measure functions; scaling to real robotics or distributed settings is untested
- CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and SimplicityICLR 2024
Only evaluated on state-observation MuJoCo benchmarks; not tested on visual observations or real robot systems
Nearest existing work
- RL Unplugged: A Suite of Benchmarks for Offline Reinforcement LearningNeurIPS 2020
- A Closer Look at Offline RL AgentsNeurIPS 2022
- Video-Enhanced Offline Reinforcement Learning: A Model-Based ApproachICML 2025
- A Minimalist Approach to Offline Reinforcement LearningNeurIPS 2021
- Offline Model-based Adaptable Policy LearningNeurIPS 2021
- MOReL: Model-Based Offline Reinforcement LearningNeurIPS 2020
- Offline RL Without Off-Policy EvaluationNeurIPS 2021
- Benchmarking Offline Reinforcement Learning on Real-Robot HardwareICLR 2023
- Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy OptimizationICLR 2024
- Simple Ingredients for Offline Reinforcement LearningICML 2024
- Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget MattersICML 2022
- Representation Matters: Offline Pretraining for Sequential Decision MakingICML 2021
- A Clean Slate for Offline Reinforcement LearningNeurIPS 2025
- Iteratively Refined Behavior Regularization for Offline Reinforcement LearningNeurIPS 2024
- Benchmarking Deep Reinforcement Learning for Continuous ControlICML 2016
Related open problems
Offline & Model-Based RL
Cross-Domain Evaluation and Robustness of Offline Model-Based RL Across State-Action Representations
Offline and model-based reinforcement learning algorithms are almost universally developed, theoretically analyzed, and benchmarked exclusively on either discrete (tabular/gridworld) or continuous (continuous control) domains. Because individual methods are rarely evaluated across alternative state-action representations or stochasticity regimes, it is currently unknown whether standard conservatism mechanisms, planning rollouts, and uncertainty bounds remain robust outside their design scope. As a result, practitioners facing mixed, hybrid, or converted state-action spaces cannot anticipate failure modes or know whether existing offline RL algorithms transfer reliably.
Offline & Model-Based RL
Stabilizing Value Learning and Policy Extraction in Sparse-Reward Offline Reinforcement Learning
Current offline reinforcement learning algorithms suffer from catastrophic training instability, late-training policy collapse, and extreme variance on sparse-reward, long-horizon tasks such as AntMaze. To achieve reported benchmark numbers, practitioners routinely resort to unprincipled workarounds, including online pre-training interactions, domain-specific hyperparameter schedules, and checkpoint selection via test-environment rollouts. Without stable training dynamics under sparse feedback, offline RL cannot be deployed in high-stakes or real-world settings where offline checkpoint selection and online environment querying are strictly prohibited.
Offline & Model-Based RL
Empirical Robustness of Model-Based RL Across Simulator Differentiability, Parallelism, and Task Horizons
Current model-based RL and planning algorithms are developed under disjoint simulation assumptions, variously requiring fully differentiable GPU physics engines, massive parallel batch environments, or short-horizon dense rewards. Because these methods are rarely tested outside their native simulator regimes, practitioners cannot determine whether algorithms relying on analytic dynamics or extreme parallel throughput transfer to standard non-differentiable environments or long-horizon agentic tasks. This leaves the empirical boundaries and failure modes of existing model-based approaches unknown across differing simulator capabilities.