Open Problems

Bandits & Online Learning

A Standard Benchmark Suite and Evaluation Protocol for Validating Bandit, Online Learning, and Sequential Decision Algorithms on Real-World Logged Data

Scope to testOpen
Possible candidate · 3/5 runs35 papers report this50% from 2025+

Generated automatically from the limitations stated in 35 papers (ICML, NeurIPS, AISTATS), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Across dozens of recent papers in sequential decision making, the empirical section is a small synthetic simulation with hand-picked parameters, and authors explicitly acknowledge that real-world validation is missing. The consequence is that theoretical guarantees and synthetic results cannot be translated into practice: practitioners cannot tell which algorithms hold up on logged production data, and researchers cannot compare against baselines because no shared real-data evaluation standard exists. Structurally, the field lacks accessible, well-curated real-world logged datasets (recommendation, clinical, resource-allocation, contract design) with standardized protocols, so every paper improvises its own toy experiment.

Why it matters

Researchers can empirically validate theoretical claims on real data with minimal effort, removing the single most cited limitation in recent sequential-decision papers; practitioners get reliable, comparable evidence for algorithm selection; and the benchmark becomes a standard target that future theory papers can and will be judged against.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Curate and release a benchmark suite of 5–10 real logged datasets suited to bandit/online-learning evaluation (e.g., open bandit datasets, clinical trial logs, e-commerce recommendation logs, pricing/knapsack-style allocation logs). Build standardized loaders, evaluation splits, and metrics. Measure: number of datasets integrated, algorithm API compatibility, reproducibility of reported metrics across three existing published algorithms run by an independent student.

  2. 2

    Define and publish an evaluation protocol: how to handle partial feedback, off-policy evaluation, safe exploration constraints, and confidence-interval reporting on logged data. Measure: agreement of offline evaluation with online deployment on at least one live or replay-based setting; variance of rankings across baselines.

  3. 3

    Re-run published state-of-the-art algorithms from the papers above on the suite and produce the first real-data leaderboard, measuring regret, reward, and computational cost against their synthetic-only reported results.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

The available public logged datasets may be too few, too small, or too legally constrained to support a genuinely diverse suite, and off-policy evaluation may be too high-variance to give reliable comparisons—leaving the benchmark as noisy and inconclusive as the synthetic experiments it replaces.

Sub-problems

  • Empirical Robustness and Benchmarking of Mechanism and Contract Design on Real-World Data

    Current algorithmic mechanism and contract design methods are evaluated almost exclusively on small, synthetic instances with sampled utilities, hand-picked externality forms, and random outcome distributions. Because these methods have not been tested across real-world platform, procurement, or contract data, it is unknown whether their theoretical performance guarantees and algorithmic rankings hold in practice. This lack of empirical evaluation leaves practitioners unable to verify whether proposed mechanisms degrade under realistic correlation structures, heavy-tailed utilities, and operational scale.

  • High-Dimensional and Large-Scale Empirical Evaluation of Riemannian Representation Learning

    Current methods in Riemannian and hyperbolic representation learning have been validated almost entirely on low-dimensional toy manifolds ($d \le 3$) and small sample sizes ($N \le 500$). Because no systematic evaluation across standard high-dimensional ($d \gg 100$) and large-scale ($N \ge 10^5$) benchmarks exists, it is unknown whether the theoretical benefits of these geometries survive in realistic settings. Practitioners cannot determine whether these methods suffer from numerical instability, optimization divergence, or vanishing metric advantages when scaled beyond toy regimes.

  • Evaluating Modern Bayesian Optimization and Experimental Design on Real-World Scientific Benchmarks

    Current algorithmic developments in Bayesian optimization and model-based experimental design are predominantly verified on synthetic test functions and small toy combinatorial instances. Consequently, practitioners in scientific domains lack empirical evidence regarding whether these methods maintain their sample efficiency and theoretical advantages on complex real-world landscapes with non-stationary noise, high-dimensional constraints, and structured discrete search spaces. Without systematic cross-domain evaluation, it is unknown which algorithmic paradigms actually generalize beyond mathematical testbeds to applied settings like protein engineering or hardware accelerator tuning.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 35 papers

Nearest existing work

Related open problems

Bandits & Online Learning

Barrier to removeOpen

Instance-Dependent Robustness Guarantees Beyond Worst-Case Regret Bounds for Constrained Online Learning

Today, online learning guarantees are only available under restrictive structural conditions—honest agents dominating malicious ones, known adversary classes, Slater's condition, smoothness assumptions—and when these fail, the bounds become vacuous or degrade catastrophically. Practitioners cannot distinguish between "the bound is loose" and "the problem is genuinely hard," because no instance-dependent analysis exists that quantifies what is actually achievable on a given problem instance. This blocks deployment in settings with few honest agents, near-zero constraint slacks, or moderately hard function classes, precisely where robust methods matter most.

Possible candidate · 2/4 runs6 papers report this67% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.