Open Problems

Efficient Attention & State-Space Models

Evaluating Toy-Task Inductive Biases and Mechanistic Claims Across Standard Pre-training and Deep Architectures

Scope to testOpen
Strong candidate · 1/1 runs4 papers report this75% from 2025+

Generated automatically from the limitations stated in 4 papers (ICLR, ICML, NeurIPS), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Architectural mechanisms and efficiency claims developed for state-space models and efficient attention variants are overwhelmingly validated only on shallow (1–6 layer) networks trained on small synthetic or algorithmic tasks. Because these mechanisms have not been evaluated across standard natural language pre-training corpora or deeper architectures, the community cannot determine whether the reported properties persist, degrade, or dissolve at scale. Consequently, theoretical insights and inductive biases identified on synthetic benchmarks remain unverified hypotheses for real-world language modeling.

Why it matters

Establishes an empirical baseline showing which synthetic-task advantages survive scaling to deep natural language models, preventing wasted compute on mechanisms that only function in toy regimes.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Standardized Multi-Scale Benchmark: Take 3–4 recent efficient attention and state-space architectures previously tested only on synthetic tasks, scale them to standard sizes (125M to 1.3B parameters, 12–24 layers), and measure perplexity and downstream task accuracy on standard pre-training corpora (e.g., SlimPajama or FineWeb).

  2. 2

    Algorithmic-to-Language Transfer Diagnostic: Train small synthetic models and standard language models side-by-side using identical architecture variants, measuring whether specific mechanistic capabilities (such as state tracking, induction heads, and associative recall) correlate with standard natural language performance.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If concurrent industrial pre-training efforts independently release full-scale pre-training ablations for these exact model variants, or if the compute budget required to demonstrate decisive separation across 1B+ models exceeds academic resources.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Efficient Attention & State-Space Models

Scope to testOpen

A Matched-Scale Testbed and Extrapolation Protocol for Testing Whether Toy-Scale Evidence Survives at Realistic Scale

Across theory-driven method papers, the empirical section reliably ends where the claim begins: guarantees about scaling, efficiency, or sample complexity are validated only on instances orders of magnitude below the regime where they matter, because realistic-scale runs are unaffordable for a single lab and no shared, property-matched task suites exist for buying at-scale evidence cheaply. The consequence is that scalability claims are effectively unfalsifiable — reviewers can neither confirm nor reject them, and methods whose advantages appear only above ~10⁸ parameters or on real data are adopted or discarded on faith. Because every group builds its own private toys, no evidence accumulates: the field cannot answer the basic question of how often toy-scale method rankings survive at realistic scale, so nobody knows what the current synthetic-only evidence base is actually worth.

Possible candidate · 2/5 runs32 papers report this56% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.