Open Problems

Inference Efficiency & Long Context

Systematic Cross-Scale Robustness and Evaluation of Efficient Inference and Long-Context Methods

Effect to explainOpen
Possible candidate · 3/5 runs8 papers report this75% from 2025+

Generated automatically from the limitations stated in 8 papers (ICML, ICLR, ACL), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Existing inference efficiency and long-context techniques are predominantly evaluated only on small models (1B–13B) and narrow synthetic tasks or single-pipeline setups. When applied to larger architectures (e.g., 70B) or realistic extended contexts, several methods exhibit severe accuracy regressions, unanticipated KV-cache memory overheads, or net throughput slowdowns. Because no unified evaluation exists across model scales and standardized inference runtimes, practitioners cannot predict whether reported efficiency gains will hold or catastrophically fail in real-world deployments.

Why it matters

Provides the first validated map of which efficiency techniques generalize across model sizes and context lengths, preventing deployment regressions and guiding future algorithm design toward actual scale bottlenecks.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Unified Evaluation Harness: Build a standardized benchmark harness on a unified inference stack (e.g., vLLM/SGLang) to evaluate open-source speculative decoding, token pruning, and KV-cache compression methods across Llama model sizes from 1B to 70B, measuring wall-clock throughput, memory consumption, and task accuracy.

  2. 2

    Context Scaling and Failure Profiling: Evaluate candidate efficiency methods across systematic context length increments (up to 32K+ tokens) on multi-task benchmarks (such as LongBench and MMLU), measuring exact degradation thresholds, out-of-memory failure points, and token-level latency overheads.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Rapid architectural evolution (such as native adoption of Multi-head Latent Attention or hybrid SSM backbones) may render post-hoc Transformer acceleration techniques irrelevant. Additionally, the GPU compute required to rigorously sweep 70B models across 32K+ contexts could exceed academic resource constraints.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Inference Efficiency & Long Context

Barrier to removePartly addressed

Calibration-Free and Gradient-Free Predictors for Efficient Inference

Current efficiency mechanisms for large language models and long-context inference—such as dynamic quantization, layer skipping, and speculative routing—rely on offline calibration phases requiring gradient access and curated calibration datasets. Consequently, these acceleration techniques cannot be deployed in privacy-sensitive environments where data cannot be stored, nor on black-box model APIs where backpropagation is unavailable. Furthermore, because predictors are model-specific, serving heterogeneous or frequently updated models incurs repeated offline profiling overhead that impedes zero-shot deployment.

Possible candidate · 2/5 runs4 papers report this50% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.