Open Problems

Inference Efficiency & Long Context

Calibration-Free and Gradient-Free Predictors for Efficient Inference

Barrier to removePartly addressed
Possible candidate · 2/5 runs4 papers report this50% from 2025+

Generated automatically from the limitations stated in 4 papers (EMNLP, ICML), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current efficiency mechanisms for large language models and long-context inference—such as dynamic quantization, layer skipping, and speculative routing—rely on offline calibration phases requiring gradient access and curated calibration datasets. Consequently, these acceleration techniques cannot be deployed in privacy-sensitive environments where data cannot be stored, nor on black-box model APIs where backpropagation is unavailable. Furthermore, because predictors are model-specific, serving heterogeneous or frequently updated models incurs repeated offline profiling overhead that impedes zero-shot deployment.

Why it matters

Enables privacy-preserving, plug-and-play acceleration of long-context models and proprietary inference APIs without offline dataset collection or gradient computation.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Test-time forward-only profiling: Build dynamic pruning and quantization selectors driven solely by streaming activation statistics (e.g., kurtosis, attention entropy) without backpropagation, measuring throughput gains and perplexity degradation against gradient-calibrated baselines.

  2. 2

    Proxy-model calibration transfer: Calibrate predictor heads on small, open-source proxy models and test zero-shot transferability to larger, black-box target models, measuring cross-architecture routing accuracy and downstream task retention.

  3. 3

    Non-parametric logit calibration: Replace softmax probability scaling with rank-order statistics and uncertainty measures that require no offline dataset, measuring calibration error and decision correctness on long-context question answering.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If the decision boundaries required for effective dynamic inference are tightly coupled to second-order loss gradients that forward-only activation statistics fundamentally fail to capture without substantial accuracy loss.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Inference Efficiency & Long Context

Effect to explainOpen

Systematic Cross-Scale Robustness and Evaluation of Efficient Inference and Long-Context Methods

Existing inference efficiency and long-context techniques are predominantly evaluated only on small models (1B–13B) and narrow synthetic tasks or single-pipeline setups. When applied to larger architectures (e.g., 70B) or realistic extended contexts, several methods exhibit severe accuracy regressions, unanticipated KV-cache memory overheads, or net throughput slowdowns. Because no unified evaluation exists across model scales and standardized inference runtimes, practitioners cannot predict whether reported efficiency gains will hold or catastrophically fail in real-world deployments.

Possible candidate · 3/5 runs8 papers report this75% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.