Open Problems

Classical Supervised Learning: Trees, Ranking & Aggregation

Empirical Robustness and Scaling of Jacobian- and Pairwise-Based Aggregation Beyond BERT-Base

Barrier to removeOpen
Possible candidate · 2/5 runs4 papers report this50% from 2025+

Generated automatically from the limitations stated in 4 papers (ICML, AISTATS, ACL), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Several classical supervised learning, ranking, and aggregation methods rely on per-example parameter Jacobians and all-pairs batch formulations, restricting their empirical verification to the linear readout layers of English BERT-base and ResNet-50. As a result, it is unknown whether the spectral variance assumptions and downstream performance of these algorithms hold on larger transformer architectures, decoder-only models, or multilingual representations. Practitioners currently have no evidence indicating whether these techniques fail gracefully or break down entirely when scaled beyond small, English-specific encoder backbones.

Why it matters

Establishes the empirical boundaries of Jacobian- and pairwise-based aggregation methods, determining whether these classical learning formulations can be reliably applied to modern transformer scales and non-English domains.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Conduct a cross-architecture benchmark of existing Jacobian- and pairwise-aggregation algorithms across larger transformer models (e.g., RoBERTa-Large, DeBERTa-v3, and small decoder-only LLMs) and multilingual backbones (e.g., XLM-RoBERTa); measure memory footprint, wall-clock scaling, and downstream ranking performance.

  2. 2

    Evaluate the trade-offs of restricted layer Jacobians versus parameter-efficient Jacobian approximations (such as low-rank projection or adapter-based Jacobians); measure approximation error and downstream aggregation accuracy.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If computing per-example parameter Jacobians proves fundamentally prohibitive on larger backbones without introducing approximation errors that render the aggregation method ineffective, or if frozen embedding baselines consistently match the method across all scale regimes.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.