Open Problems

Language Model Interpretability & In-Context Learning

Benchmarking Interpretability and In-Context Learning Mechanisms Beyond Toy Synthetic PCFGs

Scope to testPartly addressed
Possible candidate · 3/5 runs5 papers report this60% from 2025+

Generated automatically from the limitations stated in 5 papers (EMNLP, ICML, NAACL), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Mechanistic interpretability and in-context learning findings in formal linguistic settings currently rely on tiny, unambiguous PCFGs with vocabularies of fewer than 100 words and corpora of only ~20K sentences evaluated on toy models like nanoGPT. It remains completely unknown whether the specific internal circuits, parsing behaviors, and induction mechanisms documented in these toy studies survive when evaluated on richer formal grammars, ambiguous grammars, larger vocabularies, or natural syntax. Without systematic robustness evaluations across these broader settings, interpretability claims cannot be reliably extrapolated beyond the micro-benchmarks on which they were discovered.

Why it matters

It enables researchers to know precisely which mechanistic interpretability and in-context learning findings reflect generalizable model behaviors versus artifacts of minimal synthetic grammar setups.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Construct a parametric benchmark of grammars that systematically increases vocabulary size, introduces structural ambiguity, and incorporates diverse word-order topologies (e.g., VSO, OSV), measuring whether previously identified circuit structures and parsing mechanisms in nanoGPT-scale models remain stable or degrade.

  2. 2

    Evaluate existing interpretability and in-context learning probes across a spectrum of scale—from synthetic PCFGs to semi-synthetic templates and natural language corpora (e.g., Penn Treebank / Universal Dependencies)—measuring the consistency of the identified mechanistic heads and representations across data regimes.

  3. 3

    Test established PCFG-derived interpretability hypotheses on larger open-weight language models (e.g., 1B–7B parameter models) across unambiguous versus ambiguous grammar classes, measuring the degree to which scale washes out or alters the mechanistic behavior observed on toy models.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If scaling synthetic grammars to include ambiguity and larger vocabularies causes standard interpretability tools (like activation patching) to fail completely due to polysemanticity, making it intractable to obtain clear mechanistic signal outside the simplest toy setups.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Language Model Interpretability & In-Context Learning

Scope to testOpen

Cross-Linguistic Robustness of Interpretability and Cognitive Alignment in Language Models

Mechanistic interpretability claims and brain-to-LM alignment results currently rest almost entirely on English-only stimuli, English-trained models, and English-speaking participant data. Because cross-linguistic evaluation has not been conducted across these methods, it remains unknown whether identified circuits, induction heads, and representational alignments reflect general linguistic processing or are artifacts of English syntax and data abundance. Consequently, researchers cannot rely on existing interpretability mechanisms when analyzing multilingual models or studying cognitive language processing across typologically diverse languages.

Strong candidate · 5/5 runs9 papers report this33% from 2025+

Language Model Interpretability & In-Context Learning

Barrier to removeOpen

Benchmarking Whether LLM-Based Methods Survive a Change of Model: A Transfer Study Across Access Levels and Model Families

Methods built on one LLM — whether they depend on its logits, its instruction-following behavior, its pretraining data, or its annotation quality — are currently reported on one or two models and simply assumed to generalize. Because the assumptions are implicit, a follow-up researcher cannot tell whether a failure on a new model is a flaw of the method or an unexamined dependency on the original model's scale, data, or API surface. This blocks cumulative progress: every new model release forces methods to be silently re-validated or abandoned, and conclusions attributed to "LLMs" are really conclusions about a single backbone.

Strong candidate · 5/5 runs7 papers report this50% from 2025+

Language Model Interpretability & In-Context Learning

UnclassifiedOpen

A benchmark of naturally occurring hallucinations with model-generated ground truth, replacing synthetic entity-swap surrogates for detection evaluation

Today, hallucination detection methods are trained and evaluated against synthetic artifacts — entity-swapped summaries, outputs from a different model than the one under test — rather than hallucinations that actually arise during generation. This means reported detection numbers do not measure whether a method works on the failure mode that matters, and the modest predictive results (e.g., R² ≈ 0.27 on hallucination scores) may partly reflect this distribution mismatch. Detection validated only on 7B–13B open-weight models with synthetic data cannot be trusted as evidence about frontier-model behavior. Without a natural-hallucination benchmark, every detection paper inherits the same unquantified gap between "detects injected errors" and "detects real hallucinations."

Weak candidate · 1/3 runs6 papers report this33% from 2025+

Language Model Interpretability & In-Context Learning

Scope to testOpen

Testing the Transferability of Toy-Model Theoretical Guarantees to Deep Transformer Architectures

Theoretical analyses of in-context learning and mechanistic interpretability rely almost entirely on heavily idealized architectures, such as single-layer or two-layer transformers with linear attention, orthogonal embeddings, and no residual connections or normalization. Because these proofs make structural assumptions that diverge substantially from production models, it is currently unknown whether their theoretical guarantees hold in real large language models or function purely as loose analogies. Without empirical validation across the gap between toy settings and deep models, downstream interpretability methods risk relying on mechanistic claims that fail in standard architectures.

Possible candidate · 3/5 runs5 papers report this60% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.