Open Problems

Language Model Evaluation & Benchmarking

Methods and Benchmarks for LLM Judging, Ranking, and Evaluation That Work Without Token-Level Probabilities

Barrier to removeOpen
Possible candidate · 2/5 runs7 papers report this100% from 2025+

Generated automatically from the limitations stated in 7 papers (ACL, ICLR, EMNLP), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

A large fraction of scoring, ranking, and error-detection techniques for LLMs depend on logprobs, logits, or perplexities, which commercial APIs increasingly do not expose. This structurally blocks researchers and practitioners using closed models from applying state-of-the-art evaluation and ranking methods, forcing them onto weaker text-only alternatives or into hosting expensive open models. There is currently no systematic account of how much quality is lost when moving from probability-based to text-only judgments, nor of which probability signals can be reliably elicited from a model's own generated text.

Why it matters

Researchers using closed APIs regain access to confidence-aware evaluation and likelihood-based ranking; future papers can state which probability assumptions they actually need and offer text-only fallbacks.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Elicitation study: have LLMs self-report confidence, likelihood, and uncertainty through prompted natural-language outputs (verbalized confidence, sampling consistency, answer re-ranking by prompted plausibility) and measure agreement with true token-level logprobs on models where both are accessible. The gap between elicited and actual probabilities is the headline metric.

    Already done

    Paper [4] investigates Confidence-Probability Alignment by eliciting LLMs' self-reported confidence through various prompting techniques and measuring its agreement (correlation) with true token probabilities across multiple models and benchmarks.

    Other papers read (9)

    Checked 9/26/2026.

  2. 2

    Reimplement a representative set of logprob-dependent scoring/ranking methods in a text-only regime (e.g., sampling-based vote counts replacing likelihoods, multi-sample self-consistency replacing perplexity) and measure correlation with the original method's outputs and with ground truth.

    No prior work found in 10 papers

    None of the provided papers systematically evaluate or benchmark text-only adaptations of logprob- and logit-dependent LLM evaluation and ranking methods.

    Papers read (10)

    Checked 9/26/2026. Automated checks find about a third of existing work, so “no prior work found” doesn't mean the idea is new.

  3. 3

    Build a benchmark of API-accessible evaluation tasks where logprob-based and text-only methods are compared head-to-head, reporting accuracy, cost, and latency trade-offs.

    No prior work found in 10 papers

    While several papers evaluate and benchmark LLM-as-a-judge paradigms across various tasks and prompting methods, none systematically compare logprob-based evaluation techniques against text-only alternatives in terms of accuracy, cost, and latency.

    Papers read (10)

    Checked 9/26/2026. Automated checks find about a third of existing work, so “no prior work found” doesn't mean the idea is new.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Sampling-based substitutes may be so much more expensive (many generations per judgment) that they are impractical at API prices, and elicited confidence may correlate too poorly with true probabilities to support calibrated evaluation.

Sub-problems

  • Logit-Free Relevance Estimation for Black-Box LLM Reranking

    High-precision neural reranking with LLMs predominantly relies on pointwise token log-likelihoods or output logits to score query-document pairs. Proprietary, black-box LLM APIs frequently restrict or completely omit logit and logprob access, limiting API users to discrete text completions. Consequently, practitioners cannot apply state-of-the-art likelihood-based ranking frameworks to leading closed-source models without falling back to unstable listwise generation.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Language Model Evaluation & Benchmarking

Barrier to removeOpen

Label-Free and Transferable Calibration for Large Language Model Outputs

Existing calibration and uncertainty quantification techniques for large language models (LLMs) strictly require task-specific, ground-truth-labeled calibration sets with binarizable correctness annotations. In open-ended, domain-specific, or novel tasks where ground-truth labels do not exist or cannot be easily thresholded, these methods cannot be instantiated. Consequently, practitioners are structurally blocked from obtaining calibrated confidence scores, prediction intervals, or coverage guarantees for LLMs without first performing expensive per-task and per-model annotations.

Possible candidate · 2/5 runs5 papers report this100% from 2025+

Language Model Evaluation & Benchmarking

Barrier to removeOpen

Probability-Free LLM Evaluation and Benchmarking for Black-Box Models

Many standard evaluation, uncertainty estimation, and benchmarking techniques rely on token-level log-probabilities or soft output distributions. Proprietary and closed-access models (e.g., commercial chat interfaces and restrictive API endpoints) often only expose discrete text outputs, completely excluding them from probability-dependent benchmark comparisons. Relying on Monte Carlo sampling frequencies as a fallback is computationally prohibitive and prone to distortion from proprietary sampling temperatures or top-$p$ truncations. Consequently, existing evaluation suites cannot rigorously compare open-weight and closed-access models under a uniform probability-dependent protocol.

Strong candidate · 5/5 runs4 papers report this25% from 2025+

Language Model Evaluation & Benchmarking

Barrier to removePartly addressed

Multi-Model Evaluation and Ensembling Without Per-Step White-Box Logit Access

Current multi-model evaluation and collaborative generation frameworks require synchronous, step-by-step execution across multiple 32B–70B open-source LLMs with direct access to intermediate token logits. This white-box precondition structurally excludes closed-source, API-only models (which do not expose unconstrained per-step logits) from participating in these evaluation pipelines. It also prevents deployment on resource-constrained hardware unable to fit multiple large model footprints into memory simultaneously. Consequently, these benchmarking and decoding techniques cannot be applied to leading proprietary models or decentralized, asynchronous workflows.

Possible candidate · 2/5 runs3 papers report this100% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.