Language Model Evaluation & Benchmarking
Label-Free and Transferable Calibration for Large Language Model Outputs
Generated automatically from the limitations stated in 5 papers (ICML, ICLR, AISTATS), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Existing calibration and uncertainty quantification techniques for large language models (LLMs) strictly require task-specific, ground-truth-labeled calibration sets with binarizable correctness annotations. In open-ended, domain-specific, or novel tasks where ground-truth labels do not exist or cannot be easily thresholded, these methods cannot be instantiated. Consequently, practitioners are structurally blocked from obtaining calibrated confidence scores, prediction intervals, or coverage guarantees for LLMs without first performing expensive per-task and per-model annotations.
Why it matters
Enables trustworthy uncertainty quantification, risk-controlled generation, and reliable confidence scoring on open-ended generation tasks where no ground-truth calibration labels exist.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Cross-task conformal calibration transfer: Fit conformal prediction thresholds on established labeled datasets and test their coverage guarantees when transferred zero-shot to target domains, measuring empirical coverage violation and interval width across diverse generative tasks.
- 2
Self-consistency and ensemble agreement as surrogate labels: Construct calibration maps using consistency metrics across sampled decoding paths and perturbation prompts rather than true labels, measuring Expected Calibration Error (ECE) and Brier scores against supervised calibration baselines.
- 3
Continuous surrogate evaluation calibration: Formulate calibration over continuous text evaluation metrics (e.g., semantic similarity distributions) instead of arbitrary binarization thresholds, measuring interval efficiency and calibration stability across varied summarization and reasoning tasks.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
If cross-domain distribution shifts in LLM representations prove so irregular that label-free calibration guarantees cannot bound empirical coverage, the method will reduce to unprincipled heuristics. Additionally, if cheap LLM-as-a-judge annotations become accurate enough to generate high-fidelity ground truth on the fly, the need for label-free calibration may dissolve.
Evidence
Each paper's own statement of the limitation, verbatim.
- The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and TrustICML 2026
Requires a labeled calibration set with binary correctness labels per task, and correctness must be binarizable (e.g., SCITLDR uses an arbitrary 0.3 ROUGE-L threshold with ablations relegated to appendix)
- Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time ScalingICLR 2026
Requires a one-time pre-computed calibration dataset containing labeled responses for every specific LLM and PRM pairing used.
- Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-JudgeICML 2026
Requires a labeled calibration set (~60-100 examples needed before MAE plateaus), which is unavailable when no ground-truth labels exist for the target task.
- InnerThoughts: Disentangling Representations and Predictions in Large Language ModelsAISTATS 2025
Requires a large labeled training set (e.g. ~68k-128k examples per dataset), unlike unsupervised calibration methods; results are only zero-shot and restricted to multiple-choice
- Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal PredictionEMNLP 2025
Requires a sufficiently large labeled calibration dataset to ensure stable coverage and narrow interval widths.
Nearest existing work
- A Survey of Confidence Estimation and Calibration in Large Language ModelsNAACL 2024
- SConU: Selective Conformal Uncertainty in Large Language ModelsACL 2025
- Calibrating the Confidence of Large Language Models by Eliciting FidelityEMNLP 2024
- Large Language Models Must Be Taught to Know What They Don’t KnowNeurIPS 2024
- Uncertainty in Language Models: Assessment through Rank-CalibrationEMNLP 2024
- Calibrating LLM Confidence by Probing Perturbed Representation StabilityEMNLP 2025
- Calibration Across Layers: Understanding Calibration Evolution in LLMsEMNLP 2025
- Calibrating Expressions of CertaintyICLR 2025
- GrACE: A Generative Approach to Better Confidence Elicitation and Efficient Test-Time Scaling in Large Language ModelsACL 2026
- Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models’ Uncertainty?ACL 2025
- Calibrating Zero-shot Cross-lingual (Un-)structured PredictionsEMNLP 2022
- Calibrating Large Language Models Using Their Generations OnlyACL 2024
- Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language ModelsICLR 2026
- A Close Look into the Calibration of Pre-trained Language ModelsACL 2023
- Beyond the Score: Uncertainty-Calibrated LLMs for Automated Essay AssessmentEMNLP 2025
Related open problems
Language Model Evaluation & Benchmarking
Methods and Benchmarks for LLM Judging, Ranking, and Evaluation That Work Without Token-Level Probabilities
A large fraction of scoring, ranking, and error-detection techniques for LLMs depend on logprobs, logits, or perplexities, which commercial APIs increasingly do not expose. This structurally blocks researchers and practitioners using closed models from applying state-of-the-art evaluation and ranking methods, forcing them onto weaker text-only alternatives or into hosting expensive open models. There is currently no systematic account of how much quality is lost when moving from probability-based to text-only judgments, nor of which probability signals can be reliably elicited from a model's own generated text.
Language Model Evaluation & Benchmarking
Probability-Free LLM Evaluation and Benchmarking for Black-Box Models
Many standard evaluation, uncertainty estimation, and benchmarking techniques rely on token-level log-probabilities or soft output distributions. Proprietary and closed-access models (e.g., commercial chat interfaces and restrictive API endpoints) often only expose discrete text outputs, completely excluding them from probability-dependent benchmark comparisons. Relying on Monte Carlo sampling frequencies as a fallback is computationally prohibitive and prone to distortion from proprietary sampling temperatures or top-$p$ truncations. Consequently, existing evaluation suites cannot rigorously compare open-weight and closed-access models under a uniform probability-dependent protocol.
Language Model Evaluation & Benchmarking
Multi-Model Evaluation and Ensembling Without Per-Step White-Box Logit Access
Current multi-model evaluation and collaborative generation frameworks require synchronous, step-by-step execution across multiple 32B–70B open-source LLMs with direct access to intermediate token logits. This white-box precondition structurally excludes closed-source, API-only models (which do not expose unconstrained per-step logits) from participating in these evaluation pipelines. It also prevents deployment on resource-constrained hardware unable to fit multiple large model footprints into memory simultaneously. Consequently, these benchmarking and decoding techniques cannot be applied to leading proprietary models or decentralized, asynchronous workflows.