Language Model Evaluation & Benchmarking
Probability-Free LLM Evaluation and Benchmarking for Black-Box Models
Generated automatically from the limitations stated in 4 papers (EMNLP, NeurIPS), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Many standard evaluation, uncertainty estimation, and benchmarking techniques rely on token-level log-probabilities or soft output distributions. Proprietary and closed-access models (e.g., commercial chat interfaces and restrictive API endpoints) often only expose discrete text outputs, completely excluding them from probability-dependent benchmark comparisons. Relying on Monte Carlo sampling frequencies as a fallback is computationally prohibitive and prone to distortion from proprietary sampling temperatures or top-$p$ truncations. Consequently, existing evaluation suites cannot rigorously compare open-weight and closed-access models under a uniform probability-dependent protocol.
Why it matters
Enables standardized, rigorous benchmarking and confidence evaluation of closed-source, API-only models alongside open-weight models without requiring token log-probability access or expensive brute-force sampling.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Benchmark surrogate scoring: Implement and evaluate semantic consistency, perturbation sensitivity, and rank-order stability metrics derived purely from discrete text outputs, measuring their rank correlation against ground-truth log-probability metrics on open-weight models.
- 2
Sample-efficient black-box calibration: Develop and measure the sample efficiency of few-query estimation techniques (e.g., targeted verbalized confidence elicitation and paired comparison tournaments) compared to naive frequency-based sampling across standardized reasoning and QA datasets.
- 3
Cross-paradigm robustness audit: Conduct an empirical audit evaluating open-weight and closed-access models across existing benchmarks using strictly probability-free vs. probability-dependent metrics to measure rank shifts and discrepancy rates.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
If commercial API providers standardize the return of complete token log-probabilities by default, or if purely text-based surrogates fail to correlate reliably with true model uncertainty on complex long-form generation tasks.
Evidence
Each paper's own statement of the limitation, verbatim.
- Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?EMNLP 2025
Token/sequence probability methods cannot be applied to closed-source models (e.g., ChatGPT) that do not expose token probabilities, excluding those from comparison
- Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic SpaceNeurIPS 2024
Requires access to output token probabilities, unavailable for many proprietary/API LLMs (the frequency-based fallback variant is much more expensive)
- Efficient LLM Comparative Assessment: A Product of Experts Framework for Pairwise ComparisonsEMNLP 2024
Requires access to soft token-level LLM probabilities; for closed APIs exposing only sampled binary decisions the method is much less effective
- G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentEMNLP 2023
Depends on closed, paid LLM APIs (GPT-3.5/GPT-4); GPT-4 token probabilities must be approximated via sampling, and model updates may make results non-reproducible
Nearest existing work
- A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and RecommendationsEMNLP 2024
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceICML 2024
- On Speeding Up Language Model EvaluationICLR 2025
- Towards Robust QA Evaluation via Open LLMsSIGIR 2024
- When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM EnsemblingICLR 2026
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model LeaderboardsACL 2024
- Evaluation of Large Language Models via Coupled Token GenerationAISTATS 2026
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the WildICLR 2025
- Predicting the Performance of Black-box Language Models with Follow-up QueriesNeurIPS 2025
- An Empirical Analysis of Uncertainty in Large Language Model EvaluationsICLR 2025
- Prompt-to-Leaderboard: Prompt-Adaptive LLM EvaluationsICML 2025
- Cheaply Estimating Inference Efficiency Metrics for Autoregressive Transformer ModelsNeurIPS 2023
- Don’t Pass@$k$: A Bayesian Framework for Large Language Model EvaluationICLR 2026
- Calibrating the Confidence of Large Language Models by Eliciting FidelityEMNLP 2024
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMsICLR 2026
Related open problems
Language Model Evaluation & Benchmarking
Methods and Benchmarks for LLM Judging, Ranking, and Evaluation That Work Without Token-Level Probabilities
A large fraction of scoring, ranking, and error-detection techniques for LLMs depend on logprobs, logits, or perplexities, which commercial APIs increasingly do not expose. This structurally blocks researchers and practitioners using closed models from applying state-of-the-art evaluation and ranking methods, forcing them onto weaker text-only alternatives or into hosting expensive open models. There is currently no systematic account of how much quality is lost when moving from probability-based to text-only judgments, nor of which probability signals can be reliably elicited from a model's own generated text.
Language Model Evaluation & Benchmarking
Label-Free and Transferable Calibration for Large Language Model Outputs
Existing calibration and uncertainty quantification techniques for large language models (LLMs) strictly require task-specific, ground-truth-labeled calibration sets with binarizable correctness annotations. In open-ended, domain-specific, or novel tasks where ground-truth labels do not exist or cannot be easily thresholded, these methods cannot be instantiated. Consequently, practitioners are structurally blocked from obtaining calibrated confidence scores, prediction intervals, or coverage guarantees for LLMs without first performing expensive per-task and per-model annotations.
Language Model Evaluation & Benchmarking
Multi-Model Evaluation and Ensembling Without Per-Step White-Box Logit Access
Current multi-model evaluation and collaborative generation frameworks require synchronous, step-by-step execution across multiple 32B–70B open-source LLMs with direct access to intermediate token logits. This white-box precondition structurally excludes closed-source, API-only models (which do not expose unconstrained per-step logits) from participating in these evaluation pipelines. It also prevents deployment on resource-constrained hardware unable to fit multiple large model footprints into memory simultaneously. Consequently, these benchmarking and decoding techniques cannot be applied to leading proprietary models or decentralized, asynchronous workflows.