Language Model Evaluation & Benchmarking
Multi-Model Evaluation and Ensembling Without Per-Step White-Box Logit Access
Generated automatically from the limitations stated in 3 papers (ACL, EMNLP), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Current multi-model evaluation and collaborative generation frameworks require synchronous, step-by-step execution across multiple 32B–70B open-source LLMs with direct access to intermediate token logits. This white-box precondition structurally excludes closed-source, API-only models (which do not expose unconstrained per-step logits) from participating in these evaluation pipelines. It also prevents deployment on resource-constrained hardware unable to fit multiple large model footprints into memory simultaneously. Consequently, these benchmarking and decoding techniques cannot be applied to leading proprietary models or decentralized, asynchronous workflows.
Why it matters
Enables multi-model ensembling and evaluation methods to incorporate proprietary API models and run on consumer-grade hardware. It removes the dependency on dedicated multi-GPU clusters for multi-model verification.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Sequence-level rank and consensus aggregation: Implement black-box ensembling protocols that operate solely on final candidate sequences and sequence-level confidence signals rather than per-step token logits, measuring output quality and metric correlation against synchronous logit-based baselines across standard reasoning benchmarks (e.g., MMLU, GSM8K).
- 2
Asynchronous cascading verification: Build a pipeline where small local models generate candidate responses that are asynchronously evaluated or filtered by API endpoints, measuring latency, memory usage, and task accuracy relative to synchronous multi-model decoding.
- 3
Lightweight surrogate calibration: Use a single compact local model (e.g., 3B–8B) to approximate ensemble distribution shifts from black-box text samples, evaluating the degradation in scoring fidelity against full 32B/70B multi-model ensembles.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
If fine-grained token logits contain high-entropy calibration information that cannot be approximated or recovered from sampled text outputs and top-k logprobs, black-box approaches will consistently underperform white-box ensembles. Furthermore, if API providers standardize full intermediate state access, the motivation for black-box ensembling algorithms will diminish.
Evidence
Each paper's own statement of the limitation, verbatim.
- Zero-shot Large Language Models for Automatic Readability AssessmentACL 2026
Requires inference from large open-source LLMs (best results from 70B/32B models) with access to output token logits, excluding API-only models or very small local models
- A Scalable Multi-LLM Collaboration System with Retrieval-based Selection and Exploration-Exploitation-Driven EnhancementACL 2026
No efficiency optimization: no constraints on which or how many LLMs are selected, requiring substantial compute and inference time; not deployable on resource-constrained devices
- Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic UnitsEMNLP 2025
Requires running multiple LLMs' forward passes locally at every generation step, increasing compute and precluding API-based asynchronous inference; not deployable on resource-constrained hardware
Nearest existing work
- A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and RecommendationsEMNLP 2024
- Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic UnitsEMNLP 2025
- Cheaply Estimating Inference Efficiency Metrics for Autoregressive Transformer ModelsNeurIPS 2023
- Rethinking LLM Ensembling from the Perspective of Mixture ModelsICML 2026
- SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online FeedbackNeurIPS 2025
- When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM EnsemblingICLR 2026
- The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language ModelsICML 2025
- User-side Model Consistency Monitoring for Open Source Large Language Models Inference ServicesACL 2025
- SLM-MUX: Orchestrating Small Language Models for ReasoningICLR 2026
- Speculate, then Collaborate: Fusing Knowledge of Language Models during DecodingICML 2025
- Think in Parallel, Answer as One: Logit Averaging for Open-Ended ReasoningICLR 2026
- Token-Level LLM Collaboration via FusionRouteICML 2026
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the WildICLR 2025
- AdaFuse: Adaptive Ensemble Decoding for Large Language ModelsACL 2026
- On Speeding Up Language Model EvaluationICLR 2025
Related open problems
Language Model Evaluation & Benchmarking
Methods and Benchmarks for LLM Judging, Ranking, and Evaluation That Work Without Token-Level Probabilities
A large fraction of scoring, ranking, and error-detection techniques for LLMs depend on logprobs, logits, or perplexities, which commercial APIs increasingly do not expose. This structurally blocks researchers and practitioners using closed models from applying state-of-the-art evaluation and ranking methods, forcing them onto weaker text-only alternatives or into hosting expensive open models. There is currently no systematic account of how much quality is lost when moving from probability-based to text-only judgments, nor of which probability signals can be reliably elicited from a model's own generated text.
Language Model Evaluation & Benchmarking
Label-Free and Transferable Calibration for Large Language Model Outputs
Existing calibration and uncertainty quantification techniques for large language models (LLMs) strictly require task-specific, ground-truth-labeled calibration sets with binarizable correctness annotations. In open-ended, domain-specific, or novel tasks where ground-truth labels do not exist or cannot be easily thresholded, these methods cannot be instantiated. Consequently, practitioners are structurally blocked from obtaining calibrated confidence scores, prediction intervals, or coverage guarantees for LLMs without first performing expensive per-task and per-model annotations.
Language Model Evaluation & Benchmarking
Probability-Free LLM Evaluation and Benchmarking for Black-Box Models
Many standard evaluation, uncertainty estimation, and benchmarking techniques rely on token-level log-probabilities or soft output distributions. Proprietary and closed-access models (e.g., commercial chat interfaces and restrictive API endpoints) often only expose discrete text outputs, completely excluding them from probability-dependent benchmark comparisons. Relying on Monte Carlo sampling frequencies as a fallback is computationally prohibitive and prone to distortion from proprietary sampling temperatures or top-$p$ truncations. Consequently, existing evaluation suites cannot rigorously compare open-weight and closed-access models under a uniform probability-dependent protocol.