Open Problems

Uncertainty Quantification & Conformal Prediction

Principled Multiclass Extensions of Binary-Only Theoretical Guarantees in Prediction Methodology

Scope to testOpen
Strong candidate · 4/5 runs14 papers report this30% from 2025+

Generated automatically from the limitations stated in 14 papers (NeurIPS, ICLR, AISTATS), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

A large body of theory and tooling — covering uncertainty quantification, conformal methods, and attribution-style analyses — is derived and validated only for binary or single-label settings, with multiclass, multi-label, and open-set cases deferred to "future work" or handled via ad hoc one-versus-rest surrogates whose guarantees and sharpness are not established. As a result, practitioners applying these techniques to realistic multi-class problems either inherit unquantified approximation error or cannot use the methods at all (e.g., when only discrete predictions rather than continuous outputs are available). Each new paper re-derives binary results and re-defers the extension, so the field lacks a general recipe for lifting binary guarantees to K-class settings with provable validity and measured sharpness.

Why it matters

All downstream methods become deployable on multi-class, multi-label, and detection tasks with guarantees that mean something, ending the cycle of per-paper binary-only theory.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Systematic lifting framework: take 3–5 representative binary-only results and construct multiclass extensions via binary decomposition (one-vs-rest, pairwise, coded classification) versus direct multiclass formulations, then measure guarantee validity (empirical coverage/false-positive rates) and sharpness (interval widths, score dispersion) on standard multi-class benchmarks (CIFAR-10/100, tabular multiclass). Deliverable: a table of which lifting strategy preserves which guarantee at what sharpness cost.

  2. 2

    Relax interface assumptions: extend results that require unconstrained logits to settings with only discrete predictions or normalized outputs, using randomized smoothing / discretization-correction estimators; measure validity and sharpness loss relative to the logits-accessible case.

  3. 3

    Push beyond single-label: prototype multi-label and open-set variants where per-class binary reductions are natural, measuring per-label coverage and OOD false-alarm rates, using the framework from approach 1 as scaffolding.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

The sharpness penalty of multiclass extensions may grow so fast with class count that the guarantees, while valid, are vacuous in practice — making the honest conclusion "binary is all you can certify."

Sub-problems

  • Benchmarking and Extending Binary Feature Attribution Methods to Multi-Class and Label-Only Settings

    Multiple recent feature attribution methods are derived, analyzed, and evaluated strictly for binary classification tasks, often relying on continuous logit access. In practical deployments, machine learning systems frequently operate in multi-class regimes or expose only discrete prediction outputs. Because systematic cross-setting evaluations do not exist for these methods, it is unknown whether their attribution fidelity and theoretical properties persist when applied to multi-class problems and hard-label interfaces.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 14 papers

Nearest existing work

Related open problems

Uncertainty Quantification & Conformal Prediction

Scope to testOpen

Evaluating Uncertainty Quantification and Conformal Prediction Methods Beyond Binary Outcomes

Existing uncertainty quantification and conformal prediction formulations in this family have been developed and tested exclusively on binary outcome spaces ($y \in \{0, 1\}$). As a result, it is unknown whether their theoretical properties, coverage guarantees, and set efficiencies persist or degrade when applied to multiclass and continuous regression targets. Practitioners in non-binary domains are currently blocked from deploying these methods with verified reliability.

Possible candidate · 2/5 runs4 papers report this50% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.