Open Problems

Algorithmic Fairness

A Unified Formulation for Multiclass and Regression Targets under Postprocessing-Based Guarantees

Scope to testPartly addressed
Strong candidate · 5/5 runs10 papers report this83% from 2025+

Generated automatically from the limitations stated in 10 papers (ICML, ICLR, CVPR), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Ten independent lines of work — spanning fairness postprocessing, conformal prediction, and related certification-style methods — each stop at binary or, at best, categorical classification targets. Continuous outcomes and structured prediction are simply out of reach with current formulations, which means a large fraction of real prediction problems (prices, dosages, rankings, likelihoods) get either no guarantee at all or a crude discretization that distorts the guarantee. Practitioners must choose between losing the guarantee and losing the task, and every paper defers the same extension to "future work" rather than solving it jointly.

Why it matters

Every one of the ten methods becomes applicable to regression and multiclass problems without redesigning each from scratch, and the theory gives a single error-accounting template that future postprocessing results can inherit instead of re-deriving.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Re-derive the core postprocessing condition (Bayes-optimality / exchangeability / group-conditional calibration) for a continuous target, starting from a discretization-and-refinement argument: bin the target, apply the existing binary machinery per bin, then characterize the approximation error as bin width shrinks. Measure: coverage / fairness-gap of the resulting predictor versus a binning-free oracle, as a function of resolution.

  2. 2

    Extend the multiclass case first, since it is the smaller step: replace the single threshold with a score-vector construction, prove the guarantee generalizes from binary to K classes with a factor depending on K, and measure the empirical gap between the K-class guarantee and K one-vs-rest binary guarantees on standard tabular benchmarks.

  3. 3

    Test on regression by mapping the continuous problem to a binary one on derived events (e.g., exceedance probabilities across a grid of thresholds), verifying that the family of per-threshold guarantees stitches into a valid guarantee over the full output space; measure integrated coverage/calibration error against conformalized baselines.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If the family of binary guarantees cannot be shown to jointly hold without strong independence or union-bound assumptions, the stitched regression/multiclass guarantee becomes vacuous or too loose to be useful empirically.

Sub-problems

  • Benchmarking and Extending Conformal Uncertainty Quantification Across Regression and Structured Prediction Tasks

    Existing uncertainty quantification and conformal prediction methods are predominantly formulated and evaluated solely on classification benchmarks, leaving their empirical behavior on continuous and structured outputs uncharacterized. Practitioners in continuous domains (such as regression or object detection) cannot reliably determine whether non-conformity scores and theoretical coverage guarantees transfer or degrade under real-world continuous and spatial distributions. Without cross-task evaluation, it remains unknown whether these methods suffer systematic failure modes when transitioning away from discrete label spaces.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 10 papers

Nearest existing work

Related open problems

Algorithmic Fairness

Barrier to removePartly addressed

Benchmarking Algorithmic Fairness Methods Under Violations of Sensitive Attribute Preconditions

Existing algorithmic fairness methods are almost universally evaluated under rigid preconditions: fully observed or missing-at-random discrete group labels, bounded minority group representations ($\rho$-fraction), and absence of feature-dependent annotation noise. In real-world deployments, sensitive attributes are often unannotated, continuous, severely underrepresented, or corrupted by feature-dependent noise. Because fairness methods have not been systematically evaluated across these non-idealized regimes, practitioners cannot determine which, if any, algorithmic debiasing approaches retain their fairness guarantees or utility tradeoffs when theoretical preconditions fail.

Possible candidate · 1/2 runs4 papers report this75% from 2025+

Algorithmic Fairness

UnclassifiedOpen

Robustness and Sensitivity of Fairness Interventions Under Misspecified Merit Distributions and Latent Parameters

Many algorithmic fairness frameworks rely on theoretical assumptions that cannot be verified in deployment, specifically requiring known ground-truth merit distributions, unobservable transfer constants, or rigid pre-specified group partitions. In real-world applications, practitioners have access only to observed proxy labels and imperfect covariates, leaving the true merit distributions and transfer constants unknown and unbounded. Consequently, when fairness interventions are calibrated against estimated or misspecified latent parameters, theoretical parity guarantees can break down unpredictably, risking both utility loss and unintended disparity.

Possible candidate · 1/2 runs3 papers report this33% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.