Open Problems

Knowledge & Dataset Distillation

Knowledge Distillation and Rollout Transfer from Black-Box Teachers Without Logit Access

Barrier to removePartly addressed
Possible candidate · 2/5 runs3 papers report this67% from 2025+

Generated automatically from the limitations stated in 3 papers (ICML, ACL), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

High-performing knowledge distillation and policy transfer techniques rely on white-box teacher access to evaluate conditional entropy, top-$k$ predictive distributions, and log-probabilities during rollouts. This requirement structurally prevents distillation from state-of-the-art proprietary or API-only models that output only discrete decisions or text tokens without internal logits. Furthermore, existing workarounds often rely on fragile parametric assumptions, such as Gaussianity of teacher logits under data augmentation, which fail on complex, multi-modal output distributions.

Why it matters

Enables distilling state-of-the-art capabilities and uncertainty calibration from closed-source commercial APIs and black-box models into compact student architectures without internal weight or logit access.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Non-parametric uncertainty estimation via perturbation queries: Estimate teacher output entropy and top-$k$ rank statistics from empirical output frequencies across stochastic input perturbations, measuring distillation fidelity against white-box logit baselines on standard classification and text benchmarks.

  2. 2

    Logit-free importance weighting for rollouts: Design sample-weighting heuristics based on student disagreement and rejection sampling over teacher-generated sequences rather than explicit teacher logit entropy, measuring student downstream accuracy and rollout memory overhead.

  3. 3

    Distribution-free decision boundary alignment: Formulate an optimization objective that aligns student predictive distributions to teacher hard decisions using ranking loss constraints, measuring performance on black-box vision and language distillation tasks.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

The query complexity required to reliably estimate teacher uncertainty from discrete outputs could become cost-prohibitive compared to standard supervised fine-tuning on raw teacher outputs.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Knowledge & Dataset Distillation

Barrier to removeOpen

Cross-Modal 3D Distillation Without Paired and Synchronized LiDAR Streams

Existing cross-modal distillation frameworks for 3D perception strictly require synchronized, spatially calibrated LiDAR point clouds and pre-trained LiDAR teachers alongside camera or radar feeds during training. This strict precondition limits distillation to expensive research vehicles with high-end sensor rigs, preventing its application to the vast majority of production fleet datasets where LiDAR is absent, uncalibrated, or asynchronously logged. As a result, low-cost sensor suites cannot leverage pre-trained LiDAR representations unless rigid multimodal collection setups are maintained during training.

Possible candidate · 2/5 runs6 papers report this0% from 2025+

Knowledge & Dataset Distillation

Barrier to removeOpen

Knowledge Distillation Under Strict Black-Box Teacher Preconditions

Current high-performance distillation techniques rely on white-box access to intermediate representations, full logit vectors, or direct control over the teacher's training dynamics. When the teacher is a proprietary API or closed-source system that only returns discrete text or top-1 predictions, these methods cannot run. Downstream practitioners are structurally blocked from transferring capabilities from frontier closed models into compact architectures using state-of-the-art distillation losses.

Strong candidate · 5/5 runs4 papers report this75% from 2025+

Knowledge & Dataset Distillation

Barrier to removeOpen

Distillation Under Zero-Data and Unlabeled-Stream Preconditions

Current distillation frameworks depend on strict data preconditions, including access to original downstream training sets, in-domain unlabeled datasets, out-of-distribution reference data, or sample-level identity annotations. When proprietary, privacy, or security constraints prevent the release of both the underlying training set and external proxy data, these methods cannot execute at all. Furthermore, when teacher outputs arrive as an uncurated stream without identity labels, existing feature-bank and prototype-matching methods break down. Resolving these preconditions allows model compression and dataset synthesis in strictly zero-data or privacy-restricted deployments.

Possible candidate · 3/5 runs4 papers report this25% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.