Open Problems

Knowledge & Dataset Distillation

Knowledge Distillation Under Strict Black-Box Teacher Preconditions

Barrier to removeOpen
Strong candidate · 5/5 runs4 papers report this75% from 2025+

Generated automatically from the limitations stated in 4 papers (ICLR, NeurIPS, EMNLP), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current high-performance distillation techniques rely on white-box access to intermediate representations, full logit vectors, or direct control over the teacher's training dynamics. When the teacher is a proprietary API or closed-source system that only returns discrete text or top-1 predictions, these methods cannot run. Downstream practitioners are structurally blocked from transferring capabilities from frontier closed models into compact architectures using state-of-the-art distillation losses.

Why it matters

Distillation from closed-source, proprietary API-hosted models into local, highly compressed student models without requiring access to weights, hidden states, or full output distributions.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Benchmark existing white-box, logit-based, and output-only distillation techniques across standard vision and language tasks under progressively restricted teacher access (full states $\rightarrow$ logits $\rightarrow$ top-k logits $\rightarrow$ hard labels/text), measuring downstream student task accuracy.

  2. 2

    Develop a surrogate-guidance framework where a local, lightweight proxy model estimates pseudo-logits and synthetic intermediate targets from black-box teacher sample outputs, measuring distillation fidelity against true white-box baselines.

  3. 3

    Formulate an active query-selection strategy that optimizes which inputs to submit to the black-box teacher API to maximize student learning efficiency under fixed query budgets, measured by student performance per API call.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If prompting or simple supervised fine-tuning on raw black-box teacher outputs already saturates student capacity limits, rendering specialized black-box distillation formulations redundant.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Knowledge & Dataset Distillation

Barrier to removeOpen

Cross-Modal 3D Distillation Without Paired and Synchronized LiDAR Streams

Existing cross-modal distillation frameworks for 3D perception strictly require synchronized, spatially calibrated LiDAR point clouds and pre-trained LiDAR teachers alongside camera or radar feeds during training. This strict precondition limits distillation to expensive research vehicles with high-end sensor rigs, preventing its application to the vast majority of production fleet datasets where LiDAR is absent, uncalibrated, or asynchronously logged. As a result, low-cost sensor suites cannot leverage pre-trained LiDAR representations unless rigid multimodal collection setups are maintained during training.

Possible candidate · 2/5 runs6 papers report this0% from 2025+

Knowledge & Dataset Distillation

Barrier to removeOpen

Distillation Under Zero-Data and Unlabeled-Stream Preconditions

Current distillation frameworks depend on strict data preconditions, including access to original downstream training sets, in-domain unlabeled datasets, out-of-distribution reference data, or sample-level identity annotations. When proprietary, privacy, or security constraints prevent the release of both the underlying training set and external proxy data, these methods cannot execute at all. Furthermore, when teacher outputs arrive as an uncurated stream without identity labels, existing feature-bank and prototype-matching methods break down. Resolving these preconditions allows model compression and dataset synthesis in strictly zero-data or privacy-restricted deployments.

Possible candidate · 3/5 runs4 papers report this25% from 2025+

Knowledge & Dataset Distillation

Barrier to removePartly addressed

Knowledge Distillation and Rollout Transfer from Black-Box Teachers Without Logit Access

High-performing knowledge distillation and policy transfer techniques rely on white-box teacher access to evaluate conditional entropy, top-$k$ predictive distributions, and log-probabilities during rollouts. This requirement structurally prevents distillation from state-of-the-art proprietary or API-only models that output only discrete decisions or text tokens without internal logits. Furthermore, existing workarounds often rely on fragile parametric assumptions, such as Gaussianity of teacher logits under data augmentation, which fail on complex, multi-modal output distributions.

Possible candidate · 2/5 runs3 papers report this67% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.