Open Problems

Knowledge & Dataset Distillation

Distillation Under Zero-Data and Unlabeled-Stream Preconditions

Barrier to removeOpen
Possible candidate · 3/5 runs4 papers report this25% from 2025+

Generated automatically from the limitations stated in 4 papers (ICLR, ECCV, ICCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current distillation frameworks depend on strict data preconditions, including access to original downstream training sets, in-domain unlabeled datasets, out-of-distribution reference data, or sample-level identity annotations. When proprietary, privacy, or security constraints prevent the release of both the underlying training set and external proxy data, these methods cannot execute at all. Furthermore, when teacher outputs arrive as an uncurated stream without identity labels, existing feature-bank and prototype-matching methods break down. Resolving these preconditions allows model compression and dataset synthesis in strictly zero-data or privacy-restricted deployments.

Why it matters

Knowledge transfer and student model compression become possible in privacy-critical domains where raw datasets, domain-matched surrogate data, and sample identity labels cannot be shared or stored.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Inverted-generator teacher distillation: Train a generator to synthesize inputs directly from the teacher's activation statistics and logits without reference data, measuring downstream classification accuracy and synthetic sample fidelity across standard vision benchmarks.

  2. 2

    Label-free stream distillation: Formulate teacher-to-student feature transfer using contrastive clustering or optimal transport over streaming mini-batches rather than indexed identity banks, measuring performance degradation relative to full-supervision baselines.

  3. 3

    Sensitivity analysis across missing data assumptions: Benchmark existing data-free, surrogate-data, and stream-based distillation techniques across varying degrees of proxy data mismatch to quantify the performance cliff when data preconditions are violated.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Generative data-free inversion methods may suffer severe mode collapse on large-scale label spaces, failing to recover sufficient feature diversity to match even simple baseline models trained on small public proxies.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Knowledge & Dataset Distillation

Barrier to removeOpen

Cross-Modal 3D Distillation Without Paired and Synchronized LiDAR Streams

Existing cross-modal distillation frameworks for 3D perception strictly require synchronized, spatially calibrated LiDAR point clouds and pre-trained LiDAR teachers alongside camera or radar feeds during training. This strict precondition limits distillation to expensive research vehicles with high-end sensor rigs, preventing its application to the vast majority of production fleet datasets where LiDAR is absent, uncalibrated, or asynchronously logged. As a result, low-cost sensor suites cannot leverage pre-trained LiDAR representations unless rigid multimodal collection setups are maintained during training.

Possible candidate · 2/5 runs6 papers report this0% from 2025+

Knowledge & Dataset Distillation

Barrier to removeOpen

Knowledge Distillation Under Strict Black-Box Teacher Preconditions

Current high-performance distillation techniques rely on white-box access to intermediate representations, full logit vectors, or direct control over the teacher's training dynamics. When the teacher is a proprietary API or closed-source system that only returns discrete text or top-1 predictions, these methods cannot run. Downstream practitioners are structurally blocked from transferring capabilities from frontier closed models into compact architectures using state-of-the-art distillation losses.

Strong candidate · 5/5 runs4 papers report this75% from 2025+

Knowledge & Dataset Distillation

Barrier to removePartly addressed

Knowledge Distillation and Rollout Transfer from Black-Box Teachers Without Logit Access

High-performing knowledge distillation and policy transfer techniques rely on white-box teacher access to evaluate conditional entropy, top-$k$ predictive distributions, and log-probabilities during rollouts. This requirement structurally prevents distillation from state-of-the-art proprietary or API-only models that output only discrete decisions or text tokens without internal logits. Furthermore, existing workarounds often rely on fragile parametric assumptions, such as Gaussianity of teacher logits under data augmentation, which fail on complex, multi-modal output distributions.

Possible candidate · 2/5 runs3 papers report this67% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.