Open Problems

Time-Series Forecasting

A Standard Benchmark and Protocol for Genuine Zero-Shot Transfer to Truly Unseen Domains in Structured Perception Tasks

Barrier to removeOpen
Possible candidate · 2/5 runs19 papers report this57% from 2025+

Generated automatically from the limitations stated in 19 papers (CVPR, NeurIPS, ECCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current zero-shot claims in structured prediction are not actually zero-shot: evaluations use datasets sharing variable structure, sensor type, or category overlap with training data; some protocols leak a fraction of test samples or target scenes to select prompts or instructions; and others substitute qualitative visualizations for scores because no clean mapping exists. The result is that reported "zero-shot" numbers cannot be compared across papers, and the field cannot measure how far true transfer to genuinely unseen conditions actually is. Until evaluation is tightened, progress on cross-domain generalization is unmeasurable and likely overstated.

Why it matters

Trustworthy zero-shot numbers would let the field distinguish real cross-domain generalization from prior bias and leakage, and would give a reproducible target for methods aimed at closing the zero-shot-to-few-shot gap.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Audit and formalize: build a leakage taxonomy over existing zero-shot evaluation protocols (dataset-family overlap, prompt/instruction selection from test data, train-set contamination with target categories or scenes), and re-score published setups under a strict definition — measuring the gap between reported "zero-shot" accuracy and strict zero-shot accuracy on the same tasks.

  2. 2

    Construct a held-out transfer suite: for time-series, point clouds, and segmentation respectively, designate target benchmarks whose variable structure, sensor domain, and categories are disjoint from any plausible training corpus, with fixed category mappings and quantitative metrics (accuracy/AP/mIoU per class, with rare classes broken out). Measure where zero-shot methods actually land versus few-shot baselines given equal total data.

  3. 3

    Characterize failure structure: measure per-class and per-condition error as a function of distance between target domain and nearest training distribution (e.g., domain-similarity metrics), testing whether zero-shot degradation is predictable — this determines whether the wall is fundamental (prior-dominated predictions) or an artifact of evaluation.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Published authors may dispute the leakage taxonomy or argue their relaxed settings are the practically relevant ones, leaving the community without consensus on the strict protocol. Alternatively, the audit may simply confirm a well-known gap without yielding insight beyond "zero-shot is hard."

Sub-problems

  • Bridging the Zero-Shot Transfer Gap in Real-World 3D Point Cloud Perception

    3D point cloud models currently fail to generalize reliably to unseen real-world scenes, exhibiting severe performance drops on benchmarks like ScanObjectNN, ScanNetV2, and SemanticKITTI compared to supervised baselines. Current zero-shot 3D methods rely heavily on 2D vision-language priors (like CLIP) via multi-view projection or learned embeddings, but fail under sensor noise, occlusion, density variations, and tail categories. Consequently, 3D perception pipelines cannot be deployed open-vocabulary in novel indoor or outdoor environments without costly in-domain annotation.

  • A Systematic Robustness and Protocol Benchmark for Zero-Shot Segmentation Across Domains and Granularities

    Current zero-shot segmentation claims remain fragmented across narrow test settings (e.g., single tasks like RGB-D COD or standard natural-image benchmarks) without a unified assessment across granularity scales and domain shifts. Inconsistent protocols—such as unseen-pixel leakage in training images, pseudo-label contamination, and unmapped evaluation categories—prevent direct comparisons between strict zero-shot and open-vocabulary methods. Consequently, the research community cannot determine whether reported generalization failures (such as steep performance drops on subparts or medical images) stem from model architectural limits or disparate evaluation setups.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 19 papers

Nearest existing work

Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.