Open Problems

Video-Language Understanding & Temporal Grounding

Benchmarking and Mitigating Spatio-Temporal Grounding Degradation in Continuous Video-LLMs

Effect to explainOpen
Possible candidate · 3/5 runs4 papers report this50% from 2025+

Generated automatically from the limitations stated in 4 papers (ICLR, ICML, ECCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current video-language models evaluate spatial grounding primarily on sparse frames (e.g., 0.5 FPS), leaving performance in continuous video unmeasured. When models attempt spatio-temporal grounding, performance degrades significantly under multi-point trajectory supervision (e.g., mIoU drops beyond 3 tracking points) and dynamic multi-resolution token sampling raises spatio-temporal error relative to plain spatial pooling. Because full spatio-temporal box annotation is prohibitively expensive, visual grounding in video remains unresolved with sub-10% IoU baselines.

Why it matters

Establishes reliable spatio-temporal localization capabilities in continuous video streams without requiring intractable manual frame-by-frame bounding box annotations.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Evaluate open-source Video-LLMs across varying evaluation frame rates (from sparse 0.5 FPS to dense continuous video) on existing spatio-temporal datasets (e.g., VidSTG, HC-STVG), measuring bounding box mIoU and trajectory stability across temporal sampling densities.

  2. 2

    Controlled comparison of visual token reduction strategies (uniform spatial pooling vs. dynamic multi-resolution sampling) across dense frame sequences, measuring inter-frame representation consistency and spatio-temporal error.

  3. 3

    Assess the effect of varying trajectory supervision point density ($P \in \{1, 3, 5, 10\}$) and trajectory filtering heuristics on temporal grounding accuracy under noisy pseudo-box annotations.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Dense spatio-temporal foundation models or general video diffusion representations might inherently resolve inter-frame spatial tracking, rendering empirical workarounds for autoregressive Video-LLM tokenizers obsolete.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.