Open Problems

Text-to-Image & Video Generation/Editing

Benchmarking and Mitigating Upstream LLM Prompting Dependencies in Text-to-Visual Generation

Barrier to removePartly addressed
Possible candidate · 2/5 runs12 papers report this58% from 2025+

Generated automatically from the limitations stated in 12 papers (ICCV, NeurIPS, ECCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Modern text-to-image and text-to-video generation and editing methods rely heavily on external large language models (LLMs) for prompt rewriting, sub-prompt decomposition, and attribute filtering. Consequently, visual generation quality is bottlenecked by upstream LLM biases, template artifacts, and hallucinated attributes, frequently failing on ambiguous or in-the-wild user instructions unless manually tuned. Because no systematic study has evaluated visual generation frameworks across diverse upstream prompt generators and decomposition strategies, the sensitivity and failure modes of these pipelines remain unquantified.

Why it matters

Makes text-to-visual generation and editing pipelines robust to arbitrary, ambiguous user prompts without requiring manual prompt engineering or proprietary LLM filtering.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Multi-LLM Robustness Benchmark: Evaluate existing T2I/T2V prompt-dependent generation and editing pipelines across a standardized suite of upstream LLMs (varying in scale, training priors, and open/closed weights) on in-the-wild prompts, measuring visual fidelity, text-image alignment, and artifact frequencies.

  2. 2

    Visual-Feedback Prompt Decomposition: Develop a closed-loop prompt refinement mechanism where downstream visual cross-attention maps and alignment scores provide direct feedback to the upstream prompt decomposition step, reducing reliance on manual template engineering.

  3. 3

    Vision-Grounded Prompt Calibration: Train a lightweight, vision-aligned prompt adapter directly tuned on downstream generative failures rather than generic text-only LLM completions, measuring degradation rates on out-of-distribution user queries.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Rapid architectural transitions toward end-to-end native multimodal foundation models that directly process raw, uncurated user text without auxiliary LLM prompt rewriting stages could dissolve the problem.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 12 papers

Nearest existing work

Related open problems

Text-to-Image & Video Generation/Editing

Barrier to removePartly addressed

Systematically Benchmarking and Calibrating VLM-as-a-Judge Failure Modes in Image and Video Generation

Current text-to-image and video generation benchmarks overwhelmingly rely on Vision-Language Models (VLMs) like GPT-4o, Gemini, and Qwen-VL as automated judges for win rates and task success metrics. However, these VLM judges systematically exhibit low correlation with human raters on fine-grained visual details, suffer from position and same-family evaluator biases, and under-penalize visual artifacts such as boundary glitches or text corruption. Consequently, benchmark progress across generative models is structurally uninterpretable, as claimed state-of-the-art gains may reflect evaluator artifacts rather than genuine improvements in visual quality and prompt fidelity.

Possible candidate · 2/5 runs15 papers report this100% from 2025+

Text-to-Image & Video Generation/Editing

Barrier to removePartly addressed

Dense Motion Supervision Without Optical-Flow Ground Truth

Nearly every pipeline that needs dense correspondence—video generation control, camera pose estimation, motion attribution—currently anchors its supervision or its inference to optical flow, either ground-truth annotations or outputs of a pretrained model (RAFT, AllTracker). This couples training data to static-camera, high-quality, curated footage, caps dataset size (22k–630k pairs), injects flow errors directly into downstream quality (cycle-consistency errors >1000 px), and bakes in simplifying assumptions like first-frame conditioning or pinhole intrinsics. Every consumer of flow inherits the same ceiling set by flow estimators and the data they were trained on.

Strong candidate · 5/5 runs8 papers report this100% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.