Open Problems

Audio, Speech & Music Generation

Conditioning Audio and Speech Generation on Unstructured Holistic Text Descriptions

Barrier to removeOpen
Possible candidate · 3/5 runs3 papers report this100% from 2025+

Generated automatically from the limitations stated in 3 papers (CVPR, ICML), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current audio, speech, and music generation models cannot generate coherent multi-event audio directly from free-form, unformatted natural language descriptions. Instead, they depend on rigid preconditions: segment-by-segment timestamped scripts, multi-field prompt partitions, or explicit slot separations (e.g., distinguishing speech transcripts from ambient sounds). This forces users to manually author structured scripts or introduces a brittle dependency on upstream multimodal LLMs to parse and slot-fill prompts before generation can occur.

Why it matters

Direct generation of complex, layered audio scenes and speech from natural conversational descriptions without requiring intermediate prompt-structuring pipelines or manual timestamp scripting.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Benchmark existing structured audio generation models against degraded or unformatted holistic prompts across speech, sound effects, and music, measuring alignment drop and temporal degradation via CLAP scores and objective audio-text consistency metrics.

  2. 2

    Train a lightweight cross-attention text conditioning adapter that learns implicit slot decomposition directly from paired audio and raw caption datasets, measuring generative fidelity (FAD) and temporal alignment against explicit slot baselines.

  3. 3

    Formulate end-to-end audio generation using latent continuous text embeddings rather than discrete sub-token slot partitions, evaluating generation quality across variable-length free-form prompts.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Upstream MLLMs may become so fast, cheap, and seamlessly integrated into generation front-ends that externalizing prompt decomposition ceases to be a meaningful practical bottleneck.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Audio, Speech & Music Generation

Barrier to removeOpen

Black-Box Optimization and Guidance for Audio Generation Without Target ASR White-Box Access

Current audio and speech generation frameworks that optimize outputs against target ASR systems require white-box access to model internals to compute gradients or rely on explicit phoneme alignment pipelines. This precondition blocks the deployment, alignment, and evaluation of generative models against commercial, proprietary, or closed-source ASR APIs. Additionally, the requirement for differentiable phoneme alignments restricts applicability to languages that lack dedicated phonetic toolkits. As a result, generative audio models cannot be systematically guided or audited against real-world speech recognizers whose parameters and architectures are hidden.

Strong candidate · 5/5 runs5 papers report this20% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.