Audio, Speech & Music Generation

Black-Box Optimization and Guidance for Audio Generation Without Target ASR White-Box Access

Barrier to removeOpen

Strong candidate · 5/5 runs5 papers report this20% from 2025+

Generated automatically from the limitations stated in 5 papers (ACML, ICML, ECCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current audio and speech generation frameworks that optimize outputs against target ASR systems require white-box access to model internals to compute gradients or rely on explicit phoneme alignment pipelines. This precondition blocks the deployment, alignment, and evaluation of generative models against commercial, proprietary, or closed-source ASR APIs. Additionally, the requirement for differentiable phoneme alignments restricts applicability to languages that lack dedicated phonetic toolkits. As a result, generative audio models cannot be systematically guided or audited against real-world speech recognizers whose parameters and architectures are hidden.

Why it matters

Enables generative speech and audio models to be directly guided, adapted, or audited against closed-source, commercial ASR engines and low-resource languages without needing internal parameter access, gradients, or phoneme aligners.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

1
Zeroth-order gradient estimation for guided diffusion: Implement bandit or finite-difference gradient approximations (e.g., simultaneous perturbation stochastic approximation) to guide diffusion trajectories using only top-1 transcriptions or output token probabilities from black-box ASR models, measuring WER, Word Error Rate reduction, and audio quality (MOS/FD).
2
Surrogate proxy distillation under limited query budgets: Query target black-box ASR endpoints to distill an accessible, differentiable proxy ASR model that provides gradient guidance during audio generation, measuring transferability and guidance success rates as a function of query budget.
3
Policy gradient / reinforcement learning on discrete transcript feedback: Formulate speech latent or prompt optimization as a reinforcement learning problem using downstream transcription metrics (e.g., character error rate or semantic similarity) as reward signals without requiring phoneme inventories, measuring adaptation efficiency and acoustic realism.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Why it might fail

High sample complexity in estimating zeroth-order gradients across high-dimensional audio latents could make black-box guidance computationally intractable or prohibitively expensive over API rate limits. Furthermore, discrete tokenization and non-continuous text outputs from black-box ASR systems may create non-informative reward landscapes that hinder optimization.

Evidence

Each paper's own statement of the limitation, verbatim.

AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion ForcingICML 2026
Relies on synthetic training data generated from predefined TTS and T2A tool sets, which may limit generalization to complex acoustic phenomena outside the tool capabilities.
KMTalk: Speech-Driven 3D Facial Animation with Key Motion EmbeddingECCV 2024
Requires a phoneme inventory/alignment pipeline, making it language- and ASR-quality dependent unlike purely data-driven methods
Diffusion-based Adversarial Attack to Automatic Speech RecognitionACML 2024
Requires white-box access to the target ASR model to compute gradients via Cross-Entropy loss.
Real-Time Neural Voice CamouflageICLR 2022
Requires white-box or surrogate access to the target ASR model parameters for initial training.
Beyond $L_p$ Clipping: Equalization based Psychoacoustic Attacks against ASRsACML 2021
Requires white-box knowledge of the ASR model internals for gradient-based optimization.