Model Safety, Alignment & Jailbreaks
Decision-Only and Logit-Free Safety Auditing and Alignment Evaluation for Black-Box Models
Generated automatically from the limitations stated in 17 papers (NeurIPS, ICLR, ICML), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Current safety auditing, jailbreak discovery, and representation-based alignment diagnostics overwhelmingly rely on white-box preconditions such as gradient access, intermediate activations, or output token logit distributions. When models are deployed behind production APIs that return only generated text, practitioners cannot run these diagnostic and red-teaming pipelines directly. Relying on surrogate transfer from open-source models produces high false-negative rates due to incomplete transferability across distinct architectures and tokenizers. Consequently, safety assessments for proprietary, API-governed models remain structurally disconnected from the methods developed in the literature.
Why it matters
Enables rigorous, standardized safety evaluations and red-teaming audits on closed-source, API-only models without requiring surrogate model assumptions or internal weight access.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Decision-based discrete optimization: Develop black-box search algorithms (e.g., adaptive token-mutation guided by text-level feedback metrics or semantic distance proxies) that evaluate model vulnerabilities using only generated text outputs. Measure attack success rate and query efficiency across closed and open LLMs compared to white-box baselines.
- 2
Output-only surrogate calibration: Train query-efficient local emulator models solely from target text responses to approximate local decision boundaries, measuring whether generated safety-auditing prompts achieve parity with direct white-box gradient attacks.
- 3
Multi-turn semantic feedback loops: Design automated red-teaming evaluators that iteratively probe model responses using conversational reframing without requiring probability distributions, measuring safety boundary violation rates across commercial black-box APIs.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
Purely black-box, discrete token space optimization may prove too query-intensive or get readily blocked by standard API rate limits and basic output filtering defenses. If black-box transferability from frontier open-weight models becomes universally sufficient, the need for direct black-box optimization would diminish.
Evidence
Each paper's own statement of the limitation, verbatim.
- TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language ModelsICLR 2026
Requires white-box access to model gradients and logits to perform the DPTO strategy.
- FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEctionCVPR 2026
Requires white-box gradient and internal feature representation access on the surrogate source model during perturbation generation.
- Weak-to-Strong Jailbreaking on Large Language ModelsICML 2025
Requires white-box or logit-level access to the target model's output distribution.
- The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test AwarenessNeurIPS 2025
Requires white-box access to model activations and internal weight matrices for parameter editing.
- PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context OptimizationACL 2025
Requires white-box gradient access to the target model for the optimization step; black-box results rely on transfer, which is incomplete
- LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMsNeurIPS 2025
Requires full white-box access to the model's gradients and internal hidden states to perform optimization.
- Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-TuningICML 2025
Requires white-box access to model activations and weight gradients for fine-tuning.
- Improved Techniques for Optimization-Based Jailbreaking on Large Language ModelsICLR 2025
Requires full white-box access to target model gradients and logits for the primary optimization phase.
Show all 17 papers
- BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled OptimizationNeurIPS 2025
Requires white-box access to model parameters and the optimization process during training.
- AdvPrefix: An Objective for Nuanced LLM JailbreaksNeurIPS 2025
Only applicable in white-box settings where target model log-probabilities are accessible.
- Robust Prompt Optimization for Defending Language Models Against Jailbreaking AttacksNeurIPS 2024
Requires white-box gradient access to at least one representative model during the suffix optimization phase.
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language ModelsNeurIPS 2024
Requires white-box access to model weights to compute the landscape and VISAGE score.
- Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their DefensesNeurIPS 2024
Requires access to target model output logits to perform the demo-level random search optimization.
- BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language ModelsEMNLP 2024
Requires full white-box access to model parameters and intermediate layer activations.
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language ModelsICLR 2024
Requires white-box access to the target model's log-likelihoods to calculate the fitness function during optimization.
- AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge BasesNeurIPS 2024
Requires white-box access or a high-fidelity proxy of the query encoder for gradient-guided optimization.
- ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix EmbeddingsEMNLP 2024
Requires white-box access to the target model's embedding space for the initial optimization stage (transfer to black-box models is indirect)
Nearest existing work
- Enhancing the Transferability of Jailbreak Attacks on Large Language Models via Exploiting Reparameterization InvarianceACL 2026
- JULI: Jailbreak Large Language Models by Self-IntrospectionICLR 2026
- Weak-to-Strong Jailbreaking on Large Language ModelsICML 2025
- Guiding not Forcing: Enhancing the Transferability of Jailbreaking Attacks on LLMs via Removing Superfluous ConstraintsACL 2025
- Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution StrategyCVPR 2025
- Alignment-Enhanced Decoding: Defending Jailbreaks via Token-Level Adaptive Refining of Probability DistributionsEMNLP 2024
- SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware DecodingACL 2024
- SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language ModelsEMNLP 2025
- Calibrating Inference Time Alignment with Sequence-level Risk AccumulationACL 2026
- SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual ConnectionEMNLP 2025
- Speculative Safety-Aware DecodingEMNLP 2025
- Safety Alignment Should be Made More Than Just a Few Tokens DeepICLR 2025
- The Jailbreak Tax: How Useful are Your Jailbreak Outputs?ICML 2025
- Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained OptimizationNeurIPS 2024
- Towards Robust Multimodal Large Language Models Against Jailbreak AttacksCVPR 2026
Related open problems
Model Safety, Alignment & Jailbreaks
A Framework for Deploying Activation- and Logit-Based Methods Against Closed-Source LLM APIs via Instrumented Proxy Models
A very large body of techniques — safety defenses, interpretability probes, evaluation metrics, decoding controls — requires hidden states, attention maps, gradients, or token-level probabilities, and is therefore silently restricted to open-weight models. As a result, published methods are never validated on, and cannot protect or audit, the models most people actually use (GPT-4, Claude, Gemini). This is the single most common self-reported scope limitation in the literature, yet each paper treats it as an isolated footnote rather than a solvable engineering and inference problem.
Model Safety, Alignment & Jailbreaks
Recovering Internal Attention and Activation Signals from API-Visible Model Behavior
The shared wall is a dependence on inference-time extraction of layer-wise internal states — attention weights, hidden activations, per-layer execution — which are unavailable outside open-weight deployments and costly and per-model-fragile even where weights are open. Every technique built on those signals stops working the moment the model is served through an API; the sanctioned fallbacks, transferring from an open proxy or degrading to logit-only features, demonstrably forfeit most of the signal (one published detector drops from its white-box ceiling to 0.66 AUROC). Even with open weights, each release re-breaks the tooling: layer indices must be re-selected per model and multi-pass decoding adds latency. The result is a method family whose reach shrinks exactly as the most capable models become less open, and whose findings nobody without weight access can verify or use.
Model Safety, Alignment & Jailbreaks
Robust Evaluation of Non-Binary and Phrasing-Agnostic Refusal in Safety-Aligned Language Models
Current safety benchmarks and verifiable reward mechanisms evaluate model refusals primarily through binary keyword matching, hand-crafted refusal templates, or uncalibrated model-based autoraters (which show inter-annotator agreement as low as $\alpha = 0.378$). This creates a blind spot where models that partially comply, use non-standard refusal vocabulary, or alter their stance under multi-turn pushback are misclassified as either fully compliant or safely refusing. Consequently, alignment interventions optimized against these brittle metrics suffer from artificial over-refusal on benign queries while leaking safety risks through stylistic variation and conversational pressure. Without a rigorous, non-binary evaluation standard across diverse phrasing distributions, researchers cannot reliably assess whether safety alignment generalizes beyond rigid heuristic templates.
Model Safety, Alignment & Jailbreaks
Safety Monitoring and Alignment Interventions Under Black-Box and API-Only Constraints
Current state-of-the-art alignment monitoring, jailbreak detection, and activation-steering interventions explicitly require full white-box access to hidden state activations, KV caches, attention logits, or gradient flows. In practice, third-party safety auditors, downstream system builders, and end users interact with models exclusively via black-box query APIs or restricted endpoints. Because existing methods are structurally coupled to internal state inspection and modification, they cannot be deployed or evaluated on proprietary commercial models or in privacy-preserving environments.