Model Safety, Alignment & Jailbreaks
Robust Evaluation of Non-Binary and Phrasing-Agnostic Refusal in Safety-Aligned Language Models
Generated automatically from the limitations stated in 14 papers (NeurIPS, ICML, ACL), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Current safety benchmarks and verifiable reward mechanisms evaluate model refusals primarily through binary keyword matching, hand-crafted refusal templates, or uncalibrated model-based autoraters (which show inter-annotator agreement as low as $\alpha = 0.378$). This creates a blind spot where models that partially comply, use non-standard refusal vocabulary, or alter their stance under multi-turn pushback are misclassified as either fully compliant or safely refusing. Consequently, alignment interventions optimized against these brittle metrics suffer from artificial over-refusal on benign queries while leaking safety risks through stylistic variation and conversational pressure. Without a rigorous, non-binary evaluation standard across diverse phrasing distributions, researchers cannot reliably assess whether safety alignment generalizes beyond rigid heuristic templates.
Why it matters
Enables trustworthy evaluation of safety mechanisms that is robust to stylistic shifts, partial refusals, and multi-turn conversational pressure. It also allows alignment algorithms to optimize for nuanced boundary compliance rather than superficial keyword matching.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Benchmark existing keyword-, regex-, and LLM-based refusal detectors against a curated, multi-rater human dataset covering partial compliance, non-standard phrasing, and benign over-refusal queries across at least five distinct model families; measure precision, recall, and Krippendorff’s $\alpha$ across detection methods.
- 2
Construct a multi-turn pushback evaluation protocol that measures the stability of refusal decisions under sequential user pressure, quantifying the decay rate of refusal behavior compared to single-turn baseline scores.
- 3
Train and evaluate graded continuous refusal scorers against binary reward indicators in reinforcement learning from AI/human feedback; measure whether continuous scoring reduces over-refusal on benign prompts while preserving resistance to jailbreaks.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
Standard frontier LLMs acting as autoraters may improve sufficiently on zero-shot prompt-following to render specialized refusal measurement trivial without requiring new evaluation frameworks. Additionally, the boundary between partial compliance and benign refusal may remain fundamentally ambiguous even to human annotators, limiting measurement ceiling gains.
Evidence
Each paper's own statement of the limitation, verbatim.
- SafeSci: Safety Evaluation of Large Language Models in Science Domains and BeyondICML 2026
Risk refusal is reduced to a binary refusal/safety-rate determination, which cannot capture partial refusals or context nuance, and the knowledge/risk boundary rests on an untested hypothesis that anything publicly retrievable should be answered
- Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt InjectionACL 2026
Refusal rates drop substantially with general shadow questions (avg 0.84-0.86) and on some models (InstructBLIP ~0.78), especially for TextVQA
- In-Training Defenses Against Emergent Misalignment in Language ModelsICML 2026
Refusal filtering uses simple keyword matching (e.g., 'sorry', 'cannot'), which is suboptimal; the perplexity-gap selection also requires training K proxy misaligned models and only covers EM regimes similar to those proxies
- Evaluating Contextual Illegality: AI Compliance in Corporate Law ScenariosICML 2026
Single-turn evaluation without interactive user pushback prevents assessment of how models handle follow-up conversational pressure on refusals.
- AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement LearningICLR 2026
Reliance on a fixed set of predefined refusal patterns for the verifiable reward function may miss non-standard refusal styles.
- Safety Pretraining: Toward the Next Generation of Safe AINeurIPS 2025
Increased over-refusal rates where the model may decline benign queries
- Root Defense Strategies: Ensuring Safety of LLM at the Decoding LevelACL 2025
If all top-k candidates for a benign query are refusal tokens, RDS still rejects — over-correction on models with strong refusal bias remains unsolved
- OR-Bench: An Over-Refusal Benchmark for Large Language ModelsICML 2025
Refusal is treated as a false binary—models may partially comply without fully refusing or fully answering, which the benchmark's keyword/GPT-4 refusal detection does not capture
Show all 14 papers
- LLM Safety Alignment is Divergence Estimation in DisguiseNeurIPS 2025
Effectiveness is dependent on the quality and representativeness of the compliance-refusal labeling in the training data
- EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious InstructionsNeurIPS 2025
Refusal probability is optimized against a single target model (LLAMA3.1-8B-INSTRUCT), and the largest benchmark gain is on that model (364.29%), suggesting some target-model bias despite claimed generalization
- Who's asking? User personas and the mechanics of latent misalignmentNeurIPS 2024
Autorater reliability is modest: Krippendorff's alpha with human raters is 0.378 (human-human 0.415), so refusal labels are noisy
- Jailbreak Open-Sourced Large Language Models via Enforced DecodingACL 2024
Relies on a hand-crafted list of fewer than 10 negative-word/antonym pairs and a similarity threshold (η=0.8); robustness to models with different refusal vocabularies or refusal patterns outside this list is not established
- Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss LandscapesNeurIPS 2024
Refusal detection relies on a keyword-based indicator function, which the authors themselves note cannot exhaustively cover all refusal phrasings
- On the Exploitability of Instruction TuningNeurIPS 2023
The over-refusal effectiveness metric is model-based (GPT-3.5-turbo judging informative refusals) and only manually spot-checked, not validated by a broad human study
Nearest existing work
- Improving LLM Safety Alignment with Dual-Objective OptimizationICML 2025
- Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal TrainingACL 2025
- Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-TuningICML 2025
- Refusal in Language Models Is Mediated by a Single DirectionNeurIPS 2024
- Refusal Direction is Universal Across Safety-Aligned LanguagesNeurIPS 2025
- SORRY-Bench: Systematically Evaluating Large Language Model Safety RefusalICLR 2025
- Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive RefinementICLR 2026
- State-Dependent Safety Failures in Multi-Turn Language Model InteractionICML 2026
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy OptimizationNeurIPS 2025
- OR-Bench: An Over-Refusal Benchmark for Large Language ModelsICML 2025
- LLM-VA: Resolving the Jailbreak-Overrefusal Trade-off via Vector AlignmentACL 2026
- Furina: Fragmented Uncertainty-Driven Refusal Instability AttackICML 2026
- AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement LearningICLR 2026
- Please refuse to answer me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive DecodingACL 2026
- Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector AblationICLR 2025
Related open problems
Model Safety, Alignment & Jailbreaks
A Framework for Deploying Activation- and Logit-Based Methods Against Closed-Source LLM APIs via Instrumented Proxy Models
A very large body of techniques — safety defenses, interpretability probes, evaluation metrics, decoding controls — requires hidden states, attention maps, gradients, or token-level probabilities, and is therefore silently restricted to open-weight models. As a result, published methods are never validated on, and cannot protect or audit, the models most people actually use (GPT-4, Claude, Gemini). This is the single most common self-reported scope limitation in the literature, yet each paper treats it as an isolated footnote rather than a solvable engineering and inference problem.
Model Safety, Alignment & Jailbreaks
Decision-Only and Logit-Free Safety Auditing and Alignment Evaluation for Black-Box Models
Current safety auditing, jailbreak discovery, and representation-based alignment diagnostics overwhelmingly rely on white-box preconditions such as gradient access, intermediate activations, or output token logit distributions. When models are deployed behind production APIs that return only generated text, practitioners cannot run these diagnostic and red-teaming pipelines directly. Relying on surrogate transfer from open-source models produces high false-negative rates due to incomplete transferability across distinct architectures and tokenizers. Consequently, safety assessments for proprietary, API-governed models remain structurally disconnected from the methods developed in the literature.
Model Safety, Alignment & Jailbreaks
Recovering Internal Attention and Activation Signals from API-Visible Model Behavior
The shared wall is a dependence on inference-time extraction of layer-wise internal states — attention weights, hidden activations, per-layer execution — which are unavailable outside open-weight deployments and costly and per-model-fragile even where weights are open. Every technique built on those signals stops working the moment the model is served through an API; the sanctioned fallbacks, transferring from an open proxy or degrading to logit-only features, demonstrably forfeit most of the signal (one published detector drops from its white-box ceiling to 0.66 AUROC). Even with open weights, each release re-breaks the tooling: layer indices must be re-selected per model and multi-pass decoding adds latency. The result is a method family whose reach shrinks exactly as the most capable models become less open, and whose findings nobody without weight access can verify or use.
Model Safety, Alignment & Jailbreaks
Safety Monitoring and Alignment Interventions Under Black-Box and API-Only Constraints
Current state-of-the-art alignment monitoring, jailbreak detection, and activation-steering interventions explicitly require full white-box access to hidden state activations, KV caches, attention logits, or gradient flows. In practice, third-party safety auditors, downstream system builders, and end users interact with models exclusively via black-box query APIs or restricted endpoints. Because existing methods are structurally coupled to internal state inspection and modification, they cannot be deployed or evaluated on proprietary commercial models or in privacy-preserving environments.