Model Safety, Alignment & Jailbreaks
Recovering Internal Attention and Activation Signals from API-Visible Model Behavior
Generated automatically from the limitations stated in 15 papers (ICML, ICLR, CVPR), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
The shared wall is a dependence on inference-time extraction of layer-wise internal states — attention weights, hidden activations, per-layer execution — which are unavailable outside open-weight deployments and costly and per-model-fragile even where weights are open. Every technique built on those signals stops working the moment the model is served through an API; the sanctioned fallbacks, transferring from an open proxy or degrading to logit-only features, demonstrably forfeit most of the signal (one published detector drops from its white-box ceiling to 0.66 AUROC). Even with open weights, each release re-breaks the tooling: layer indices must be re-selected per model and multi-pass decoding adds latency. The result is a method family whose reach shrinks exactly as the most capable models become less open, and whose findings nobody without weight access can verify or use.
Why it matters
Every analysis that currently assumes weight ownership becomes runnable against API-served models, so frontier behavior can be scored, attributed, and monitored by people who will never hold the weights; and new methods can be designed against a stable API-visible interface, ending the per-model re-tuning cycle.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Recoverability audit on open models (startable immediately). On two or three open model families, log per-example attention maps and hidden states alongside API-visible observables — output text, token logprobs, and next-token distributions under token deletion, paraphrase, and order-swap perturbations. Fit simple regressors/rankers mapping observables to token-salience vectors. Measure Spearman correlation of recovered vs. true layer-wise attention salience, and downstream AUROC using recovered vs. true signals. First curves for one model family within a month.
- 2
Query-budgeted behavioral substitutes. Implement occlusion, minimal-pair logprob-contrast, and self-consistency estimators of per-token influence under a fixed budget (~20 queries/example), no internals. Measure gap-closure against both the logit-only baseline and the white-box ceiling from (1) on the same open models, then run the identical estimators through closed APIs; track AUROC, correlation with ground-truth attention, and per-example cost in dollars and latency versus multi-pass internal extraction.
- 3
Zero-configuration port and sweep. Port the published internal-state methods onto the substitutes with no per-model layer selection, then run across ≥3 open families and ≥2 closed APIs on the original benchmarks. Measure the spread of task metrics with vs. without per-model tuning, and total runtime vs. the published multi-pass variants.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
If layer-wise attention patterns are largely epiphenomenal — decoupled from anything measurable in the output distribution — recovered substitutes will never beat logit-only features, and the honest conclusion flips to "the information isn't on the API side." A second risk is economics: if near-ceiling recovery requires hundreds of queries per example, running an open proxy model remains cheaper and the practical case collapses.
Sub-problems
Black-Box and Single-Pass Visual Grounding Without Internal Attention Extraction
Current visual hallucination mitigation and grounding methods in vision-language models rely on intercepting intermediate attention distributions across specific layers or executing multiple forward passes per token. This requirement structurally prevents their deployment on commercial black-box APIs (e.g., GPT-4V) and production serving frameworks (e.g., vLLM, TensorRT-LLM) where attention tensors are fused and never materialized to memory. Consequently, practitioners are forced to choose between unmitigated hallucinations or running unoptimized, white-box model forks with prohibitive inference latency. Developing output-level or single-pass steering mechanisms that operate strictly on output logits or post-hoc verification removes this infrastructure dependency.
Black-Box RAG Attribution and Reliability Without White-Box Internal States
Current high-performing methods for hallucination detection, context attribution, and selective retrieval in RAG depend strictly on white-box access to internal attention maps, hidden states, or value vectors. Because leading commercial LLMs (e.g., GPT-4, Claude) only provide black-box text outputs and limited token probabilities, these RAG control and verification methods cannot be deployed on frontier closed models. Practitioners are forced to choose between using state-of-the-art closed LLMs without attribution/reliability safeguards or using weaker open-source models solely to retain white-box inspection.
Evidence
Each paper's own statement of the limitation, verbatim.
- When Context Bites: Detecting RAG Poisoning via Document-Level Attention CollapseSIGIR 2026
Requires white-box access to the LLM generator's internal attention weight matrices during decoding.
- Understanding and Mitigating Token-Pruning-Induced Vulnerabilities in VLMsICML 2026
Requires white-box access to internal attention weights and hidden states, restricting applicability to open-source VLMs
- SlotGCG: Exploiting the Positional Vulnerability in LLMs for Jailbreak AttacksICLR 2026
Requires white-box access to compute gradients and internal attention maps for VSS calculation.
- Predictive Prefetching for Retrieval-Augmented GenerationICML 2026
Requires white-box access to LLM hidden states, attention, and value vectors, excluding closed-API models like GPT-4/Claude; a logit-only degraded variant drops to 0.66 AUROC, losing most gains
- Estimating Tail Risks in Language Model Output DistributionsICML 2026
Requires white-box access to target model weights and internal activations to compute steering vectors and token likelihoods.
- Decoding Web Memorization: A Semantic Membership Inference Attack on LLMsTheWebConference 2026
Requires white-box or gray-box access to the target LLM to extract multi-layer attention weight matrices across multiple input passes.
- Constructive Distortion: Improving MLLMs with Attention-Guided Image WarpingICLR 2026
Requires per-model layer selection for attention extraction (e.g., layer 20 for LLaVA, 16 for Qwen) and a two-pass inference for the base variant, adding latency
- Breaking the Illusion: When Positive Meets Negative in Multimodal DecodingCVPR 2026
Adds inference overhead from three forward passes per token plus external BLIP-ITM attention extraction (exact cost deferred to supplementary material, not quantified in main text)
Show all 15 papers
- Attention Weights as an Indicator: Analyzing and Improving Document Utilization in Retrieval-Augmented GenerationACL 2026
Requires white-box access to internal attention weights, so closed-source models like ChatGPT/GPT-4 cannot be used
- When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and UnderstandingNeurIPS 2025
Increased inference latency due to the requirement of calculating and comparing layer-wise attention distributions during decoding.
- When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated ExplanationsACL 2025
Requires white-box access to internal activations and attention weights for the logit lens and attention analyses.
- LLMScan: Causal Scan for LLM Misbehavior DetectionICML 2025
Requires white-box access to model internals, including attention scores and the ability to intervene on layer-wise execution.
- Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention LensCVPR 2025
Requires white-box access to the model's internal attention weights and hidden states during inference.
- Boosting Knowledge Utilization in Multimodal Large Language Models via Adaptive Logits Fusion and Attention ReallocationNeurIPS 2025
Relies on access to internal attention weights and output logits, limiting applicability to closed-source MLLMs
- Attention Eclipse: Manipulating Attention to Bypass LLM Safety-AlignmentEMNLP 2025
Requires white-box access to attention weights and gradients, excluding direct use against closed-source proprietary models (only transfer from an open proxy model works)
Nearest existing work
- AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMsICML 2026
- Stealing part of a production language modelICML 2024
- Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMsICLR 2026
- Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM SafetyEMNLP 2025
- JULI: Jailbreak Large Language Models by Self-IntrospectionICLR 2026
- Towards Intrinsic Interpretability of Large Language Models: A Survey of Design Principles and ArchitecturesACL 2026
- Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language ModelsICML 2024
- Attention is not ExplanationNAACL 2019
- Auditing Black-Box LLM APIs with a Rank-Based Uniformity TestICLR 2026
- The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?ECCV 2024
- Revitalizing Black-Box Interpretability: Actionable Interpretability for LLMs via Proxy ModelsACL 2026
- LatentQA: Teaching LLMs to Decode Activations Into Natural LanguageICLR 2026
- Model Equality Testing: Which Model is this API Serving?ICLR 2025
- Hidden No More: Attacking and Defending Private Third-Party LLM InferenceICML 2025
- Taylor Unswift: Secured Weight Release for Large Language Models via Taylor ExpansionEMNLP 2024
Related open problems
Model Safety, Alignment & Jailbreaks
A Framework for Deploying Activation- and Logit-Based Methods Against Closed-Source LLM APIs via Instrumented Proxy Models
A very large body of techniques — safety defenses, interpretability probes, evaluation metrics, decoding controls — requires hidden states, attention maps, gradients, or token-level probabilities, and is therefore silently restricted to open-weight models. As a result, published methods are never validated on, and cannot protect or audit, the models most people actually use (GPT-4, Claude, Gemini). This is the single most common self-reported scope limitation in the literature, yet each paper treats it as an isolated footnote rather than a solvable engineering and inference problem.
Model Safety, Alignment & Jailbreaks
Decision-Only and Logit-Free Safety Auditing and Alignment Evaluation for Black-Box Models
Current safety auditing, jailbreak discovery, and representation-based alignment diagnostics overwhelmingly rely on white-box preconditions such as gradient access, intermediate activations, or output token logit distributions. When models are deployed behind production APIs that return only generated text, practitioners cannot run these diagnostic and red-teaming pipelines directly. Relying on surrogate transfer from open-source models produces high false-negative rates due to incomplete transferability across distinct architectures and tokenizers. Consequently, safety assessments for proprietary, API-governed models remain structurally disconnected from the methods developed in the literature.
Model Safety, Alignment & Jailbreaks
Robust Evaluation of Non-Binary and Phrasing-Agnostic Refusal in Safety-Aligned Language Models
Current safety benchmarks and verifiable reward mechanisms evaluate model refusals primarily through binary keyword matching, hand-crafted refusal templates, or uncalibrated model-based autoraters (which show inter-annotator agreement as low as $\alpha = 0.378$). This creates a blind spot where models that partially comply, use non-standard refusal vocabulary, or alter their stance under multi-turn pushback are misclassified as either fully compliant or safely refusing. Consequently, alignment interventions optimized against these brittle metrics suffer from artificial over-refusal on benign queries while leaking safety risks through stylistic variation and conversational pressure. Without a rigorous, non-binary evaluation standard across diverse phrasing distributions, researchers cannot reliably assess whether safety alignment generalizes beyond rigid heuristic templates.
Model Safety, Alignment & Jailbreaks
Safety Monitoring and Alignment Interventions Under Black-Box and API-Only Constraints
Current state-of-the-art alignment monitoring, jailbreak detection, and activation-steering interventions explicitly require full white-box access to hidden state activations, KV caches, attention logits, or gradient flows. In practice, third-party safety auditors, downstream system builders, and end users interact with models exclusively via black-box query APIs or restricted endpoints. Because existing methods are structurally coupled to internal state inspection and modification, they cannot be deployed or evaluated on proprietary commercial models or in privacy-preserving environments.