Open Problems

Machine Unlearning

Machine Unlearning and Verification Under Black-Box Model Access Constraints

Barrier to removeOpen
Strong candidate · 1/1 runs3 papers report this100% from 2025+

Generated automatically from the limitations stated in 3 papers (ICML, ICLR), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current machine unlearning algorithms and formal auditing frameworks overwhelmingly require white-box access to model parameters, gradients, or pre- and post-unlearning checkpoint weights. This structurally prevents unlearning operations and compliance verification on closed-source, API-hosted, or third-party proprietary models where internal weights cannot be inspected or directly modified. Without methods and evaluation protocols that operate under black-box or query-only access, unlearning remains unusable for hosted foundation models and unverifiable by external auditors.

Why it matters

Enables external verification of compliance with deletion requests and makes unlearning strategies applicable to proprietary, API-mediated model deployments.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Benchmark black-box unlearning approximations (such as logit suppression, decoding-time intervention, and retrieval-augmented negative prompting) against standard white-box gradient ascent across benchmark forget sets (e.g., ToFU), measuring forget quality and general utility retention.

  2. 2

    Develop and evaluate black-box verification protocols that test whether target concepts have been removed using only input-output interaction and confidence scores, measuring auditing sensitivity without access to model weights or algorithm specifications.

  3. 3

    Quantify the empirical performance gap between white-box parameter edits and query-level black-box interventions across varying model scales and unlearning task granularities (e.g., verbatim sequence removal vs. factual knowledge erasure).

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Black-box unlearning without parameter updates may reduce entirely to surface-level guardrailing, making it vulnerable to trivial jailbreaks and structurally incapable of matching true weight-space forgetting.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Machine Unlearning

Scope to testOpen

A Standardized Multi-Architecture Testbed of Memorized Concepts, Generation Fingerprints, and Editable Structures in Text-to-Image Models Beyond Stable Diffusion v1.x

Nearly every quantitative claim about text-to-image models — memorization, unlearning success, watermark robustness, deepfake provenance, concept editability — is anchored to a handful of frozen Stable Diffusion v1.x/v2 checkpoints, because those are the only models with established benchmarks, known memorized prompts, and validated concept labels. Newer architectures (DiT-based diffusion, DeepFloyd, autoregressive and GAN generators, video models) are largely untested, so results that appear robust may be artifacts of one 2022 model family. The field cannot tell whether its findings are properties of generative modeling or properties of Stable Diffusion 1.4.

Strong candidate · 5/5 runs22 papers report this88% from 2025+

Machine Unlearning

Effect to explainPartly addressed

Benchmarking the Adversarial Robustness and Reversibility of LLM Unlearning

Existing LLM unlearning methods are predominantly evaluated on standard, benign queries, giving a false sense of compliance with privacy and copyright demands. Empirical evidence shows that "unlearned" knowledge remains extractable via adversarial jailbreaks, latent-space elicitation, and few-shot relearning. Without systematic evaluation across these extraction vectors, practitioners have no way to verify whether a model has actually eliminated sensitive data or merely applied a superficial suppression mask.

Possible candidate · 2/5 runs6 papers report this83% from 2025+

Machine Unlearning

Effect to explainPartly addressed

Empirical Robustness and Sensitivity of Machine Unlearning Under Realistic Data and Checkpoint Constraints

Existing machine unlearning algorithms for language models are predominantly evaluated under idealized conditions: complete access to forget sets, clean entity anchors, standard prose, and checkpoints saved immediately before target data exposure. In practice, unlearning requests frequently present partial forget data, domain variations like code, noisy or alias-heavy entity mentions, and checkpoints separated from target exposure by billions or trillions of tokens. Practitioners currently cannot predict whether an unlearning method that succeeds on curated benchmarks like TOFU will retain any efficacy when deployed under these real-world data and provenance constraints.

Strong candidate · 1/1 runs4 papers report this50% from 2025+

Machine Unlearning

Barrier to removePartly addressed

Machine Unlearning Under Degraded Operational Preconditions

Existing machine unlearning methods rely on strict operational preconditions: access to clean retain datasets ($D_r$), full white-box parameter access, historical pre-training checkpoints, or original pre-unlearned calibration weights. In production environments, intermediate training checkpoints are routinely deleted to save storage, retain data is often inaccessible due to data governance policies, and deployment interfaces may restrict full weight access. Without methods that function in the absence of these preconditions, deployed models cannot legally or practically comply with data deletion requests.

Possible candidate · 2/5 runs4 papers report this75% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.