Open Problems

Neural Text Generation & Decoding

Benchmarking and Extending Machine-Generated Text Detection Under Black-Box, Mixed-Author, and Multi-Source Settings

Barrier to removeOpen
Strong candidate · 4/5 runs3 papers report this67% from 2025+

Generated automatically from the limitations stated in 3 papers (ACL, NeurIPS, EMNLP), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current machine-generated text detection frameworks rely on preconditions that fail in real-world deployments: access to token-level log-probabilities from the source model, abundant paired human reference corpora, and strictly binary (100% human vs. 100% machine) document inputs. Because these methods have not been evaluated across black-box commercial APIs, partially edited or collaborative human-AI text, and multi-model source attribution, their true utility is unknown. Practitioners cannot reliably determine whether a piece of text was generated, co-authored, or which specific model produced it when log-probabilities are inaccessible.

Why it matters

Enables trustworthy auditing of text provenance for closed-source models and hybrid human-AI documents without requiring proprietary access to token log-probabilities.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Construct a standardized evaluation benchmark containing text generated across varying blend ratios (0% to 100% machine-written at sentence and token granularities) from both open-source and closed-source LLMs without log-probability access, measuring AUROC and F1 degradation across existing surrogate-based and zero-shot detectors.

  2. 2

    Develop a black-box multi-class attribution and boundary-detection method using surface-level linguistic features and rank-based perturbation probing, measuring source attribution accuracy and boundary localization error (token-level IoU).

  3. 3

    Evaluate the susceptibility of black-box detectors to inherited corpus biases and hallucinations by measuring false-positive rates on domain-shifted, non-native English, and factually erroneous human text.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If watermarking schemes or proprietary API metadata become standard across all major LLM providers, post-hoc statistical detection and attribution may become obsolete. Additionally, frontier LLM outputs may converge so closely to human distributions that black-box detection without log-probabilities hits an unavoidable statistical upper bound on short passages.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Neural Text Generation & Decoding

Barrier to removePartly addressed

Zero-Shot Machine-Generated Text Detection Under Strict API and Log-Probability Constraints

Current zero-shot machine-generated text detectors impose heavy access preconditions on the suspected source model, requiring either exact token log-probabilities, specialized completion API endpoints exposing top-K logprobs, or multiple live regeneration queries per candidate text. These requirements fail when evaluating outputs from proprietary models that only provide text-level chat interfaces without log probabilities, when source API access is restricted or cost-prohibitive, or when the generating model identity is unknown. Under these realistic operational constraints, falling back to surrogate scoring models results in steep performance drops (cross-model AUROC falling to 0.6–0.9), leaving auditors without viable tools to verify proprietary or uncooperative model outputs.

Strong candidate · 4/5 runs4 papers report this25% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.