Neural Text Generation & Decoding

Zero-Shot Machine-Generated Text Detection Under Strict API and Log-Probability Constraints

Barrier to removePartly addressed

Strong candidate · 4/5 runs4 papers report this25% from 2025+

Generated automatically from the limitations stated in 4 papers (ICLR, ICML, EMNLP), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current zero-shot machine-generated text detectors impose heavy access preconditions on the suspected source model, requiring either exact token log-probabilities, specialized completion API endpoints exposing top-K logprobs, or multiple live regeneration queries per candidate text. These requirements fail when evaluating outputs from proprietary models that only provide text-level chat interfaces without log probabilities, when source API access is restricted or cost-prohibitive, or when the generating model identity is unknown. Under these realistic operational constraints, falling back to surrogate scoring models results in steep performance drops (cross-model AUROC falling to 0.6–0.9), leaving auditors without viable tools to verify proprietary or uncooperative model outputs.

Why it matters

Enables independent third-party auditing and detection of synthetic text generated by closed-source, proprietary, or unknown models without recurring API costs or dependency on provider-exposed log probabilities.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

1
Construct a standardized benchmark evaluating existing zero-shot, surrogate, and perturbation-based detectors across four concrete access regimes (full logprobs, top-K completion logprobs, chat-only text queries, and completely offline surrogate scoring), measuring AUROC degradation and cost per detection across diverse proprietary and open target generators.
2
Develop an offline intrinsic detection method that leverages local open-weight representation geometry rather than querying the suspected source API, measuring cross-generator AUROC and false-positive rates on out-of-domain text.
3
Formulate a local surrogate-perturbation framework that approximates token sensitivity without making API regeneration calls to the target model, measuring transferability across model families and resilience to decoding parameter variations.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Why it might fail

If the representation gap between proprietary closed-source models and open-source reference models is too wide, any offline surrogate-based or intrinsic metric may fundamentally fail to separate human text from state-of-the-art model outputs without white-box likelihoods.

Evidence

Each paper's own statement of the limitation, verbatim.

Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text DetectionICLR 2025
Requires the proprietary model to expose a Completion API with top-K logprobs; models offering only ChatCompletion-style APIs without this cannot be used as scoring models
DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated TextICLR 2024
Requires K (5-10+) API regeneration calls per candidate text, so detection cost is multiplied versus single-pass classifiers, and relies on continued API access to the suspect model
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability CurvatureICML 2023
Requires white-box access to the source model's log probabilities, which costs money via APIs and is unavailable for ChatGPT; performance drops substantially when using a surrogate scoring model (cross-model AUROC falls to ~0.6-0.9)
Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection via Querying ChatGPTEMNLP 2023
Requires querying the ChatGPT API for every text to detect, incurring monetary cost and dependence on the very model being detected