Open Problems

Language Agents & Tool Use

Benchmark Verification for Language Agents Without Ground-Truth Test Suites or Reference Repositories

Barrier to removePartly addressed
Possible candidate · 2/5 runs3 papers report this67% from 2025+

Generated automatically from the limitations stated in 3 papers (ACL, ICML), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current benchmarks for code-generating and paper-reproducing language agents rely strictly on pre-existing public GitHub repositories with functional, high-coverage unit tests as ground truth. This filters out the vast majority of scientific literature and closed-source software where code is missing, incomplete, or untested. As a result, agent capabilities cannot be benchmarked on genuine zero-code reproduction tasks or messy real-world codebases without human experts manually engineering executable harnesses.

Why it matters

Enables automated, rigorous evaluation of language agents across arbitrary scientific domains and proprietary codebases without requiring author-released code or pre-existing unit test suites.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Build an automated execution-verification pipeline that extracts empirical claims, mathematical constraints, and input/output contracts directly from paper text and pseudo-code, measuring the agreement rate between contract-derived assertion suites and ground-truth execution across 50 papers with known code.

  2. 2

    Evaluate multi-agent differential consensus, where multiple diverse agent architectures generate independent implementations from paper descriptions, measuring the precision and recall of automated fault detection using cross-execution consistency rather than author-provided reference code.

  3. 3

    Design a metric for task validity in zero-test repositories based on metamorphic testing and property-based invariant generation, measured against codebases with established high-coverage test suites as an empirical upper bound.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If text-only specification ambiguity fundamentally prevents automated invariant extraction without human-in-the-loop clarification, or if semantic divergence across independent implementations cannot be distinguished from valid algorithmic design choices without ground-truth artifacts.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Language Agents & Tool Use

Effect to explainOpen

Benchmarking and Diagnosing Agent Framework Failures Across Open-Weight LLM Backbones

Current language agent and tool-use frameworks are overwhelmingly benchmarked on proprietary frontier models (GPT-4 class), masking severe brittleness on open-weight and smaller models. When executed on 7B–8B backbones, these frameworks experience catastrophic failure rates, violating task constraints in over 37% of cases and frequently generating invalid API queries or malformed function calls. This restricts practical agent deployment exclusively to cloud APIs, blocking autonomous tool use in privacy-sensitive, air-gapped, or resource-constrained local environments.

Possible candidate · 2/5 runs7 papers report this100% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.