Language Agents & Tool Use

Benchmarking and Diagnosing Agent Framework Failures Across Open-Weight LLM Backbones

Effect to explainOpen

Possible candidate · 2/5 runs7 papers report this100% from 2025+

Generated automatically from the limitations stated in 7 papers (ACL, TheWebConference, NAACL), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current language agent and tool-use frameworks are overwhelmingly benchmarked on proprietary frontier models (GPT-4 class), masking severe brittleness on open-weight and smaller models. When executed on 7B–8B backbones, these frameworks experience catastrophic failure rates, violating task constraints in over 37% of cases and frequently generating invalid API queries or malformed function calls. This restricts practical agent deployment exclusively to cloud APIs, blocking autonomous tool use in privacy-sensitive, air-gapped, or resource-constrained local environments.

Why it matters

Reliable deployment of tool-using language agents on locally hosted, open-source models without dependency on proprietary commercial APIs.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

1
Systematic Cross-Backbone Robustness Benchmark: Deploy a standardized suite of representative agent frameworks across 10 open-weight models (3B to 70B parameters) across diverse domains, measuring tool-calling validity, schema adherence rates, and end-to-end task completion.
2
Interface and Schema Robustness Evaluation: Test whether constrained decoding (e.g., JSON grammar masking), schema decomposition, and intermediate syntax error-correction loops systematically recover tool invocation accuracy on 7B–8B backbones.
3
Scaffold Simplification Study: Compare full multi-turn autonomous agent loops against modular, single-step deterministic state-machine scaffolds across model scales to quantify the minimum backbone capacity required for each level of agent autonomy.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Why it might fail

Rapid baseline capability jumps in sub-8B instruction-tuned models might solve schema compliance natively, rendering agent scaffolding adaptations obsolete, or high domain variance in agent tasks may prevent framework-level interventions from generalizing.

Evidence

Each paper's own statement of the limitation, verbatim.

ST-LEGO: Large Language Models as Modular Architects for Traffic PredictionTheWebConference 2026
Performance depends on LLM backbone quality: LLaMA3.1-8B satisfied task constraints in only 62.96% of capacity-oriented cases and shows high variability, and removing mutation degraded Qwen2.5-7B MAE on PEMS03 from 10.63 to 17.14
MavenCoder: Competitive Code Generation via Model Adaptive Planning Strategies and Multi-Perspective Verification EnhancementACL 2026
Evaluated only on large LLMs (GPT-4.1-nano, GPT-4o-mini, Qwen3-Coder+); medium-scale locally deployed models untested
SheetAgent: Towards a Generalist Agent for Spreadsheet Reasoning and Manipulation via Large Language ModelsTheWebConference 2025
Inappropriate for backbones other than GPT-4/3.5-class models: smaller LLMs (Qwen-1.5, Llama 3) show high rates of improper function calls and inaccurate queries
SELFGOAL: Your Language Agents Already Know How to Achieve High-level GoalsNAACL 2025
Performance depends on the backbone LLM's understanding and summarization ability; smaller/weaker models may not fully realize the framework's potential (e.g., GPT-4-built tree adds +2.87 Auction / +3.10 Bargaining over a GPT-3.5-built tree)
Planning-Driven Programming: A Large Language Model Programming WorkflowACL 2025
Bounded by the backbone LLM's reasoning capacity; residual failures stem from ambiguous problem descriptions (e.g., HumanEval problems 91, 132, 145)
LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge RetrievalEMNLP 2025
Performance depends heavily on the backbone LLM's function-calling and reasoning ability; LLMs sometimes misunderstand schema descriptions, and weaker backbones degrade sharply (LLaMP with Llama3-8B has bulk modulus MAE ~50 GPa and SCoR 0.029)
ExploraCoder: Advancing Code Generation for Multiple Unseen APIs via Planning and Chained ExplorationACL 2025
Performance is bounded by the underlying LLM's capability; small models with weak long-context and API-usage ability may gain little