Benchmarking the Robustness of Syntax-Guided NLP Across Parser Degradation, Languages, and Formalisms

The problem

Syntax-guided models and parser-assisted NLP pipelines rely heavily on upstream parsers, yet their evaluations remain largely confined to high-resource English benchmarks using high-quality silver or gold parses. Current literature explicitly notes that methods have not been evaluated when parsers degrade under domain shift, when deployed on low-resource languages lacking specialized parsers, or when adapted across differing formalisms (e.g., constituency versus dependency trees). Consequently, practitioners cannot determine whether syntax-guided gains persist under realistic, noisy, or cross-lingual parsing conditions, or if parser error propagation negates structural benefits entirely.

Why it matters

Establishes empirical bounds on when incorporating syntactic parses is beneficial versus harmful in downstream NLP pipelines, enabling reliable deployment in low-resource and domain-shifted settings.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

1. 1

   Systematic Parser Corruption and Domain Shift Benchmark: Test downstream syntax-reliant models (e.g., syntax-augmented LMs, structured alignment, syntax-guided generation) across controlled noise levels (synthetically perturbed trees) and out-of-domain text corpora, measuring downstream task degradation as a function of upstream parse F1/UAS.

   Check for prior work
2. 2

   Cross-Lingual Evaluation across Low-to-Medium Resource Parsers: Evaluate existing syntax-guided methods on non-English languages from Universal Dependencies where parser quality varies widely, measuring whether syntax-guided models outperform syntax-free baselines in low-resource settings.

   Check for prior work
3. 3

   Cross-Formalism Robustness and Adaptation: Implement and evaluate syntax-dependent architectures using both dependency and constituency representations across identical downstream tasks to quantify how sensitive performance is to the choice of syntactic formalism.

   Check for prior work

Why it might fail

If modern dense foundation models make syntax-augmented architectures entirely obsolete across all evaluated downstream tasks, rendering robustness bounds irrelevant to current practice.

Evidence

Each paper's own statement of the limitation, verbatim.

• Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMsACL 2026

  Performance is sensitive to parser quality and domain shift: a weak parser drops BLiMP to 78.25 and CLOTH to 80.66, and corrupted trees act as harmful noise, making real-world use brittle when parses degrade

• Sneaking Syntax into Transformer Language Models with Tree RegularizationNAACL 2025

  Requires constituency-parsed data (silver parses from e.g. Benepar), limiting applicability to English and constituency-structured languages; experiments are English-only

• What Does Parameter-free Probing Really Uncover?ACL 2024

  Qualitative analysis of individual parse pairs has small manual coverage; numerical shift tables alone cannot fully characterize the discrepancies

• Structured Tree Alignment for Evaluation of (Speech) Constituency ParsingACL 2024

  Not directly applicable to other parse types such as dependency trees; alignment constraints would need modification, which is left as future work

• S2GSL: Incorporating Segment to Syntactic Enhanced Graph Structure Learning for Aspect-based Sentiment AnalysisACL 2024

  Requires external parsers at inference: Stanford dependency parsing and a CRF constituency parser, so errors from these tools propagate; syntax/segment quality constrains the method

• Pushdown Layers: Encoding Recursive Structure in Transformer Language ModelsEMNLP 2023

  Requires constituency-parse-annotated data; unavailable for many languages lacking strong off-the-shelf parsers, and all experiments are English-only

• NEUROSTRUCTURAL DECODING: Neural Text Generation with Structural ConstraintsACL 2023

  Relies on a single language (English) and syntactic dependency constraints only; unclear performance for low-resource languages where parsers are weaker

• Hexatagging: Projective Dependency Parsing as TaggingACL 2023

  Does not model dependency arcs directly, making outputs harder to interpret than graph-based parsers that score arcs explicitly

Nearest existing work

• A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languagesEMNLP 2019
• An Evaluation of Parser Robustness for Ungrammatical SentencesEMNLP 2016
• CSSL: Contrastive Self-Supervised Learning for Dependency Parsing on Relatively Free Word Ordered and Morphologically Rich Low Resource LanguagesEMNLP 2024
• On the Benefit of Syntactic Supervision for Cross-lingual Transfer in Semantic Role LabelingEMNLP 2021
• SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented ParserEMNLP 2022
• One model, two languages: training bilingual parsers with harmonized treebanksACL 2016
• Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMsACL 2020
• Multilingual Pre-training with Universal Dependency LearningNeurIPS 2021
• On the Relation between Syntactic Divergence and Zero-Shot PerformanceEMNLP 2021
• Semantic Parsing for English as a Second LanguageACL 2020
• Grammar as a Foreign LanguageNeurIPS 2015
• Syntax-Enhanced Neural Machine Translation with Syntax-Aware Word RepresentationsNAACL 2019
• Low Resource Dependency Parsing: Cross-lingual Parameter Sharing in a Neural Network ParserACL 2015
• Exploiting Rich Syntactic Information for Semantic Parsing with Graph-to-Sequence ModelEMNLP 2018
• Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence RepresentationsEMNLP 2019