Information Extraction & Entity Recognition
Quantifying and Mitigating Upstream Parser Brittleness in Syntactic Information Extraction
Generated automatically from the limitations stated in 3 papers (ACL, EMNLP, NAACL), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Syntactic-semantic information extraction pipelines depend strictly on upstream dependency parsers and part-of-speech taggers, leaving downstream relation and entity extraction vulnerable to cascading errors. When applied to domain-specific jargon or subjected to minor parser version updates, upstream accuracy drops and alters the extracted syntactic structures. This leaves practitioners unable to guarantee extraction stability across domains or library releases without manual parser fine-tuning.
Why it matters
Enables reliable, domain-transferable deployment of syntax-guided information extraction without catastrophic degradation from parser version drift or domain-specific terminology errors.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Multi-parser sensitivity benchmark: Systematically evaluate standard rule- and dependency-based IE systems across varying parser backends (e.g., SpaCy versions, Stanza, CoreNLP) and domain-shifted corpora (biomedical, legal, technical), measuring downstream F1 variance and error propagation rates.
- 2
Uncertainty-aware and k-best syntactic extraction: Formulate extraction rules over n-best dependency parses or parser confidence distributions rather than a single 1-best tree, measuring extraction recovery on out-of-domain text.
- 3
Soft-constraint syntactic graph matching: Replace strict tree-traversal matching with approximate subgraph isomorphism or soft syntactic alignments, measuring precision retention when upstream dependency edges are noisy or mislabeled.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
End-to-end neural relation extraction and direct large language model prompting may entirely bypass explicit syntactic dependency pipelines, rendering dependency-error mitigation obsolete for modern extraction workflows.
Evidence
Each paper's own statement of the limitation, verbatim.
- Structured Semantic Information Helps Retrieve Better Examples for In-Context Learning Applied to Few-Shot Relation ExtractionACL 2026
Requires high-quality dependency parsing and named entity recognition to construct the syntactic-semantic rules.
- CQE: A Comprehensive Quantity ExtractorEMNLP 2023
Relies heavily on dependency parsing and POS tagging — errors propagate through the pipeline, and even SpaCy version changes produce varying results
- RExtractor: a Robust Information ExtractorNAACL 2015
Strictly dependent on the accuracy of the underlying dependency parser, which may struggle with domain-specific jargon
Nearest existing work
- ICE: Rapid Information Extraction Customization for NLP NovicesNAACL 2015
- English Multiword Expression-aware Dependency Parsing Including Named EntitiesACL 2017
- An Evaluation of Parser Robustness for Ungrammatical SentencesEMNLP 2016
- Exploiting Rich Syntactic Information for Semantic Parsing with Graph-to-Sequence ModelEMNLP 2018
- RExtractor: a Robust Information ExtractorNAACL 2015
- Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information ExtractionEMNLP 2023
- SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented ParserEMNLP 2022
- Automatically Selecting the Best Dependency Annotation Design with Dynamic OraclesNAACL 2018
- English Resource SemanticsNAACL 2016
- Transforming Dependencies into Phrase StructuresNAACL 2015
- Ckylark: A More Robust PCFG-LA ParserNAACL 2015
- Leveraging Dependency Forest for Neural Medical Relation ExtractionEMNLP 2019
- Transition-based Dependency DAG Parsing Using Dynamic OraclesACL 2015
- A Domain-independent Rule-based Framework for Event ExtractionACL 2015
- A general framework for information extraction using dynamic span graphsNAACL 2019
Related open problems
Information Extraction & Entity Recognition
Error-Aware Entity Linking Evaluation and Propagation-Bounded Pipelines for Downstream NLP Tasks
Virtually every system that touches entities — relation extraction, KBQA, semantic parsing, coreference, retrieval-augmented generation — treats entity linking as a solved preprocessing step and evaluates against gold links, so the systems' true field performance is unknown and unlinkable to their stated failure modes. When linking is wrong, downstream modules cannot detect, recover from, or bound the damage; reported failure analyses put linking errors at 15–35% of end-to-end errors. Today nobody can quantify how much of a downstream system's headroom is actually upstream, because no benchmark couples controlled link corruption with propagation measurement.
Information Extraction & Entity Recognition
Semantic Typing and Grounding of Pronominal and Nominal Mentions Across Divergent Coreference Schemas
Current entity recognition and extraction pipelines exhibit sharp performance drops when evaluated on schemas that include pronominal and nominal mentions, dropping to as low as 42% recall on datasets like ACE 2005. Because most taggers assume proper named entities, models fail to type clusters composed entirely of pronouns or misclassify ambiguous pronouns (such as mistaking collective 'we' as person rather than organization) in the absence of cross-sentence antecedent context. Consequently, roughly 10% or more of coreference mentions remain completely ungrounded or unassigned, preventing reliable document-level entity extraction.