Error-Aware Entity Linking Evaluation and Propagation-Bounded Pipelines for Downstream NLP Tasks

The problem

Virtually every system that touches entities — relation extraction, KBQA, semantic parsing, coreference, retrieval-augmented generation — treats entity linking as a solved preprocessing step and evaluates against gold links, so the systems' true field performance is unknown and unlinkable to their stated failure modes. When linking is wrong, downstream modules cannot detect, recover from, or bound the damage; reported failure analyses put linking errors at 15–35% of end-to-end errors. Today nobody can quantify how much of a downstream system's headroom is actually upstream, because no benchmark couples controlled link corruption with propagation measurement.

Why it matters

Honest end-to-end evaluation of entity-dependent systems, and pipelines that degrade gracefully instead of silently failing on the 15–35% of inputs where linking goes wrong.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

1. 1

   Build a propagation benchmark: take 4–6 public systems (relation extraction, KBQA, RAG) and inject calibrated entity-linking corruptions (wrong-entity, nil, ambiguous-rare) at controlled rates, measuring downstream metric degradation curves and per-error-type sensitivity. Deliverable: propagation coefficients per task, showing where a 5% link error rate does 5% vs. 30% damage.

   Check for prior work
2. 2

   Add link-confidence propagation: equip linkers with calibrated uncertainty scores, then test whether downstream systems conditioned on those scores (defer, re-link, abstain) recover degraded performance versus ignoring them. Measure recovery ratio per task.

   Check for prior work
3. 3

   Joint re-ranking at the boundary: train a lightweight verifier that re-scores top-m link candidates using downstream-model feedback signals, measuring end-to-end accuracy on WebQSP-style and RE benchmarks against fixed-link baselines.

   Check for prior work

Why it might fail

The corruption benchmark may find propagation is roughly linear and boring everywhere, yielding no useful signal for error-aware conditioning — and real linker errors may be so correlated with hard inputs that synthetic corruption understates the problem.

Sub-problems

• Error-Tolerant and Backtracking Semantic Parsing for Knowledge Base Question Answering Under Noisy Entity Linking

  Current knowledge base question answering (KBQA) and semantic parsing systems treat entity linking as an immutable prerequisite or assume gold topic entities as given. When initial entity linking produces incorrect nodes, the entire downstream reasoning and graph traversal phase fails catastrophically, accounting for 15% to 35% of total benchmark errors. Because reasoning engines cannot backtrack, revise, or jointly re-rank candidate entity anchors during execution, existing methods are unusable in realistic, end-to-end uncurated settings.

• Robust Knowledge Graph Retrieval-Augmented Generation Under Entity Linking Uncertainty and Failure

  Current graph-augmented RAG frameworks strictly require upstream entity linking and named entity recognition to identify seed nodes before traversing knowledge graphs. When queries involve rare entities, ambiguous surface forms, or concepts outside the knowledge base, this hard preprocessing pipeline either drops retrieval entirely or cascades incorrect seed entities into generation. Consequently, hybrid text-graph RAG systems cannot reliably operate on long-tail domain queries or informal text where standard entity linkers produce noisy or null outputs.

Evidence

Each paper's own statement of the limitation, verbatim.

• S-Path-RAG: Semantic-Aware Shortest-Path Retrieval Augmented Generation for Multi-Hop Knowledge Graph Question AnsweringTheWebConference 2026

  Relies on an entity linker for seed quality; wrong entity links degrade results, mitigated only by top-m candidate retention and LLM disambiguation requests

• Named Entity-Driven Graph Smoothing to Enhance Pretrained Document Embeddings in Clustering TasksSIGIR 2026

  Requires corpus-appropriate NER pipelines (e.g., CamemBERT, DeBERTa, or GPT-5) and entity embedding training, introducing upstream dependency on entity extraction quality.

• KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question AnsweringICML 2026

  Assumes gold topic entities are given (entity linking is assumed solved), so failures from incorrect entity linking are excluded

• Exploration-and-Thinking: Agentic Reasoning over Knowledge Graphs via an LLM-RL Synergized FrameworkSIGIR 2026

  Depends on entity linking as a prerequisite; linking errors propagate to downstream reasoning

• Backjump-on-Graph: Empowering Large Language Models with Reinforced Retrospective Exploration for Agentic Knowledge Graph ReasoningICML 2026

  Assumes correct initial entity linking; linking errors propagate through the whole reasoning phase

• RAED: Retrieval-Augmented Entity Description Generation for Emerging Entity Linking and DisambiguationEMNLP 2025

  Relies solely on Wikipedia as knowledge source, so entities poorly covered or emerging outside mainstream documentation cannot be described; index passage filtering also requires entities to be present in the training datasets

• HydraRAG: Structured Cross-Source Enhanced Large Language Model ReasoningEMNLP 2025

  Requires high-precision entity alignment between text and Knowledge Graphs, which may degrade with ambiguous or rare entities

• Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language ModelsICML 2025

  Dependent on the accuracy of initial entity linking to map question entities to the Knowledge Graph.

Nearest existing work

• Entity Linking for Spoken LanguageNAACL 2015
• A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking SystemsEMNLP 2023
• Robustness Evaluation of Entity Disambiguation Using Prior Probes: the Case of Entity OvershadowingEMNLP 2021
• CleanCoNLL: A Nearly Noise-Free Named Entity Recognition DatasetEMNLP 2023
• LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using UncertaintyTheWebConference 2024
• Lambda: Learning Matchable Prior For Entity Alignment with Unlabeled Dangling CasesNeurIPS 2024
• Separating Retention from Extraction in the Evaluation of End-to-end Relation ExtractionEMNLP 2021
• Improving Neural Entity Disambiguation with Graph EmbeddingsACL 2019
• Improving Entity Disambiguation by Reasoning over a Knowledge BaseNAACL 2022
• EDIN: An End-to-end Benchmark and Pipeline for Unknown Entity Discovery and IndexingEMNLP 2022
• Comparative evaluation of boundary-relaxed annotation for Entity Linking performanceACL 2023
• Joint Entity Recognition and DisambiguationEMNLP 2015
• BioFEG: Generate Latent Features for Biomedical Entity LinkingEMNLP 2023
• A Comprehensive Evaluation of Biomedical Entity Linking ModelsEMNLP 2023
• Aligning Knowledge and Text Embeddings by Entity DescriptionsEMNLP 2015

Related open problems