Open Problems

Low-Resource Languages & Dialects

Reliability-Calibrated Annotation Protocols for Subjective and Low-Resource labeling Tasks

Effect to explainOpen
Possible candidate · 2/5 runs19 papers report this60% from 2025+

Generated automatically from the limitations stated in 19 papers (ACL, EMNLP, NAACL), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Across many NLP annotation efforts, gold standards rest on so few annotators—and on tasks so subjective—that reported inter-annotator agreement is often moderate at best (κ 0.18–0.46), and in some cases reliability fails outright (agreement too weak to include in evaluation at all). Because papers rarely distinguish "the task is ambiguous" from "the pool was too small," downstream users cannot judge how trustworthy the gold labels are, and resources with single-annotator or single-community provenance silently embed one worldview. Structurally, there is no standard way to report, audit, or correct for annotation reliability when it is low or unmeasured.

Why it matters

Trustworthy evaluation in low-resource and subjective domains becomes possible: datasets can be compared on evidence quality, not just size, and papers can claim reliability instead of assuming it.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Reliability auditing of existing datasets: take 15–20 public datasets with published agreement statistics and re-estimate per-item label confidence using bootstrap over annotator subsamples, measuring how downstream model rankings change when only high-confidence items are used. Deliverable: a "reliability report card" format for datasets.

  2. 2

    Minimum-annotator/minimum-agreement guidelines: run controlled annotation studies on the same subjective task with pools of 1, 3, 5, and 10 annotators, plotting measured agreement stability and downstream metric variance as a function of pool size, to produce evidence-based thresholds for when a gold standard is fit for purpose.

    Partly done

    Paper [9] establishes empirical and bootstrap methods for measuring reliability as a function of rater pool size ($k$-rater reliability) on WordSim-353, but does not evaluate downstream metric variance or provide general fit-for-purpose thresholds across subjective tasks.

    Other papers read (9)

    Checked 9/26/2026.

  3. 3

    Disagreement-preserving labels: pilot releasing distributions or per-annotator labels (rather than adjudicated gold) for one contested task, measuring whether models trained on soft labels match human judgment distributions better than those trained on collapsed gold.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If re-analysis shows model rankings are robust to low agreement—i.e., noisy gold labels don't actually change conclusions—then the audit protocol solves a problem with little practical consequence.

Sub-problems

  • Benchmarking Subjective Content Moderation Under Persistent Multi-Annotator Disagreement

    Current computational social science and content moderation benchmarks collapse subjective judgments into a single adjudicated "gold" label, despite observed inter-annotator agreement metrics falling into poor or moderate ranges (e.g., Fleiss' $\kappa < 0.45$, Gwet's AC1 $< 0.30$). These labels are routinely produced by small, demographically narrow pools (often 2–4 in-person annotators, the paper authors, or single-region crowdsource workers), baking localized cultural and political worldviews directly into ground truth. As a result, standard evaluation pipelines cannot distinguish between a model that fails at content understanding and one that simply aligns with a valid alternative perspective absent from the original annotator pool.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 19 papers

Nearest existing work

Related open problems

Low-Resource Languages & Dialects

Effect to explainPartly addressed

A Benchmark and Labeling Framework for Community-Contingent Hate Speech Annotations, Replacing Single Gold Labels with Distributions of Plausible Ones

Every current hate speech dataset collapses a genuinely plural judgment — what counts as hateful depends on annotator identity, community norms, reclamation practices, and exposure to implicit forms — into one label, then trains and evaluates models against it. This means models cannot represent the disagreement they were built from, audits of pretraining data and model outputs in low-resource settings have no trustworthy ground truth to measure against, and moderation systems calibrated on one community's thresholds misfire (or stay silent) on another's counter-speech, quotes, and reclaimed slurs. Structurally, nobody today can state, for a given utterance, how hateful-label assignments vary across the populations that matter — so "biased model" and "ambiguous data" are permanently confounded.

Possible candidate · 3/5 runs8 papers report this25% from 2025+

Low-Resource Languages & Dialects

Barrier to removeOpen

Metric Inflation and Reliability Limits in IPA-Space Generation Evaluation

Multilingual systems increasingly evaluate generations in International Phonetic Alphabet (IPA) space to bridge script disparities, yet standard character-level metrics like chrF inflate scores because IPA relies heavily on shared Latin base characters. Furthermore, automatic grapheme-to-phoneme tools (such as Epitran) introduce systematic transcription errors and inventory mismatches that are masked when evaluating against synthetic or machine-translated benchmarks. Researchers currently cannot distinguish whether high IPA-level chrF gains reflect genuine semantic translation quality or superficial phonetic transcription artifacts.

Possible candidate · 2/5 runs3 papers report this100% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.