Open Problems

Low-Resource Languages & Dialects

Metric Inflation and Reliability Limits in IPA-Space Generation Evaluation

Barrier to removeOpen
Possible candidate · 2/5 runs3 papers report this100% from 2025+

Generated automatically from the limitations stated in 3 papers (ACL, NAACL, EMNLP), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Multilingual systems increasingly evaluate generations in International Phonetic Alphabet (IPA) space to bridge script disparities, yet standard character-level metrics like chrF inflate scores because IPA relies heavily on shared Latin base characters. Furthermore, automatic grapheme-to-phoneme tools (such as Epitran) introduce systematic transcription errors and inventory mismatches that are masked when evaluating against synthetic or machine-translated benchmarks. Researchers currently cannot distinguish whether high IPA-level chrF gains reflect genuine semantic translation quality or superficial phonetic transcription artifacts.

Why it matters

Reliable evaluation of phoneme-based multilingual language models and translation pipelines without artificial score inflation. Practitioners can accurately benchmark phonetic representations against native-script baselines.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Construct a diagnostic evaluation suite comparing character-level overlap metrics (chrF, BLEU) on parallel native-script text versus Epitran-derived IPA representations across multiple non-Latin language families, measuring metric correlation with human adequacy and quality judgments.

  2. 2

    Quantify the error propagation of upstream G2P conversion on downstream generation evaluation by perturbing phonetic inventory granularity, broad versus narrow transcriptions, and G2P tool choice across benchmark datasets.

  3. 3

    Develop a weighted phonetic distance metric that penalizes phonological feature divergence rather than surface character n-gram overlap, measuring its calibration against human ratings on low-resource translation pairs.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If community consensus moves entirely to end-to-end neural semantic metrics (like COMET) that bypass surface/phonetic transcriptions altogether, or if phoneme-based NLP tokenization fails to gain wider adoption beyond niche low-resource benchmarks.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Low-Resource Languages & Dialects

Effect to explainOpen

Reliability-Calibrated Annotation Protocols for Subjective and Low-Resource labeling Tasks

Across many NLP annotation efforts, gold standards rest on so few annotators—and on tasks so subjective—that reported inter-annotator agreement is often moderate at best (κ 0.18–0.46), and in some cases reliability fails outright (agreement too weak to include in evaluation at all). Because papers rarely distinguish "the task is ambiguous" from "the pool was too small," downstream users cannot judge how trustworthy the gold labels are, and resources with single-annotator or single-community provenance silently embed one worldview. Structurally, there is no standard way to report, audit, or correct for annotation reliability when it is low or unmeasured.

Possible candidate · 2/5 runs19 papers report this60% from 2025+

Low-Resource Languages & Dialects

Effect to explainPartly addressed

A Benchmark and Labeling Framework for Community-Contingent Hate Speech Annotations, Replacing Single Gold Labels with Distributions of Plausible Ones

Every current hate speech dataset collapses a genuinely plural judgment — what counts as hateful depends on annotator identity, community norms, reclamation practices, and exposure to implicit forms — into one label, then trains and evaluates models against it. This means models cannot represent the disagreement they were built from, audits of pretraining data and model outputs in low-resource settings have no trustworthy ground truth to measure against, and moderation systems calibrated on one community's thresholds misfire (or stay silent) on another's counter-speech, quotes, and reclaimed slurs. Structurally, nobody today can state, for a given utterance, how hateful-label assignments vary across the populations that matter — so "biased model" and "ambiguous data" are permanently confounded.

Possible candidate · 3/5 runs8 papers report this25% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.