arXiv:2608.20047 · v2

Auditing Cross-Lingual Fairness in Language Model Watermarking

首次提交 2026-08-20 21:48

版本更新 2026-11-10 14:11

雷达观测 2026-11-11 13:08

摘要

Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements. Multilingual deployment exposes evaluation-design choices that are inconsequential on English but determine conclusions cross-lingually. We propose an evaluation framework with four components: detection thresholds calibrated empirically per deployment context, a threshold-independent companion measurement that distinguishes calibration failures from detection failures, three disjoint quality measurement paradigms (distributional, paired-semantic, and reference-perplexity), and a generalized-entropy decomposition of cross-language disparity over a typological family partition. Applied to six watermarking schemes, three open-weight generators, eleven languages spanning four scripts and eight typological families, and both base and instruction-tuned regimes, the framework reveals failure modes that single-language, single-paradigm evaluation cannot surface. Against a shape-matched null, detection disparity is predominantly between-family on the typological partition, indicating that detection gaps are structural to language properties rather than idiosyncratic to particular languages; quality disparity is large but not reliably family-structured.

作者备注：Accepted at NeurIPS 2026 Evaluations & Datasets Track