Persian medical natural language processing remains limited by the scarcity of domain-specific resources and insufficient public characterization of their coverage and quality. We present Gaokerena, an openly released suite of Persian medical resources comprising an 85.5-million-word corpus of 101,954 web articles; MF3QA, a 24,000-pair patient-doctor question-answering dataset; 1,878 Iranian medical examination questions (KOPP); 1,089 manually translated medical MMLU questions (FA_MED_MMLU); and approximately 18,000 machine-translated MedMCQA training items filtered by two language-model evaluators. We characterize the resources' composition, orthographic variation, redundancy, topical coverage, privacy indicators, answer-key balance, extraction artefacts, and translation limitations. As application case studies, we adapt Aya-Expanse-8B into Gaokerena-V, trained on the corpus and MF3QA, and Gaokerena-R, developed through preference-based reasoning distillation. On translated medical MMLU, Gaokerena-V achieves 49.31% accuracy under direct prompting, compared with 46.65% for Aya-Expanse-8B; the paired exact McNemar test is statistically inconclusive (p = 0.0502). Gaokerena-R reaches 52.80% using five-sample self-consistency and a verifier, but this higher-compute setting is not directly comparable. Under matched five-sample chain-of-thought self-consistency without a verifier, Gaokerena-R scores 48.58%, versus 46.92% for Aya-Expanse-8B, a non-significant difference (p = 0.190). English evaluation also reveals a substantial performance gap relative to Persian. To our knowledge, Gaokerena provides the first openly released and systematically characterized suite of Persian medical resources alongside generative models built on them. These protocol-specific results do not establish an isolated training advantage or clinical readiness.
Are These Programmers Ripe Enough to Eat? Evaluating How LLMs Handle Lexical Knowledge Conflicts in Translation
Dongxu Wu, Sara Court, Micha Elsner
AI 速读 基于摘要
作者构建受控汉英翻译任务,考察大语言模型在提示提供的词义与模型先验词汇知识冲突时如何取舍。
cs.CL · cs.LG
查看原文摘要
External lexical resources can help large language models (LLMs) translate texts in low-resource languages and specialized domains, but these resources may conflict with models' prior lexical knowledge. To isolate this conflict, we construct a controlled Mandarin-to-English translation task in which familiar source words are assigned new meanings within and across semantic categories. We treat these prompt-supplied meanings as contextual knowledge (CK) and systematically evaluate the effects of word frequency, sentence context, and prompt framing on the translated outputs. Across nine models, CK reliance is greater for less frequent source words and in sentence contexts that support the supplied meanings. Prompt framing can reduce CK reliance, but seven of nine models favor CK even when explicitly instructed to ignore the supplied meanings. Within model families, larger models are more likely than smaller models to utilize CK in their outputs. Three follow-up experiments further show that modal weakening lowers CK reliance, the effect of repetition depends on prompt layout, and the effect of reasoning varies across models. Overall, our controlled study reveals how CK reliance varies across conditions and models, motivating new methods to detect and evaluate lexical conflicts and guide models' lexical choices in LLM-based translation.
We present the Whiel (a contraction of Whi(le-R)el) system for Lean-certified reasoning about imperative relational programs. The system can encode containment and equivalence problems for Datalog, its extensions with negation, and the core fragment of graph query languages. It can also represent optimized or incremental evaluation algorithms that are not expressible in the original query language, as well as first-order specifications of integrity constraints, schema mappings, and views arising in data integration, exchange, and sharing. Whiel's program states are database instances, and its imperative structure enables compositional Hoare-style reasoning with first-order relational specifications, using prophecy variables and leveled invariants to support inductive arguments. Whiel supports neurosymbolic discovery of loop invariants whose verification conditions can be checked using the Vampire first-order theorem prover. Counterexamples are fed back into the neurosymbolic pipeline, while proposed solutions are translated into Lean proof certificates. To validate the Whiel system, we evaluate its performance on a curated benchmark suite. In summary, we envision the Whiel system as a tool for tackling mathematically demanding, typically undecidable problems involving complex query and constraint languages, with a high degree of trust in its output.
Safety alignment for Large Language Models is heavily focused on English with limited multilingual representation, particularly damaging low-resource language performance. Related open training datasets are often limited in size or do not follow region-localized harmful topics to broaden the scope of alignment beyond cultures. We propose a scalable template-based augmentation pipeline to produce massive multilingual refusal and benign corpora, extracting safe templates from harmful prompts, translating them and then filling in the slots with culturally authentic entities, both harmful and benign. As a result, we release MSafeTemp, a validated corpus spanning 60 languages useful for multilingual safety alignment and evaluation. Using the dataset, we show that English-only evaluation misjudges multilingual safety: for several models, attacks that fail in English succeed once translated, while culturally localized variants expose further failures that translation alone misses. We also confirm that augmenting safety training data through templates allows for higher diversity and quality of alignment, resulting in improved safety performance beyond English. MSafeTemp and the respective pipeline are aimed at facilitating research and development of safer and more responsible models for less-represented cultures.