Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences
Boyuan Deng, Shuyi Fan, Hongyang Zhang, Xinhong Xie
原文摘要
Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. Once the evidence is prepared and the calculation is written as code, this choice reduces to one semantic judgment among listed candidates. We evaluate Jev on this step, using a harness that follows its documented guidance. The study compares twelve model configurations on twenty Choices in ten scientific cases built from published sources, each repeated five times. We measure semantic selections, downstream outputs and final claim labels separately. Jev tied with five other configurations at complete semantic correctness and achieved the lowest observed median latency and known mean cost among successful responses. On one question, whether samples shared a culture, three comparison models made seven wrong selections that changed downstream counts while preserving the correct final label. These results support using Jev to select relations when the evidence and candidates are prepared in advance, and show that evaluating it requires checking the relations and quantities that a workflow will reuse.
cs.CL · cs.AI