Nishtha Agarwal, Adam Visokay, Astha Wahi, Kwabena Owusu Aninkora, Amy Gu, Tyler McCormick, Li Liu
Systematic reviews synthesize biomedical evidence to inform clinical and public-health decision-making, but require domain experts to manually screen thousands of studies against detailed eligibility criteria. We investigate whether language models can assist with this bottleneck by learning from expert feedback on title and abstract screening decisions and model-generated reasoning. We introduce the Context-Aware Reasoning Framework (CARF), a human-in-the-loop framework that uses expert feedback on model-generated reasoning to adapt language models to context-dependent screening tasks. We evaluate CARF on systematic reviews of verbal autopsy and cause-of-death ascertainment methods used in global public health mortality surveillance. CARF shows competitive screening performance compared to evaluated models while retaining high sensitivity, and is more inclined to retain ambiguous, context-dependent studies for human review. Together, these findings suggest that expert-corrected reasoning is a promising supervision signal for context-dependent scientific screening, while retaining a role for human oversight.
Adam Visokay, Soumyakanti Pan, Molly Griston, Polina Iasakova, Mark Nepf, Denis Peskoff, Riya Sinha, Jikai Sun 等 13 位作者
In less than five years, generative artificial intelligence (AI) has dramatically reshaped the public-facing web. Using a sample of more than 5 million crawlable webpages, we demonstrate that the share containing AI-generated text has increased precipitously, but unevenly. Our analyses reveal substantial differences in AI composition across language and domain categories (e.g., .edu, .com, or .gov). A higher fraction of English-language webpages contain AI-generated text compared to pages in other languages, and prevalence is substantially lower on .gov and .edu domains compared to .com. These distinctions persist even after accounting for compositional effects. We also find heterogeneity based on a page's position in the connectivity structure of the web. AI-generated text is most common and has increased most rapidly in pages that are the least central in the graph, which drives much of the overall growth in AI-generated text. A slower increase has taken place for more central pages, with the most central tier emerging as the least likely to contain AI-generated text. Methodologically, we use multiple AI detection tools, and generate estimates using a Bayesian hierarchical model that incorporates corrections for the detectors' statistical properties (both false positives and false negatives) and allows for different detector performance across groups, while propagating uncertainty in detector error rates into prevalence estimates. Our work suggests that AI is a prominent, but varied, contributor to public information and internet-based communication.