GUEST:使用与评估GenAI工具支持系统文献综述的初步指南
GUEST: Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews
Barbara Kitchenham, Sebastián Pizard, Lech Madeyski, Ronnie de Souza Santos, Martin Shepperd, David Budgen
AI 速览:作者通过快速综述与经验反思提出GUEST框架,含51条建议,指导在系统文献综述(SLR)中规划、执行和透明报告GenAI辅助工作及其评估。
Context: Generative AI (GenAI) and Large Language Models (LLMs) are increasingly used for academic tasks in software engineering and beyond, including systematic literature reviews (SLRs). However, whilst their capabilities are rapidly developing, there is no guarantee they can meet the rigour, reliability, and transparency that SLRs require. Objectives: To support researchers intending to conduct SLRs using GenAI, or those conducting empirical studies to evaluate how well GenAI supports SLR tasks. Methods: First, we conducted a rapid review to identify studies that propose guidelines for evaluating and using GenAI and LLMs to support SLRs. Second, we drew on thought experiments, relevant guidance from the literature, and our own experience from conducting SLRs and evaluating tools, to develop recommendations for how to use and assess GenAI in the context of SLRs. Results: We classify methodological issues for the roles of evaluation and reviewing into four categories spanning SLR practice adjustments, general secondary-study standards, GenAI evaluation practices and core empirical rigour. Building on this, we introduce GUEST (GenAI Use and Evaluation in SLR Tasks) which is a comprehensive framework of 51 recommendations designed to guide researchers through planning, executing and transparently reporting GenAI-assisted literature reviews and validation studies. Conclusion: Current empirical evidence indicates that GenAI cannot be relied upon to perform unsupervised systematic reviews so requires human oversight. Nevertheless, it can offer cost-effective assistance for repetitive screening tasks and secondary validation. By standardizing how these tools are deployed and evaluated, the GUEST framework provides a necessary methodological baseline for trustworthy GenAI-assisted research in software engineering.