将 Hugging Face 模型编目到软件工程活动:自动化与发现
Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findings
Alexandra González, Xavier Franch, David Lo, Silverio Martínez-Fernández
AI 速读 基于摘要
作者构建了覆盖146项软件工程任务的分类体系,并通过仓库挖掘为 Hugging Face 上的预训练模型建立面向软件工程的目录,最终得到5,027个软件工程预训练模型。
cs.SE · cs.LG
查看原文摘要
Context: Open-source Pre-Trained Models (PTMs) provide extensive resources for various Machine Learning (ML) tasks, yet these resources lack a classification tailored to Software Engineering (SE). Objective: To address this gap, we derive a taxonomy encompassing 146 SE tasks and apply an SE-oriented classification to PTMs in a popular open-source ML repository, Hugging Face (HF). Method: Our repository mining study followed a five-phase pipeline: (i) identification of SE tasks from the literature; (ii) collection of PTM data from the HF API, including model card descriptions and metadata, and the abstracts of the associated arXiv papers; (iii) text processing to ensure consistency; (iv) a two-phase validation of SE relevance, involving humans and LLM assistance, supported by five pilot studies with human annotators and a generalization test; and (v) data analysis. This process yielded a curated catalogue of 5,027 SE PTMs. Results: We find that most SE PTMs target code generation and coding, emphasizing implementation over early or late development stages. In terms of ML tasks, text generation dominates within SE PTMs. Notably, the number of SE PTMs has increased markedly since 2023 Q2, while benchmarking remains limited: only 5.55\% of all SE PTMs in our catalogue provide evaluation results. Conclusions: Our catalogue reveals documentation and transparency gaps, highlights SE activity imbalances, and supports automated SE scenarios, such as the selection of suitable PTMs.