NodeSyn: Socially Aligned Synthetic Data for AI Evaluation
Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang, Yanzhou Pan, Erin van Liemt, Darlene Neal, Kshitij Pancholi
原文摘要
Recent advancements in generative AI facilitate large-scale synthetic data genera- tion for model evaluation. However, without targeted approaches, these datasets often lack the sociotechnical nuance required for sensitive domains. We introduce NodeSyn, an evidence-grounded methodology that generates socially relevant syn- thetic queries by leveraging a fine-tuned taxonomy generator (TaG) anchored in real-world evidence. Evaluated against four mainstream LLMs, NodeSyn elicited failure rates up to five times higher than human-authored benchmarks. Ablation studies confirm that our granular taxonomic expansion significantly drives these failure rates, while independent validation reveals critical deficiencies in prominent guard models.
cs.LG · cs.CL