← 研踪 PaperTrailACL Anthology · ACL 2026:acl:2026.acl-demo.19 · 记录 #1
DialogGuard: Multi-Agent Psychosocial Safety Evaluation Interface of Sensitive LLM Responses
来源日期（仅到月份） 2026-07
来源记录日期 2026-07
雷达观测 2026-11-01 22:08
摘要
LLM-based agents are increasingly deployed for mental-health support and crisis counselling, yet recent evaluations reveal that commercial therapy chatbots respond appropriately only about half the time in clinical scenarios.Clinicians and safety engineers are called upon to audit these systems, but existing tools do not surface the subtler psychosocial harms (manipulation, discrimination, psychological distress) nor produce the explainable rationales that practitioners need.We present DialogGuard, an open-source system that lets practitioners inspect, stress-test, and create audit trails for prompted LLM agents across five psychosocial safety dimensions.The system wraps around arbitrary generative models through four LLM-as-a-judge pipelines (single-agent scoring, dual-agent correction, multi-agent debate, and majority voting), each grounded in shared three-level rubrics.Through its web interface, practitioners evaluate agents in two modes (Live Chat and Manual Input) and review per-dimension risk scores with natural-language rationales.Experiments on PKU-SafeRLHF show that dual-agent correction provides the best accuracy-robustness trade-off, and a formative study with 12 practitioners confirms that the system supports prompt auditing, safety inspection, and supervisory decision-making.Code and demo: https://github.com/lhannnn/dialogguard-web.
论文发现出处
这些页面提供论文线索，不代表同行评审或质量认证。
观测到的版本
平台记录号代表本站观测到的元数据变化，不是作者修订号。来源日期仅到月份或年份时，按该月或该年首日存档。
记录 #1 · 2026-07 · 最新观测版本
DialogGuard: Multi-Agent Psychosocial Safety Evaluation Interface of Sensitive LLM Responses
LLM-based agents are increasingly deployed for mental-health support and crisis counselling, yet recent evaluations reveal that commercial therapy chatbots respond appropriately only about half the time in clinical scenarios.Clinicians and safety engineers are called upon to audit these systems, but existing tools do not surface the subtler psychosocial harms (manipulation, discrimination, psychological distress) nor produce the explainable rationales that practitioners need.We present DialogGuard, an open-source system that lets practitioners inspect, stress-test, and create audit trails for prompted LLM agents across five psychosocial safety dimensions.The system wraps around arbitrary generative models through four LLM-as-a-judge pipelines (single-agent scoring, dual-agent correction, multi-agent debate, and majority voting), each grounded in shared three-level rubrics.Through its web interface, practitioners evaluate agents in two modes (Live Chat and Manual Input) and review per-dimension risk scores with natural-language rationales.Experiments on PKU-SafeRLHF show that dual-agent correction provides the best accuracy-robustness trade-off, and a formative study with 12 practitioners confirms that the system supports prompt auditing, safety inspection, and supervisory decision-making.Code and demo: https://github.com/lhannnn/dialogguard-web.
查看此版本解读 →查看此版本原文 ↗