Olivier Tieleman, Samuel J. Bell, Ziyi Zhu, Ting Su, Elizabeth Kroll, Thomas D. Hull, Caitlin A. Stamatis
Depression is a leading cause of disability worldwide, yet validated questionnaires such as the Patient Health Questionnaire-9 (PHQ-9) often go uncompleted, yielding sparse, biased data. Users of AI mental health applications often describe their mood, sleep and functioning unprompted, producing a continuous record of symptoms. Here, we predict PHQ-9 scores from these conversations alone. As in many clinical language tasks, labels are scarce and imbalanced. We address both by fine-tuning Qwen 3.5 27B with a regression head and augmenting 3,111 ground-truth labels with pseudo-labels from a frontier reasoning model and iteratively trained intermediate models, expanding to 5,524 users with more balanced scores. On 422 held-out users, the model achieved MAE $= 3.98$, Pearson $r = 0.67$, and AUC $= 0.88$ at the clinical threshold (PHQ-9 $\geq 10$); AUC was $> 0.80$ across severity thresholds from PHQ-9 $\geq 3$ to PHQ-9 $\geq 24$, demonstrating that the model captures depression severity across the full clinical spectrum. These results support passive depression monitoring without self-report.
Context engineering has emerged as a primary lever for improving AI systems without parameter updates. Recent work showing that textual gradients do not function as real gradients motivates treating automatic prompt optimization (APO) as black-box search. We introduce SPO (Stochastic Prompt Optimization), a framework for stochastic search over prompt space, and compare three strategies of increasing sophistication: error-informed random search, a genetic algorithm with evolutionary operators, and SAGE (SPO via Agent-Guided Exploration), a multi-agent pipeline with diagnostic code execution. Across three benchmarks, no single strategy dominates; effectiveness depends on the interaction of landscape structure with error type. We further deploy SAGE on a mental-health chatbot under a continuous optimization paradigm, where it compounds eight cycles of individually-noisy A/B tests into a statistically robust gain in next-day retention. We argue that coupling qualitative diagnosis with quantitative validation is what makes agentic optimization effective for open-ended task-oriented dialogue.
LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be averaged out by increasing the number of scenarios or generations. These biases are often similar in magnitude to the model differences that benchmarks are designed to detect, resulting in unreliable rankings when single-judge evaluations are used. We introduce a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components. Based on this analysis, CyclicJudge, a round-robin assignment of judges to scenarios, is demonstrated to be the optimal strategy for a fixed judge panel and judge-call budget: it cancels the average additive judge offsets while matching the cost of single-judge evaluation. Empirical results on MT-Bench and MindEval validate the effectiveness of CyclicJudge as predicted, across both general-purpose and domain-specific evaluation settings, reducing benchmark score variance and stabilizing the resulting model ranking.
Ziyi Zhu, Olivier Tieleman, Caitlin A. Stamatis, Luka Smyth, Thomas D. Hull, Daniel R. Cahn, Jinghong Chen, Matteo Malgaroli
Realistic user simulation is crucial for training and evaluating multi-turn dialogue systems, yet creating simulators that accurately replicate human behavior remains a significant challenge. An effective simulator must expose the failure modes of the systems under evaluation. This work introduces Direct Iterative Adversarial Learning (DIAL), an adversarial framework that iteratively enhances user simulator realism through a competitive dynamic between a generator (user simulator) and a discriminator. When applied to mental health support, a domain characterized by diverse failure types and a critical dependence on realistic user behavior for failure detection, DIAL restores lexical diversity diminished by supervised fine-tuning and drastically reduces discriminator accuracy. The resulting simulator exhibits a strong correlation between simulated and real failure occurrence rates while maintaining low distributional divergence of failure modes. These findings indicate that DIAL is a promising method for developing realistic user simulators in multi-turn dialogue, facilitating reliable and cost-effective system evaluation prior to deployment.