Shuyan Huang, Alexander Scarlatos, Jaewook Lee, Andrew Lan
Recent advances in large language models (LLMs) have led to the development of AI-powered tutoring systems that provide interactive support via dialogue. To enable these tutoring systems to provide personalized support, it is essential to assess student performance at each turn, motivating knowledge tracing (KT) in dialogue settings. However, existing dialogue-based KT approaches often ignore question difficulty modeling and rely on opaque latent representations from LLMs, hindering accurate and interpretable prediction. In this work, we propose an interpretable difficulty-aware conversational KT framework built upon LLMs, which explicitly models students' abilities and the difficulty of tutor-posed tasks at each turn. The framework incorporates the original textual question and the next tutor-posed task to estimate the student's knowledge state and the difficulty of the upcoming turn. Furthermore, it integrates Item Response Theory to map LLM's outputs into student ability and question difficulty parameters, enabling interpretable prediction of student performance grounded in cognitive theories of learning. We evaluate the framework on two tutor-student dialogue datasets. Both quantitative and qualitative results show that our framework outperforms existing KT baselines, meanwhile generating interpretable outputs consistent with cognitive theory.
As human developers increasingly collaborate with large language model (LLM)-powered coding agents, adapting to individual developer preferences, potentially via agent skills, may offer way to reduce repeated guidance and improve collaboration efficiency. However, existing approaches primarily focus on building skills for each task rather for each human developer. In this work, we propose a framework that combines rule-based bootstrapping with LLM-based refinement to distill developer-specific skills from their interactions with coding agents. We further develop an interactive replay framework with an LLM-based developer simulator for evaluation. We conduct experiments on a collection of replayable real-world developer-agent interaction sessions. We find that overall, task-specific skills are the most effective; However, personalized skills become more effective when developer preferences are manifested with enough frequency, especially when developers work on tasks that are highly related. These findings provide empirical insights suggest that i) broadly transferable procedural knowledge can be more robust than developer-specific preference signals and ii) effective personalization requires mining from rich user preference signals.