Jenny T. Liang, Mihika Bairathi, Wayne Chi, Ameet Talwalkar, Nishant Subramani, Valerie Chen
Imperfections in AI-generated code require that software developers modify generations through manual code edits. However, little is known about how developers edit AI-generated code, and to what extent LLMs can infer implicit developer intent from this data, given the lack of high-quality, realistic code editing data. This information is essential for designing and building usable AI-developer interactions. To address this gap, we introduce DECODE (Developer Edits of Code Dataset), a dataset of 53.6K real-world in-IDE code edits of AI-generated code sourced from 1K+ developers. First, we use DECODE to analyze when, why, and how AI-generated code is edited. Developers invest little effort in editing code and abandon AI-generated code quickly, thereby motivating the need for LLMs to understand developer intents through their edits. Therefore, we next use DECODE to understand to what extent LLMs could infer developer edits. Predicting code edits is a challenging task for both frontier and fine-tuned models. Frontier models generate edits that best reflect developer intents, but are prone to over-editing and reverting developer-written edits. Fine-tuning LLMs can improve their ability to predict edits that are more similar to developers' as well as better reflect developer intent, but do not fully bridge the gap to frontier models. Together, our results demonstrate the unique insights on developer and LLM behavior that can be obtained through studying realistic code edit data to promote more better developer-AI interactions.
The development of the collaborative critique (CC) usability inspection method was driven by the need for systems, ranging from the simple to the complex, to lessen the workload on their users by being better collaborative partners. In the collaborative view of system-user interaction, the role of the system is expanded to that of a partner that must help its users achieve their goals. This perspective shifts the dynamic from focusing on the user's ability to make effective and efficient use of the system to the system's usefulness within a given task, user, and interaction context, thereby reducing the user's cognitive and physical efforts. The CC method consists of a walkthrough evaluation in which the questions answered at each step of the interaction assess the collaborative strength of the system. This method is further distinguished by the inclusion of questions that access the behavior of the system in error situations, which are known to be particularly burdensome to users of complex systems. In this paper, we provide a detailed description of the CC method and its application and demonstrate its usefulness in evaluating an interface. The results of a laboratory evaluation show that the method performed well in identifying the problems experienced by users across all severity levels and predicted almost all of the problems associated with error situations.
Chenyi Li, Haoyu Hu, Raja Marjieh, Katherine M Collins, Max Kreminski, Ilia Sucholutsky, Dalton Conley, Nori Jacoby
As AI agents increasingly participate in creative teamwork, writing emerges from hybrid human-AI teams. Although AI can improve the creativity of individual outputs, these gains may come at the cost of homogenization. What is the best role for human intervention? We introduce a transmission-chain paradigm inspired by cognitive models of writing, with four stages: Plan, Write, Review, and Revise. Each stage receives the previous output and passes its own forward. Using data from 333 participants across 427 stories, we compared creativity, diversity, and enjoyment across human, AI, and hybrid chains. Consistent with prior work, all-AI chains produced more creative but less diverse stories than all-human chains. Introducing a single human into an otherwise all-AI chain increased diversity depending on role. Human planning provided the best balance: it preserved AI-level creativity, increased diversity that persisted through downstream AI stages, and maintained enjoyment. We discuss implications for designing human-in-the-loop creative pipelines.
Artificial intelligence (AI) technology has gradually been integrated into automated vehicles. However, how this technology could systematically support takeover transitions in SAE Level 3 automated vehicles remains unknown, even though this level may remain a major form of driving automation in the near future. To address this, we implemented the three AI roles proposed by the triadic Human-AI collaboration framework (Advisor, Co-pilot, and Guardian) as potential authority-ability configurations for AI agents within an automated system. We then examined their interactions with explanation type (why and action) and directional semantics (ipsilateral and contralateral) in an automated driving simulator study with 24 participants. Results showed that, across the three AI roles, Advisor had the shortest decision-making time, followed by Guardian and Co-pilot. Advisor and Guardian were associated with higher decision-making accuracy and higher satisfaction ratings than Co-pilot. Guardian was associated with the lowest workload, followed by Advisor and Co-pilot. The effect of explanation type on decision-making time depended on AI role. Action explanations were faster under Advisor, whereas why explanations were faster under Co-pilot. For directional semantics, ipsilateral semantics were associated with shorter decision-making time than contralateral semantics. Overall, these findings provide preliminary guidance for designing AI-role authority-ability configurations and their associated speech-based takeover messages.
Haiwen Li, Crystal Qian, Ben Usman, Stephanie Chan, Michiel Bakker, Srivignesh Rajendran, Avneesh Sud, Thomas Leung 等 12 位作者
As AI systems increasingly mediate human communication, identifying conditions under which they augment collective sense-making is a critical challenge. Through a randomized experiment (N=1,173 U.S. participants across 391 small-group deliberations), we investigate whether AI-guided self-reflection improves subsequent deliberative discourse. We show that engaging with a non-directive AI chatbot instructed to structure self-reflection prior to group deliberation yields significantly higher deliberation quality (measured by the Discourse Quality Index, DQI) compared to a baseline of no self-reflection (Cohen's d=0.31) and a condition of unassisted self-reflection (d=0.27), and the positive impact is most pronounced in groups with disagreement. This improvement may be driven by more sophisticated argument justifications, greater consensus-building effort, and increased respect for counterarguments. Furthermore, after self-reflection, participants in the AI condition reported higher empathy toward dissenting peers and a deeper understanding of opposing reasoning. Beyond observed behavior, our findings are empirically relevant to open debates between political philosophy and psychology surrounding self-reflection, demonstrating that interactive AI could transform solitary thinking into productive internal reflection. Together, these results demonstrate that even simple AI-based reflective scaffolds can foster more productive group dynamics, opening new avenues for leveraging AI to augment collective intelligence.
Greta Warren, Christine M. Lehane, Jessica Heal, Sila Eser, Denise M. Leto, Isabelle Augenstein
Generative AI systems increasingly mediate how people seek information, yet communicating AI factuality to laypeople remains challenging. Through a mixed-methods investigation involving interviews (N=8), a survey (N=1,000), and an experiment (N=5,151), we examine how people evaluate AI-generated information for gardening, mathematical, and medical queries, and how factuality indicators (confidence and sources) shape perceptions. Our interviews and survey show people selectively verify AI responses using external and AI-provided evidence, moderated by contextual factors. Although confidence-indicators affected trust and accuracy perceptions, people hold diverse, ambiguous interpretations of AI confidence. Sources were viewed as crucial, yet users raised concerns about source reliability and verification effort. Our experiment showed that confidence and source-based indicators influenced initial perceptions, but effects disappeared once participants learned the response's correctness. Medical queries elicited skepticism regardless of indicator. Findings motivate helping users interpret model confidence and integrating source information into the interface to support calibrated trust in AI-supported information-seeking.
Abhinav Goel, Agostino Capponi, Alfio Gliozzo, Chaitya Shah
AI agents can now write software from a plain-English description, but code that runs is not the same as code that does what was asked. This gap matters most for smart contracts, the programs behind Web3 applications that move money on a blockchain and cannot be fixed once deployed. We ask how to check whether an AI-written contract behaves as it should, and where the requested behavior gets lost. We design an evaluation built on execution rather than opinion. We replay the same transactions on an AI-written contract and on the original it should reproduce and measure how often they agree. We test every grading tool on planted mistakes, compare against an answer key written without the description, and audit our own scorer until known wrong answers cannot pass. Common methods, such as AI judges and standard analysis tools, miss obvious mistakes, and scoring choices matter too. Checking only the functions every system includes hid about half the difference between pipelines. We then propose an elicitation-first architecture. Before writing code, the agent asks the client a short checklist of questions, checks the contract against an independently written executable answer key, and returns disagreements to the client instead of silently rewriting code. Asking first matters far more than reviewing or patching afterwards, across three AI models, while automated security fixes often break requested behavior. A few well-chosen questions recover much of what a short description omits, and their value can be estimated without the original contract. To our knowledge, this is the first study in smart contract synthesis to gather missing requirements before generation and check the result against an independently written executable model. The lessons extend to any setting where AI agents turn requests into programs. We release our tools, 600 tasks and all results.
Stephen Prochaska, Mert Bayar, Ashlyn B. Aske, Joseph Schafer, Kate Starbird
Recent elections in the United States have seen online rumors become a regular feature of election discourse as voters and election officials work to navigate a sea of digitally spread claims and propaganda. For the uninitiated, these rumors can feel overwhelming and random, sowing doubt about the integrity and legitimacy of recent elections. In this paper, we reviewed and qualitatively coded a large dataset of 590 election rumors collected during the the 2020 and 2022 national elections. We found that there are tropes that recur time and time again in election rumors, highlighting how these rumors are not as random as they may first appear. We make two primary contributions in this work: first, we introduce a hierarchical framework that structures observed tropes in three tiers (Meta-category, Meso-trope, and Micro-trope). Second, combining the framework with our coded data, we identify a visible grammar of election rumors, highlighting how in the majority of cases a rumor contains an Actor, and Object, and an Action, with optional grammatical components of Suggested Remedies and Scene & Setting. The framework and grammar can be used by other researchers to act as the underlying information architecture of analyses examining similar rumors, enabling the rapid identification, analysis, and prediction of election rumors.
Chenyi Huang, Arun Josephraj Arokiaraj, Bin Wu, Sahan Bulathwela
Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool. Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be. Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps. The annotator writes their own labelling codebook; a frontier model proposes a starting set from a sample of their own traces, answers first, and is asked exactly the questions the human is asked, making annotation a joint human-AI process. Annotrace measures agreement between any two annotators, human or AI. On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at $κ= 0.82$.
Sanjida Islam Era, Tanjeem Azwad Zaman, Ishika Tarin Ime, Ashequl Islam Tofa, Farzin Awsaf Zaman, A. B. M. Alim al Islam
Examining whether technical accessibility measures align with users' usability experiences across web systems in Bangladesh is of utmost significance, given the country's rapidly expanding digital ecosystem. We, therefore, evaluated 207 web systems using WAVE, AChecks, and Squidler and surveyed 103 young adult users. Automated evaluation revealed mixed patterns across government and non-government sectors, with neither consistently performing better across accessibility, functionality, web practices, and performance. In contrast, participants clearly favored non-government platforms for navigation, usability, clarity, and cross-device experience. Qualitative analysis identified three barriers to practical access: system instability disrupting task completion, interaction and navigational friction, and socio-technical barriers involving information access, service continuity, and trust. These findings show that automated accessibility measures capture only part of users' experiences. Accessible web services therefore require attention beyond detectable accessibility problems to reliable task progression, understandable interaction, information access, and continuity across the broader service process.
Images co-edited with AI are often labeled to disclose AI use. However, creators feel these labels do not sufficiently describe their contributions to the editing process. Editing histories preserve requests, suggestions, and revisions, but how do creators use these logs to judge contribution? We investigated this with Roop, a mixed-initiative design probe that links image regions to editing histories (spatial provenance) and summarizes interactions in a heat map. Twenty participants edited images, annotated human and AI contributions, then reviewed the spatial provenance heat map. Participants credited themselves for directing changes while crediting AI for determining visual details. They judged contribution partly by whether their interactions achieved their intentions. Reviewing provenance helped participants remember interactions without always changing whom they credited. These findings motivate provenance systems that preserve enough context for creators to explain how interactions shaped their work, alongside the record of what happened.
Mingwei S. G. Li, Susie S. Y. Li, Isabel Moreira de Oliveira, Ryan P. Adams, Remco Chang
Generative models support content creation across text, images, videos, and design disciplines, yet their interfaces rely primarily on natural language. While effective for specifying high-level intent, language can be inefficient and imprecise for fine-grained control, iterative refinement, and open-ended exploration. We present PolyGandr, a visual interface for exploring relationships between latent representations and generated outputs, complementing natural-language model steering. PolyGandr formulates generative control as high-dimensional navigation, enabling users to define starting points, steering directions, and step sizes through a unified map-and-mixing-palette interface. Combining global exploration with local refinement supports both broad design-space navigation and fine-grained output manipulation without repeatedly translating visual intent into prompts. We evaluate PolyGandr across three use cases: StyleGAN2 face generation, Stable Diffusion chair design, and rotational kirigami engineering. A formative study indicates that PolyGandr complements existing workflows by supporting real-time editing while simultaneously providing an overview and means to explore the broader design space.
Collaborative virtual reality (VR) produces rich multimodal traces of team behavior, yet practical methods for characterizing time-varying team communication remain limited. We present time-dynamic semantic-activity entanglement (TSAE), inspired by the entanglement concept introduced by Gloor et al. (2022), to describe communication-based team synchronization within discrete time windows. TSAE is modeled by combining relative speaking activity with contextual semantic alignment for participant pairs, which is aggregated into participant-level trajectories and summarized through the Gini coefficient to understand the distribution of team-level communication synchrony. This approach is employed with German-language transcripts from fifteen collaborative VR gameplay sessions. Transcript utterances are represented using a long-context embedding model with a late-chunking strategy and analyzed within fixed game-time windows. A representative session illustrates shifting dyadic alignment and low-to-moderate participant contributions, with sporadic periods of high entanglement. Across fifteen game sessions, the metrics indicate complementary insights, offering a qualitative contrast to the core-periphery pattern reported in the foundational work. Hence, these descriptive findings support the study of communication as a dynamic process and provide a mathematically grounded approach to analyze its evolving distribution in shared interactive VR spaces.
Ethan Eddy, Evan Campbell, Scott Bateman, Erik Scheme
The recognition of dynamic gestures via discrete myoelectric control is becoming an attractive input modality for event-based human-machine interaction. While recent work suggests that discrete cross-user models are possible (i.e., no calibration data is required from the end user), few have evaluated techniques to optimize these systems and their generalizability to new randomly prompted data simulating real-world discrete control is relatively unknown. Using the 612-user EMG-EPN dataset, this work benchmarks five temporal deep-learning approaches, evaluates the tradeoff between performance and model complexity, and compares hand-crafted and machine-learned feature extraction techniques. The CNN-GRU achieved the best accuracy of 93% when tested on 184 previously unseen users. These models were then evaluated on a different, more realistic, unsegmented dataset, wherein users were randomly prompted to elicit discrete gestures during a continuous data collection. This dataset was recorded five years apart and in a separate country from the data used to train the original model. Results show that the cross-user results scale to this new dataset (accuracies of 96%) even though no feedback was given to users during the elicitation of gestures. These findings highlight the exciting potential of discrete zero-shot cross-user myoelectric control and offer valuable insights for the continued advancement of this promising technology.
Zheng Lian, Fan Zhang, Lan Chen, Yazhou Zhang, Rui Liu, Jinyang Wu, Haoyu Chen, Xiaobai Li 等 11 位作者
Open-Vocabulary Multimodal Emotion Recognition (OV-MER) aims to predict emotions without being constrained by predefined label spaces, thereby enabling fine-grained emotion understanding. Unlike traditional discriminative methods, OV-MER leverages generative models to capture the full spectrum of emotions and employs emotion wheels (EWs) for metric calculation. Previous approaches primarily rely on token-level loss during training. However, this objective is misaligned with the metrics used in OV-MER, and these metrics cannot be directly optimized via gradient backpropagation. To address this limitation, we turn our attention to reinforcement learning, as this strategy can optimize non-differentiable objectives. We term this framework AffectGPT-RL. Furthermore, we conduct extensive experiments to elucidate the role of reinforcement learning in this task, revealing the necessity of the reasoning process, the impact of different rewards, and the generalizability to other emotion tasks such as sentiment analysis and basic emotion recognition. Experimental results demonstrate that AffectGPT-RL yields significant performance improvements on OV-MER. Beyond this task, we also achieve remarkable performance gains on basic emotion recognition, attaining state-of-the-art results on MER-UniBench. To the best of our knowledge, this is the pioneering work exploring the role of reinforcement learning in OV-MER, providing valuable guidance for subsequent researchers. Our code is provided at: https://github.com/zeroQiaoba/AffectGPT/tree/master/AffectGPT-RL.
Simon WS Fischer, Hanna Schraffenberger, Miranda L. van Hooff, Serge Thill, Pim Haselager
A major challenge in AI-supported decision-making is achieving appropriate reliance on AI. To reduce overreliance on AI, approaches for cognitive friction have been proposed that promote reflection and critical thinking. In this paper, we investigate how two such approaches are perceived by medical experts in a real-world decision task. We replicated an existing decision support system used in clinical practice and extended it with outputting reflective questions and what-if scenarios. We gathered clinicians' (n=7) feedback on this prototype through in-situ interviews. While the questions were perceived as unhelpful for reflective thinking in itself, their potential was seen in serving as reminders to consider relevant information, e.g., when attention wanes. Inspecting what-if scenarios was found useful for considering alternative outcomes and informing discussions with patients. Our participants regarded the prototype as a promising training tool for novice clinicians. From the clinicians' feedback, we make recommendations for designing friction in human-AI interactions.
Dong Shu, Ziyang Guo, Negar Kamali, Matthew Groh, Jessica Hullman
Explanations of LLM outputs are considered an important way to make model behavior legible, helping human users make good decisions on downstream tasks. However, default training approaches for improving explanations target model task performance or human preference rather than human decision quality. To bridge this gap, we introduce \textbf{HD-RL}, a reinforcement learning framework for optimizing LLM explanations for human decision making. Our approach calibrates a human decision simulator on observed human decisions, applying a proxy reward model to optimize the LLM to generate explanations that help people to make good decisions. We apply this framework to AI-generated text detection. In our initial data collection phase, we observed that while LLMs possess strong autonomous detection capabilities, the explanations they generate for users yield near-random human accuracy (51.3\%). By applying the HD-RL framework to the open-source Qwen 3.6 model, we generate optimized explanations that improve human performance, achieving a GLMM-estimated human accuracy of 63.7\% in a randomized between-subjects experiment with repeated measures and outperforming RLVR, RLAIF, and other baselines by over 8\%.
Chengwen Zhang, Jianyang Wang, Chang Liu, Yukun Wang, Yuan Xu, Zhoutong Ye, Jie Cai, Chun Yu 等 9 位作者
Human-robot collaboration requires partners to coordinate interdependent contributions toward a shared goal. Joint Action theory provides concepts for understanding this coordination, but how these concepts are instantiated and combined across empirical research remains fragmented. We conducted a systematic review following PRISMA 2020, identifying 83 empirical papers published between 2021 and 2026. Through theory-driven framework analysis, we organized the literature into nine constructs across three capacities: Representation, Prediction, and Integration. We analyze how these constructs are distributed and co-occur across studies, revealing where empirical evidence concentrates and where it is absent. By mapping empirical HRC research onto a unified Joint Action framework, this review offers a coherent overview of HRC studies, identifies gaps in current evidence, and provides guidelines for future research.
Emmanuel Ajaero, Pascal Spiegler, George Rakovich, Marta Kersten-Oertal, Yiming Xiao
Video-Assisted Thoracoscopic Surgery (VATS) is a minimally invasive procedure, where precise surgical port placement is vital, yet it relies heavily on surgeon experience and mental mapping of pre-operative imaging in the operating room. To address this challenge, we present an accessible, interactive mobile augmented reality (AR) framework, HILMARTS, to enable accurate and efficient pre-operative virtual model-to-patient registration for AR-guided surgical navigation. Our system runs on a commercially available LiDAR-equipped iPad and supports a user-in-the-loop workflow through a two-stage coarse-to-fine registration process. First, the user performs a manual alignment using the AR view on the iPad. This initial maneuver is then refined through automatic point-cloud registration leveraging the LiDAR sensor. We further introduce a novel occupancy-grid-based Surface Coverage Score to provide real-time, actionable feedback to the operator on the geometric completeness of LiDAR data capture for reliable point-cloud-based alignment. In a phantom study with 16 participants, the system achieved a 3.31+/-1.26mm registration accuracy in ~4 min with a mean System Usability Scale (SUS) of 84.1. These findings highlight the system's accuracy, efficiency, robustness, and potential clinical usability, and could support its applicability to other AR-based surgical navigation tasks.
Zhi Zheng, Binglin Liu, Weihao Chen, Lan Luo, Yu Mei, Jie Cai, Chun Yu, Yuanchun Shi
In early mathematics, children learn abstract concepts through concrete representations, such as manipulating objects, drawing, and speaking. Learning with peers can enrich this process by exposing children to diverse strategies and enabling them to exchange ideas through interaction. We realized this potential with virtual peers in Peers4U, a multimodal learning system to enhance children's mathematical conceptual understanding. Virtual peers driven by LLMs are personalized to the learner's cognitive level and assigned multimodal solutions with inspectable processes. During learning, peers engage in mixed-initiative interactions by responding, initiating topics, and advancing discussions along a pedagogical progression. A user study with 22 children and two experts showed our learner-centered topic orchestration significantly outperformed a persona-centered baseline, and children spontaneously authored new artifacts to demonstrate inspired ideas or share their own strategies, producing significantly more solutions when grouped with more Leading Peers. We contribute design insights for multimodal, learner-centered virtual peer systems.
Ekaterina Soroka, Niels Seidel, Marcus Specht, Kamila Misiejuk
Although learning design (LD) plays a central role in shaping online courses, large-scale empirical evidence on how courses are actually structured remains scarce, and existing analytics work has characterised courses through student engagement with specific learning activities rather than through indicators of the design itself. This study addresses both gaps through an analysis of learning management system (LMS) data from 606 courses at a distance university. Using LMS data, we first applied dimensionality reduction and clustering to a structural variable set, from which three distinct LD types emerged empirically: Compact, Expansive, and Interactive and statistically characterised how their design elements differ across disciplines, activity load, and interfaces. Building on this typology, we derived three LD indicators, activity-mode balance, structural clarity, and analytics readiness, each combining multiple LMS-derived variables into higher-order constructs that target a different aspect of LD. All three indicators differentiated significantly across the LD types and added pedagogically meaningful dimensions for interpretation. These contributions advance the operationalisation of LD in technology-enhanced learning research by demonstrating that higher-order LD indicators can be derived directly from routinely collected LMS data, and offer empirical insight into how LD manifests at scale within a large distance university.
Science fiction often features characters or objects that undergo radical changes in form while still being recognizable as the same entity. For example, a vehicle might unfold into a robot, or one body might flow or shape-shift into another. We use such transformations as a reflective lens to study similar shape-shifting in visualization, where one visual representation transforms into a completely different one. For example, a node-link diagram might transform into a parallel coordinates plot as a user's task changes from understanding relationships to identifying correlations. Based on four instances of science-fiction transformation, our paper highlights two key questions: First, why is a transformation into a target form needed, or what is the purpose? Second, what remains recognizable across the transformation from source to target, or what is the correspondence? Through four hypothetical transitions between visual representations of a multifaceted dataset, we demonstrate that purpose is also essential to visualization transformations. Furthermore, we show that the question of correspondence results in different levels of complexity in terms of designing and understanding transitions. Depending on whether correspondence is 1:1, 1:N, or N:1, visualization transformations may be direct and easy to understand, or they may require more elaborate transition constructs, implying increased cognitive effort for interpretation. Particularly, cases of 1:N and N:1 correspondence pose challenges for designing visualization transformations.
Artificial intelligence (AI) has made it easier than ever to produce research outputs. Adding an agentic tool to a research team's work potentially extends the team's capabilities but calls for an active approach to organizing the research process. The pressing question, therefore, is how research teams should organize their work with this technology. We contend that the question is not new: software engineering and security research have long studied how teams of people working with machines keep control of projects with many contributors. We distill principles from the practices these literatures report and transpose them to AI-assisted research. The principles form TRACE: Track records, Review against sources, Accept independently, Control transitions, and Evaluate corrections. Together, the five principles give researchers a common basis for remaining in control while integrating AI into their workflows. By bounding units of work and attending to how each is shaped, they can help researchers pace AI-assisted work to human attention rather than to the speed at which AI produces content. We also introduce TRACE-OS, an operating system for AI-assisted research projects. It puts the principles into practice through shared instructions, tools, and records that fit researchers' existing workflows. With TRACE, we aim to join a conversation about the governing principles through which the research community can benefit from AI. We contend that a simple enough and useful standard can contribute to harnessing AI capabilities, allowing teams to delegate parts of their work to AI, while also building trust within teams and the broader research community.
Kim Zierahn, Cristina Cachero, Anna Korhonen, Nuria Oliver
Human personality inventories are increasingly used to characterize large language models (LLMs), compare systems, and inform downstream governance claims. Yet, these inventories were developed and validated for humans, and it remains unclear whether they are valid for non-human systems. We present a systematic psychometric evaluation of Big Five personality measurement in LLMs. We ask three research questions: Do Big Five inventories a) appropriately describe LLMs, b) capture meaningful differences between models, and c) reflect internal factors consistent with human personality? We assess the content validity of five candidate Big Five inventories and administer the best-performing inventory to N = 264 LLMs spanning 50 model families. Our findings are threefold. First, Big Five items adapted for LLMs achieve acceptable content validity, whereas the original human-developed items do not. Second, Big Five inventories fail to capture meaningful differences across LLMs: between-model variance accounts for only 7% - 17% of the total score variance. Third, LLMs responses do not reproduce the canonical Big Five five-factor structure of human personality, with four of the five personality facets collapsing into one (r >= .90). Moreover, comparisons between base and instruction-tuned variants suggest that alignment training shifts Big Five scores toward socially desirable profiles. These findings demonstrate that Big Five inventories do not measure a construct equivalent to human personality in LLMs. Thus, using human personality frameworks to characterize, benchmark, compare, or govern LLMs risks producing misleading conclusions. We highlight the need for evaluation frameworks that are specifically designed and validated for LLMs, rather than transferring human psychological constructs without first establishing their validity.
Web systems increasingly predict users' decisions from what users say, yet when interests partly conflict, users may promise to cooperate to win the other side's cooperation and then not do so. Sales or service agents that rely on such promises may grant concessions to users who then do not keep their promises, at a cost to the party they represent. We test how LLMs combine users' statements with decision records to predict users' decisions, using 1,260 rounds in which 42 users played a repeated prisoner's dilemma-style game with five agents, each prompted to express a dominant Big Five trait, and could talk with the agent before every decision. Across eight open-weight LLMs, predictions from decision records separated cooperation from defection well (AUC = .80), and adding dialogue brought no detectable gain. With dialogue, predictions agreed with users' statements more often, and errors concentrated on rounds in which users broke their statements: accuracy on rounds with a statement barely changed (.73 to .74), while it rose on kept statements (.76 to .95) and fell sharply on broken ones (.65 to .17). Predictions weighted statements more heavily than users' behavior warranted: when users stated a decision they had not yet made in that game, the predicted probability of keeping it rose from .13 to .69, whereas users kept .40 of such statements. Users' decisions also varied with the agent they faced, but predictions hardly reflected this: users cooperated more with the Agreeableness agent than with the Extraversion agent (70.6% vs. 54.5%), whereas predicted cooperation was nearly identical (64.5% vs. 63.1%). Users also cooperated more with an agent when they themselves scored higher on that agent's trait, whereas predicted cooperation barely changed with these scores. Dialogue-based predictions should therefore weigh users' statements by their earlier decisions and the agent they face.
Grip pressure, including contact force and its distribution across the hand, is central to skilled grasping but is not directly visible. This makes it difficult to teach, learn, and self-correct in grip-dependent motor tasks, where verbal instruction and tactile demonstration provide only partial guidance and no real-time, spatially resolved feedback tied to the learner's own grip. Using an instrumented glove, we compare pressure feedback visualizations for such guidance, evaluating how visualization type and, secondarily, sensor-region mapping shape pressure-matching performance in novices. In a study using baseball and table tennis as representative grip tasks (N=28), we find that a continuous gradient visualization produces significantly more accurate pressure matches and lower gaze entropy than a discrete bar display, despite participants perceiving the bar as more precise for pressure magnitude. Perceived precision and objective performance therefore diverged: the encoding participants judged more precise was not the one they matched more accurately with. Grip-tailored sensor-region mappings further reduced perceived temporal demand and effort, although this subjective workload advantage did not extend to objective matching accuracy. Overall, continuous pressure feedback supported more accurate grip matching and more concentrated viewing, and we present design considerations for real-time pressure feedback in motor skill guidance.
Poetry conveys emotional experiences through language. This project investigates how poetic emotional experiences can be represented through dynamic mathematical visualisations while further developing a pipeline for this purpose. By leveraging a pretrained RoBERTa-based emotion classifier, the system analyses the fine-grained emotional states within a poem. These detected emotions are then mapped onto the parameters of strange attractors, complex structures from dynamical systems theory, to generate abstract, deterministic, and dynamic imagery. This work situates itself at the intersection of poetry, psychology, and mathematics, aiming to extend the poetic experience beyond language into the sensory domain
The Reality-Virtuality (RV) continuum helps in describing the relationship between physical and computer-generated environments. However, the increasing number of Extended Reality (XR) paradigms introduces new technologies. Their characteristics cannot be completely represented by a single real-virtual dimension. It can be noted that XR systems not only vary in their proportion of real and virtual content but also adds, suppresses, or transforms information available to the user. This work proposes a framework for characterizing seven XR realities, namely Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), Augmented Virtuality (AV), Phenomenal Reality (PR), Annexing Reality (AnR), and Diminished Reality (DR). It uses a two-dimensional extension of the conventional 1D RV continuum. The first dimension helps in identification of the position of an XR technology between real and virtual environments. The second dimension captures information addition and subtraction. The proposed approach gives a qualitative basis for comparing existing and emerging XR paradigms within a unified XR framework.
Generative AI (GenAI) is shifting software engineering students' work from producing artefacts towards evaluating and supervising AI-generated outputs. However, little is known about what characteristics students identify in outputs when reporting interventions, or how they exercise oversight during authentic software engineering work. We analysed 962 student-AI interaction turns documented by 145 first-year software engineering students through structured AI Journals, including 172 student-declared interventions. Using a multi-dimensional coding framework, we characterised the output characteristics identified in these reported interventions, students' reasons for intervening, the actions they took, and how those actions were enacted. Fitness problems were identified in 74.4% of interventions, compared with 16.9% involving correctness problems, indicating that oversight more often concerned whether plausible AI outputs were suitable for the engineering task than whether they contained explicit errors. Among classifiable interventions, students most frequently modified generated outputs (51.0%) or refined their prompts (28.2%), with both response types commonly associated with engineering fit. Within-student analysis further showed that outputs perceived as containing minor flaws were associated with lower evaluation confidence than outputs perceived as perfect. Taken together, these findings characterise novice AI oversight as a process that extends beyond error detection to encompass both judgement of fitness for purpose and active intervention in AI-assisted engineering work.
Third-party community-based bots are central to how social platforms such as Discord work. They moderate content, manage roles, play music, and support AI-assisted conversations across millions of servers. Yet when a bot is installed in a server, it can access messages, user profiles, and server metadata concerning members who did not participate in the installation decision, with no standardized way to communicate the associated security and privacy implications. In this paper, we analyze 23,102 third-party Discord bots and 183,142 user reviews spanning 49 detected languages and five years of feedback (2021-2026). We employ LLM-based knowledge distillation and fine-tune a DeBERTa-based classifier to identify candidate security and privacy (S&P)-relevant reviews. After manually validating the complete classifier-positive set, we retain 8,832 reviews identified as S&P-relevant, representing an observed pipeline yield of 4.82%, and map them to a 31-theme S&P taxonomy extending prior work. Our results show that S&P-related reviews focus heavily on Data Sharing (30.5%) and Data Collection (16.2%). Confession and anonymity bots have pipeline-identified SPRRs above 50%, more than ten times the ecosystem-wide pipeline yield of 4.82%. Star ratings may obscure the frequency and valence of S&P-related discussion, while conventional metadata features, such as permission count and administrator status, show only weak associations with user-reported S&P concerns. We then build BotLens, a proof-of-concept auditing framework that combines review-derived S&P signals with bot metadata to provide on-demand decision support for the community-based bot ecosystem. BotLens exposes six tools through a Model Context Protocol server, supporting bot-level checkups, server-wide comparison, and integration with AI agents.