A process-oriented perspective on pre-service teachers' self-efficacy and their motivational messages: Using large language models to classify teachers' speech

Abstract

Background: Recent studies have examined the relation between teacher motivation, motivational messages and student learning but are limited to an achievement-related context, primarily using survey data. Moreover, our understanding of the relation between various teacher characteristics, such as teacher self-efficacy (TSE), and their motivational message use remains limited. Aims: Our study tested whether teacher speech can be classified into self-determination (SDT)-based motivational messages and reliably assessed with a large language model (LLM). Additionally, we analysed the relation between pre-service TSE and their motivational message use. Sample: For our first aim, we used human-rater annotations from 119 pre-service teachers’ classroom recordings. For our second aim, we used data from 103 pre-service teachers (52.69% female; Mage = 22.98, SDage = 3.26, Minage = 19, Maxage = 34) who participated in a survey and were video-recorded while teaching. Methods: First, we manually classified pre-service teachers’ motivational messages based on transcripts and used human-rater annotations to fine-tune an LLM. Second, we analysed the relation between pre-service TSE and motivational message use. Results and Conclusions: The fine-tuned LLM demonstrated promising performance in assessing SDT-based motivational messages but needs further refining to assess thwarting messages. The analysis with human annotation showed that pre-service TSE for classroom management positively affected the frequency of relatedness-supportive messages. Pre-service TSE for student engagement increased the likelihood of never using a competence- or relatedness-thwarting message. Pre-service TSE for instructional strategies reduced the frequency of autonomy-supportive messages. LLM-based analyses showed slightly different results but did not contradict human annotation-based analyses.