Can Affective Pedagogical Agents Improve Multimedia Learning Outcomes?

Yanqing Wang, Shaoying Gong, Tiantian Jiang, Yanan Wu
Key Laboratory of Adolescent Cyberpsychology and Behavior, Ministry of Education, and School of Psychology, Central China Normal University, Wuhan 430079

Abstract

In recent years, researchers have devoted considerable attention to how affective pedagogical agents influence learning. An affective pedagogical agent is a type of instructional agent that can evoke emotional experiences in learners through multiple modalities such as speech, facial expressions, and body movements. Most existing studies have found that affective agents can effectively elicit positive emotions in learners (d_{positive emotions} = 0.45) and enhance intrinsic motivation (d_{intrinsic motivation} = 0.52), though their effects on learning outcomes are relatively modest (d_{retention} = 0.18, d_{comprehension} = 0.32, d_{transfer} = 0.14, d_{combined} = 0.32). Researchers have explained these effects from various theoretical perspectives, including emotional contagion theory, emotional response theory, the Cognitive-Affective Theory of Learning with Media (CATLM), cognitive load theory, and interference theory. Future research should further investigate the effects of affective agents by examining experimental manipulations, boundary conditions, and internal mechanisms.

Keywords: affective pedagogical agent, emotion, intrinsic motivation, learning outcomes, multimedia learning

The popularity of instructional videos has prompted educators to consider how to design videos that optimize learning (Beege et al., 2017; de Koning et al., 2018; Merkt et al., 2020). Previous research has often focused on the cognitive processes underlying video design (e.g., selection, organization, and integration). In recent years, the role of affective processing (e.g., emotion and motivation) in learning has also attracted increasing attention (Gong et al., 2017; Endres et al., 2020; Lawson et al., 2021; Oker et al., 2020; Park et al., 2015; Plass et al., 2014; Uzun & Yıldırım, 2018; Um et al., 2012; Shangguan et al., 2020). How, then, can instructional design promote affective processing? One approach involves incorporating an affective pedagogical agent (APA) into the video learning interface. As part of emotional design for interactive features in online learning environments, an affective agent applies emotional design elements (e.g., happy facial expressions) to a pedagogical agent, aiming to evoke positive emotions, increase learning motivation, and thereby improve learning outcomes (Guo & Goh, 2015). Numerous empirical studies have examined the effects of affective agents across various domains, including science knowledge learning (Horovitz & Mayer, 2021; Schneider et al., 2022), information literacy games (Guo et al., 2015), and web-based software tutorials (Baylor & Kim, 2009). Based on recent advances in research on affective agents in multimedia learning, this paper elaborates on the concept and experimental manipulation of affective agents, discusses the theoretical foundations for their effects on learning processes and outcomes, and comprehensively reviews empirical studies that have found either facilitative or inhibitory effects, with the aim of providing insights for future research on the instructional value of affective agents.

2. Concept and Manipulation of Affective Agents

An affective pedagogical agent is an instructional agent that influences learners' emotional experiences through facial expressions, voice, body movements, and verbal information (Guo & Goh, 2015). Based on a review of previous research, affective agents can be categorized into two main types: expressive affective agents and empathic affective agents. Expressive affective agents influence learners' emotional experiences solely through their own emotional expressions (e.g., using smiling facial expressions and enthusiastic voices) (Beege et al., 2020; Lawson et al., 2021; Liew et al., 2017). Empathic affective agents, by contrast, provide emotional feedback based on learners' performance or emotional states (e.g., nodding, encouragement, and empathy) with the aim of regulating learners' emotions and motivating them to persist (Ba et al., 2021; Guo et al., 2014, 2015). Although researchers have operationalized affective agents differently, these agents share two important characteristics: (1) they are computer-screen agents capable of emotional expression, and (2) they are designed to increase learners' positive emotions and intrinsic motivation to ultimately facilitate learning.

Empirical studies have employed different design approaches for expressive and empathic agents. For expressive agents, researchers primarily manipulate the agent's facial expressions, voice, and posture (gestures) to display smiling faces, enthusiastic voices, and high levels of body movement. For example, in Liew et al. (2017), university students in the enthusiastic agent condition learned from an agent with smiling facial expressions and an enthusiastic voice, whereas those in the neutral agent condition learned from an agent with neutral facial expressions and voice. In Horovitz and Mayer (2021), the positive agent was endowed with happy facial expressions, voice, gestures, and body posture, while the bored agent displayed bored facial expressions, voice, gestures, and body posture.

Regarding the manipulation of empathic agents, researchers have primarily designed them using either parallel empathy or reactive empathy. Parallel empathy occurs when the agent mimics the learner's emotions—for instance, displaying happy facial expressions when it detects the learner's pleasant mood (Arroyo et al., 2009). Reactive empathy involves the agent providing feedback on the learner's emotions or behaviors through verbal and nonverbal cues such as verbal encouragement, applause, nodding, or clapping. For example, Terzis et al. (2012) used Facereader and manual detection to monitor learners' emotional states in real time, and the affective agent would display the same emotion as the learner and provide specific feedback. When the agent detected a happy expression, it would show a happy face and simultaneously display text messages such as "This quiz makes you happy, keep up the good work!" to provide emotional support. In Guo and Goh (2016), learners engaged with an information literacy game where the positive agent would display an encouraging smile when they answered incorrectly and encourage them with messages like "Don't be discouraged, read the question carefully." In the neutral agent condition, the agent maintained neutral facial expressions throughout the game and provided feedback without emotional encouragement (e.g., "Yes, that is correct").

In summary, current manipulations of affective agents are primarily achieved through emotional design of pedagogical agents embedded in video materials. Although these approaches have comprehensively considered design methods, several issues warrant further reflection. First, numerous studies have overlooked the influence of agent type when designing affective agents. For instance, Horovitz and Mayer (2021) used both human and virtual agents, while Guo and Goh (2016) used ghost-shaped agents. Lawson et al. (2021) found that when investigating whether learners could recognize emotions expressed by agents, human instructors displayed emotions more vividly and recognizably than virtual agents, particularly for high-arousal emotions such as happiness and frustration. Thus, different agent types may influence the effectiveness of affective agent manipulations and ultimately affect research outcomes. Second, researchers have not strictly controlled the level of emotional design in affective agents. Some studies have used single emotional cues (e.g., smiling) (Liew et al., 2016), while others have used multiple cues (e.g., smiling, voice, and gestures) (Ba et al., 2021). Which level of emotional cue is most effective remains unanswered in current research.

3. Can Affective Agents Evoke Positive Emotions in Learners?

Before examining how affective agents influence learners' emotions, a crucial question is whether learners can recognize the emotions expressed by the agent. From a theoretical perspective, the Cognitive-Affective Theory of Learning with Media (CATLM; Moreno & Mayer, 2007) posits that when an affective agent is presented in an instructional video, the first critical step is for learners to recognize the emotional state (positive, neutral, or negative) displayed by the agent. Recognition of the agent's emotion is not only an important test of successful manipulation but also a key entry point for investigating the effects of affective agents.

Lawson et al. (2021) examined whether university students could recognize emotions (happy, satisfied, frustrated, or bored) expressed by an instructor teaching a mathematics lesson on binomial probability distribution. Results showed that when viewing videos with a happy agent, learners rated happiness significantly higher than frustration and boredom, though the distinction from satisfaction was less clear. When viewing videos with satisfied, frustrated, or bored agents, learners could better differentiate among the expressed emotions. Chen et al. (2012) found that learners could accurately recognize anger, sadness, surprise, happiness, and neutral emotions in a computer-assisted learning environment, but were less accurate in identifying fear, worry, and disgust. These findings indicate that learners do not always successfully recognize agents' emotions, making pre-experimental validation essential.

Recognizing the agent's emotion is only the first step in testing affective agent effectiveness; the primary purpose of incorporating affective agents is to examine whether this instructional design influences learners' emotions. According to emotional contagion theory, an individual's emotional state is susceptible to another's emotional expressions (Hatfield et al., 1994). In social interactions, people unconsciously express their emotions through nonverbal information such as facial expressions, body movements, and posture, while simultaneously perceiving signals conveyed by others. Research suggests that emotional contagion can also occur in human-computer interaction (Tsai et al., 2012), where the agent's emotions directly influence learners' emotions (Feng, 2020; Krämer et al., 2013; Ku et al., 2005; Liew et al., 2016). Additionally, emotional response theory posits a close relationship between instructors' enthusiastic verbal and nonverbal cues and learners' affective responses (Horan et al., 2012). According to this theory, positive verbal and nonverbal cues from pedagogical agents can induce positive emotional experiences. Similarly, CATLM suggests that when learners recognize positive emotions in pedagogical agents, they exhibit the same emotions (e.g., seeing a happy agent makes students happy).

To more clearly demonstrate the effects of affective agents, this paper calculated Cohen's d effect sizes from relevant empirical studies (see Table 1) and adopted Fiorella and Mayer's (2015) method of computing median effect sizes to quantify these effects, thereby overcoming the limitation of narrative reviews that cannot evaluate effect magnitude. Studies were included based on the following criteria: (1) empirical research; (2) primary comparison between affective agents and non-affective (neutral) agents or no-agent conditions, though one study compared positive and bored agents (Horovitz & Mayer, 2021); (3) detailed reporting of dependent variables (emotion, motivation, cognitive load, and learning outcomes). Among the 16 studies that explicitly reported learners' emotional states, 10 found that the agent's emotions simultaneously evoked positive emotions in learners. For example, Liew et al. (2017) found that learners in the enthusiastic agent condition reported more positive emotions than those in the neutral agent condition. Wang et al. (2019) similarly found that agents with rich facial expressions evoked higher levels of emotion in learners. However, five studies found no effect of affective agents on learners' emotions (van der Meij, 2013; Beege et al., 2020), and one study found that affective agents reduced learners' positive emotions (Liew et al., 2016).

In summary, most studies support the effect of affective agents on learners' positive emotions, with a median effect size of d_{positive emotions} = 0.45. Notably, some previous studies have neglected to test learners' recognition of agents' emotions, raising the question of whether null findings reflect unsuccessful manipulation or genuine ineffectiveness—an issue requiring future investigation.

4. Do Affective Agents Influence Learners' Subjective Experiences?

If affective agents can evoke positive emotions, how do these emotions influence subjective experiences such as motivation and cognitive load during learning? Learners' perceptions of their motivational levels and cognitive load are closely related to subsequent learning outcomes, making the examination of subjective experiences a crucial aspect of testing affective agent effectiveness.

According to emotional response theory, positive emotions induced by instructors' verbal and nonverbal cues can increase learners' motivation and produce approach behaviors toward learning. CATLM (Moreno & Mayer, 2007) similarly posits that when affective agents evoke positive emotions, these emotional changes enhance motivation. As shown in Table 1, among 17 studies that reported learning motivation, 16 found that affective agents increased learners' motivational levels (Baylor & Ryu, 2003; Saerbeck et al., 2010; Horovitz & Mayer, 2021; van der Meij, 2013). For instance, van der Meij (2013) found that learners in the affective agent condition reported higher motivation and were more likely to experience flow during software tutorial training. Horovitz and Mayer (2021) found that both happy human and virtual agents increased learners' intrinsic motivation. Only one study found that learning with a smiling agent tended to reduce motivation (Liew et al., 2016), possibly because learners perceived the smile as fake rather than genuine, which undermined trust in the agent and consequently reduced motivation. Overall, despite variations in affective agent design across studies, the vast majority demonstrate the relative advantage of affective agents in enhancing learner motivation, with a median effect size of d_{intrinsic motivation} = 0.52.

Cognitive load theory (CLT) proposes three types of cognitive load (Sweller, 2005): intrinsic cognitive load (ICL), extraneous cognitive load (ECL), and germane cognitive load (GCL). ICL is associated with the inherent complexity of learning materials, while ECL results from non-optimal instructional design. High levels of these two loads may impair learning. GCL refers to the cognitive resources devoted to schema construction and generative processing, which facilitates learning. To avoid cognitive overload, instructional designers should minimize ECL and increase GCL, thereby optimizing the use of limited cognitive resources to achieve optimal learning outcomes. According to CLT, incorporating pedagogical agents into multimedia learning environments may increase ECL because learners must process additional (learning-irrelevant) information.

In empirical research examining the relationship between affective agents and cognitive load, eight studies measuring ICL found no effect of affective agents, with a median effect size of d_{ICL} = -0.01. Since ICL is determined by material complexity and learners' prior knowledge, it is difficult to modify through instructional design, making these null findings unsurprising. Among eight studies reporting ECL, seven found that affective agents did not impose additional cognitive load. Only one study found that affective agents increased ECL when learners were under high mental load (Beege et al., 2020, Exp. 1a), yielding a median effect size of d_{ECL} = 0.09. Of seven studies reporting GCL, three showed that affective agents increased germane cognitive load (Feng, 2020, Exp. 2; Xie, 2020, Exp. 1; Beege et al., 2020), directing more mental resources toward comprehending learning materials, while three found no difference between affective and neutral agents. However, one study investigating the effects of an enthusiastic voice on non-native university students found that strong prosodic features actually reduced GCL (Liew et al., 2020), suggesting that non-native learners may benefit more from flatter prosody (Davis et al., 2019). The median effect size for affective agents on GCL was d_{GCL} = 0.08. Overall, affective agents appear to have minimal impact on reducing ECL and increasing GCL, though these results also suggest that adding affective agents to video learning may not increase learners' extraneous cognitive load.

Although researchers have explored affective agents' influence on subjective experiences, current investigations remain limited, focusing primarily on motivation and cognitive load. Future research should measure a broader range of subjective experiences (e.g., learning satisfaction, interest, sense of achievement) and examine their causal relationships with learning outcomes to more comprehensively understand the internal mechanisms through which affective agents operate.

5. Can Affective Agents Improve Learning Outcomes?

Although affective agents may evoke positive emotions and positively influence subjective experiences such as motivation, researchers are more concerned with their effects on learning outcomes in actual instructional practice. According to emotional response theory and CATLM, affective agents can improve learning outcomes by influencing learners' approach behaviors or motivation levels. However, based on interference theory (Moreno et al., 2001), affective agents constitute learning-irrelevant material whose facial expressions and gestures may attract learners' attention, thereby reducing attention to and processing of learning content and ultimately interfering with learning outcomes.

Previous literature has primarily used four measures to assess affective agents' impact on learning outcomes: retention tests, comprehension tests, transfer tests, and combined tests. Retention tests assess learners' ability to recall or recognize information directly available in the learning materials (Mayer, 2009). Comprehension tests evaluate understanding of important information (Um et al., 2012). Transfer tests examine learners' ability to apply knowledge to solve new problems (Mayer, 2009). Combined tests measure overall learning performance (e.g., the sum of retention, comprehension, and transfer scores).

As summarized in Table 1, among 32 empirical studies reporting learning outcomes, 14 found that adding affective agents to video learning facilitated learning. For example, Liew et al. (2017) designed two 3D pedagogical agents to teach programming knowledge—one conveying enthusiastic verbal and nonverbal behaviors and the other displaying neutral behaviors—and found that university students achieved better learning outcomes in the enthusiastic agent condition. Bringula et al. (2018) similarly found that agents providing feedback on learners' behavior through facial expressions significantly improved seventh-grade students' mathematics performance. van der Meij (2013) had elementary school students learn computer-related knowledge with or without an affective agent and found that the affective agent group performed better on post-tests.

However, 17 studies showed no significant differences in learning outcomes between affective and control groups. For instance, Horovitz and Mayer (2021) found that university students learning about binomial probability from happy agents did not outperform those learning from bored agents. Guo and colleagues also found no positive effects of affective agents on learning outcomes (Guo et al., 2015; Guo & Goh, 2016). One study even found that adding an agent with facial expressions hindered learners' comprehension (Frechette & Moreno, 2010). Calculating median effect sizes revealed d_{retention} = 0.18, d_{comprehension} = 0.32, d_{transfer} = 0.14, and d_{combined} = 0.32.

These findings, combined with the results in Table 1, indicate that the effects of affective agents on learning outcomes are not robust. This inconsistency may be attributed to moderating variables. First, learner characteristics are closely related to affective agent effectiveness. For example, (1) working memory capacity: the amount of information people can process simultaneously in working memory is limited, and exceeding this capacity may interfere with learning (Mayer, 2014, 2020). Beege et al. (2020) found that for learners with low working memory capacity, affective agents increased extraneous cognitive load, interfering with processing of key information and hindering learning, whereas for those with high capacity, affective agents facilitated learning. (2) Grade level: Hernández et al. (2009) tested affective agents in an intelligent tutoring environment across four experiments and found them more effective for lower-grade learners but not for improving comprehension in higher-grade learners.

Second, affective agent type may contribute to inconsistent findings. Liew et al. (2016) found that an expressive agent with only a smiling facial expression had no effect on emotions, motivation, or learning outcomes, whereas Liew et al. (2017) improved the design by using an empathic agent with smiling expressions, head movements, and enthusiastic comments, which yielded positive effects. Additionally, task type may moderate affective agent effectiveness. Some studies found positive effects in linear algebra tasks (Bringula et al., 2018), while effects disappeared in atypical learning tasks such as information literacy games (Guo & Goh, 2016). Finally, test timing may influence effectiveness, with some research suggesting that benefits are more likely to emerge on delayed tests (Horovitz & Mayer, 2021). Investigating these potential moderating variables represents an important direction for future research.

6. Summary and Future Directions

Based on the above discussion and the results summarized in Table 1, most previous researchers have found that adding an affective pedagogical agent to video learning can evoke learners' positive emotions (d_{positive emotions} = 0.45) and enhance their motivation levels (d_{intrinsic motivation} = 0.52). However, affective agents have very weak effects on cognitive load (d_{ICL} = -0.01; d_{ECL} = 0.09; d_{GCL} = 0.08) and modest effects on learning outcomes (d_{retention} = 0.18; d_{comprehension} = 0.32; d_{transfer} = 0.14; d_{combined} = 0.32). Researchers have explained the potential effects of affective agents from different theoretical perspectives, including emotional contagion theory, emotional response theory, and CATLM, suggesting that affective agents may improve learning outcomes by influencing learners' emotions and motivation (or approach behaviors) (see Figure 1). The few studies finding inhibitory effects support interference theory, which posits that agents' rich facial expressions and gestures may attract learners' attention, reducing focus on key information and thereby interfering with learning. However, many studies have found no differences between affective and non-affective agents, suggesting that the robustness of affective agents' effects on learning outcomes requires continued examination. Overall, under the guidance of these different theories, researchers have attempted to apply affective agents to instructional practice. Despite inconsistent findings, learners are generally happier and more motivated with positive affective agents. Therefore, in educational practice, instructional designers may consider presenting a positive pedagogical agent to help learners study more joyfully.

Figure 1. The learning process through which affective agents facilitate learning from the perspectives of emotional contagion theory, emotional response theory, and CATLM (Note: Arrows indicate possible causal directions). Based on Hatfield et al., 1994; Horan et al., 2012; Horovitz & Mayer, 2021; Lawson et al., 2021; Moreno & Mayer, 2007.

Future research should systematically investigate the effects and mechanisms of affective agents in several ways:

First, researchers should focus on the manipulation and evaluation methods of affective agents. Early studies compared affective agents with neutral agents using different design elements (facial expressions, voice, verbal feedback, body movements, etc.). However, which single element or combination works best? Are more emotional design elements always more effective? Future research should explore these questions in greater detail. Additionally, current evaluation methods have limitations. Some researchers have used self-report measures to assess affective agent design, but the validity of this approach remains questionable. Some studies have not even tested whether learners could successfully recognize the agent's emotions (Baylor & Kim, 2009; Guo et al., 2015; Liew et al., 2017). Future research should strengthen evaluation of affective agent design by adding post-learning questions such as: "How noticeable and recognizable was the agent's positive facial expression?" (1 = not noticeable at all, 5 = very noticeable) and "How enthusiastic was the agent's voice?" (1 = not enthusiastic at all, 5 = very enthusiastic). Multiple assessment methods (self-report + objective measurement) should be used to validate affective agent design effectiveness.

Second, researchers should examine boundary conditions affecting affective agent effectiveness. Previous reviews suggest that effectiveness may be influenced by potential moderators such as agent type (Liew et al., 2016, 2017), learners' working memory capacity (Beege et al., 2020), and grade level (Hernández et al., 2009). Beyond these variables, other factors such as learners' prior knowledge, emotional states, material difficulty, and learning duration remain unexplored. Notably, learners experience various emotional states during learning, including both positive and negative emotions. Learners with high emotion regulation ability can effectively regulate negative emotions and maintain optimal learning states, whereas those with low emotion regulation ability struggle with emotion regulation and are more likely to experience negative emotions (Graziano et al., 2007). From this perspective, adding a positive affective agent may be particularly beneficial for low emotion regulation ability learners. Therefore, future investigation of learners' emotion regulation ability as a moderating variable is essential.

Third, researchers should explore the underlying mechanisms of affective agents. With the rise of educational neuroscience, researchers increasingly use eye-tracking, functional near-infrared spectroscopy (fNIRS), and other technologies to reveal the cognitive mechanisms underlying learning phenomena. Some studies have found that pupil dilation in eye-tracking measures may reflect cognitive load during learning (Lee et al., 2020). Future researchers could adopt this method to examine whether the additional cognitive load induced by affective agents causes pupil changes and how these changes relate to learning outcomes. Additionally, based on theoretical speculation that affective agents may serve as decorative materials irrelevant to learning content and interfere with attention to key information, future research could use direct eye-tracking measures (e.g., fixations, fixation duration, and fixation counts on areas of interest) to test whether affective agents act as attention guides or attention distractors, thereby deepening understanding of learners' attentional patterns. Furthermore, future research could use fNIRS or EEG to explore how different types of pedagogical agents (e.g., positive affective agents vs. neutral agents) influence learners' brain activity and identify which brain regions are associated with facilitative or inhibitory effects, thereby revealing the neural mechanisms underlying affective agent effectiveness. For example, a recent study found that learners' positive processing of learning materials in video learning may be associated with higher theta oscillations in the frontal and central cortices (Pi et al., 2021). Can affective agents induce theta oscillations in these regions? Future research could use EEG to investigate this question.

Fourth, researchers should focus on theoretical refinement. Although both facilitative and inhibitory theories of affective agents have received some support, neither can adequately explain cases where affective agents neither facilitate nor inhibit learning. Our review found that some studies found no differences between affective and control groups (neutral or bored agents) (Guo et al., 2015; Kim et al., 2007), with null results attributed to potential boundary conditions. Future research should consider how to incorporate these moderating variables into theoretical frameworks to enhance explanatory power. Moreover, current theoretical accounts of how affective agents influence learning outcomes are based on speculation, lacking examination of relationships between internal subjective variables (emotion, motivation) and learning outcomes. Some studies have found that affective agents evoked emotions and motivation without improving learning outcomes (Horovitz & Mayer, 2021). Future research should test theoretical hypotheses and seek new theoretical evidence by constructing mediation models to reveal how affective agents influence learning outcomes through process variables (emotion, motivation, situational interest, etc.).

Fifth, researchers should explore additional effects of affective agents. Previous research has primarily focused on direct effects on positive affective states, motivation, and learning outcomes (Beege et al., 2020; Liew et al., 2017), largely neglecting other potential benefits, such as whether affective agents can protect learners from negative influences. During learning, learners must regulate their internal processing; successful self-regulation helps them adapt to environmental challenges, whereas failed regulation may lead to ego depletion and interfere with learning (Baumeister, 2014). Can affective agents mitigate the detrimental effects of ego depletion on learning performance? Additionally, in real learning environments, learners experience not only positive but also negative emotions. Can adding a positive affective agent to video learning reduce learners' negative emotions? Future research should investigate these questions more comprehensively.

Sixth, researchers should simultaneously consider affective and cognitive processing. Emotion and cognition are equally important in learning, so instructors should consider how to evoke learners' emotional states while guiding active cognitive processing. Fiorella and Mayer (2015) reviewed eight generative processing strategies commonly used in learning (e.g., self-generated drawing, self-explanation) and advocated combining effective instructional design with appropriate generative learning strategies to achieve optimal instructional effects (Fiorella et al., 2020). Building on this, researchers could investigate whether combining affective agents with generative processing strategies better promotes learning. Furthermore, pedagogical agents can provide not only emotional feedback but also cognitive support, yet only Xie (2020) has combined emotional and cognitive feedback to examine affective agent effects. Future research could adopt this approach, using Facereader facial expression recognition technology combined with think-aloud protocols to deeply examine the dynamic relationship between emotion and cognition during learning, thereby promoting more enjoyable and efficient learning.

Seventh, researchers should examine the ecological validity of affective agents and extend them to real-world instructional environments. Current research has mostly used convenient samples of university students, short learning materials (less than 10 minutes), and controlled laboratory settings. Future research should test affective agent effects in more authentic instructional contexts with elementary or secondary school students and longer learning materials (e.g., 30 minutes), thereby advancing the translation of experimental research to real-world teaching.

Table 1 Effects of Affective Agents on Learners' Emotions, Subjective Experiences, and Learning Outcomes

Study Agent Manipulation Learning Domain Emotion (E) Motivation (M) ICL ECL GCL Retention (R) Comprehension (C) Transfer (T) Combined (U) Ba et al., 2021 Emotional expression/feedback (facial, voice, verbal) Problem-solving model E*(0.75) M*(0.82) - - - - - T*(0.77) - Baylor & Ryu, 2003 Emotional feedback (motivational & affective support) Math word problems - M*(0.53) - - - - - - - Baylor & Kim, 2009 Emotional expression (rich facial expressions) Software programming - - - - - - - - - Beege et al., 2020; Exp. 1a Emotional expression (enthusiastic voice) Solar system planets E*(0.44) - ICL(0.40) ECL*(-0.62) GCL*(0.86) R*(0.46) - T*(0.19) - Beege et al., 2020; Exp. 1b Emotional expression (enthusiastic voice) Solar system planets E*(0.50) - ICL(-0.11) ECL*(0.59) GCL(-0.12) R(-0.36) - T(0.04) - Bringula et al., 2018 Emotional expression/feedback (text & expression feedback) Linear algebra equations - - - - - - - - U*(0.71) Chen et al., 2012; Exp. 2 Emotional feedback (facial, gesture, voice feedback on emotions) Computer knowledge - - - - - - - - - Frechette & Moreno, 2010 Emotional expression (smiling) Astronomy knowledge - - - - - - - - - Guo & Goh, 2016 Emotional expression/feedback (encouraging smile, verbal, nodding, applause) Information literacy game - - - - - - - - - Guo et al., 2014 Emotional expression/feedback (pleasant face, gestures, encouragement) Information literacy - - - - - - - - - Guo et al., 2015 Emotional expression/feedback (pleasant face, gestures, encouragement) Information literacy - - - - - - - - - Hernández et al., 2009 Emotional feedback (verbal or behavioral encouragement) Intelligent tutoring - - - - - - - - - Horovitz & Mayer, 2021 Emotional expression (happy face, posture, voice) Binomial probability E*(1.82) M*(1.84) - - - - - - - Jaques, 2009 Emotional feedback (emotionally active attitude, encouraging messages) Instructional design concepts - - - - - - - - - Kim et al., 2007; Exp. 1 Emotional expression (facial, verbal, head movement) C programming - - - - - - - - - Kim et al., 2007; Exp. 2 Emotional expression (facial, verbal, head movement) C programming - - - - - - - - - Liew et al., 2016; Exp. 1 Emotional expression (smiling face) - E*(-0.35) M*(-0.31) - - - - - - - Liew et al., 2017 Emotional expression/feedback (smile, enthusiastic voice/nodding, comments) - E*(0.58) M*(0.67) - ECL(-0.12) - - C*(4.5) U*(0.24) - Liew et al., 2020; Exp.1 Emotional expression (enthusiastic voice) Computer competence E*(0.33) - - - GCL*(-0.57) - - - - Liew et al., 2020; Exp.1 Emotional expression (enthusiastic voice) Computer competence E*(0.47) - - - - - - - - Roselyn Lee et al., 2007 Emotional feedback (expressing empathy, providing verbal support) Technology use - M*(1.26) - - - - - - - Saerbeck et al., 2010 Emotional expression/feedback (happy when correct, sad when incorrect) - - M*(0.43) - - - - - - - van der Meij, 2013 Emotional feedback (facial, head movement, verbal feedback) Computer knowledge - M*(0.76) ICL(0.20) ECL(0.35) GCL(-0.01) R*(1.02) - - - van der Meij et al., 2015 Emotional expression (realistic facial expressions) Physics dynamics - M*(0.63) ICL(-0.07) ECL(0.11) GCL*(0.43) R*(1.82) - - - Veletsianos, 2009 Emotional expression/feedback (smiling, gestures, verbal feedback) - - M*(0.46) ICL(-0.34) ECL(0.09) GCL(0.08) R*(0.68) - - - Wang et al., 2019 Emotional expression (rich facial expressions) - E*(1.77) M*(-0.04) ICL(0.31) ECL(0.29) GCL*(0.41) R(0.17) - - - Feng, 2020; Exp. 1 Emotional expression (smiling face, enthusiastic voice) Chemical synapse transmission E*(0.28) M*(0.35) ICL(0.06) ECL(0.09) GCL(-0.12) - - T*(0.55) - Feng, 2020; Exp. 2 Emotional expression (smiling face, enthusiastic voice) Chemical synapse transmission E*(0.68) M*(0.52) - - GCL(0.15) - - T*(0.53) - Feng, 2020; Exp. 3 Emotional expression (smiling face, enthusiastic voice) Chemical synapse transmission E*(0.47) - - - - - - - - Xie, 2020; Exp. 1 Emotional feedback (facial & verbal feedback on emotions) Psychological statistics E(0.07) - ICL(-0.45) ECL(-0.08) GCL*(0.43) - - - - Xie, 2020; Exp. 2 Emotional feedback (facial & verbal feedback on emotions) Psychological statistics E(0.19) - - - - - - - -

Note: R = retention performance; C = comprehension performance; T = transfer performance; U = combined test performance (not distinguishing specific tests); E = positive emotion; M = motivation; ICL = intrinsic cognitive load; ECL = extraneous cognitive load; GCL = germane cognitive load; * indicates significant difference between affective and control groups; / indicates value not reported in study; Values in parentheses represent Cohen's d effect sizes comparing affective versus non-affective agents.

References

Feng, X. (2020). The effect of affective pedagogical agents on multimedia learning: The moderating role of learner characteristics (Master's thesis). Central China Normal University, Wuhan.

Xie, K. (2020). The effects of agent's emotional and cognitive feedback on formative assessment (Master's thesis). Central China Normal University, Wuhan.

Gong, S., Shangguan, C., Zhai, K., & Guo, Y. (2017). Effects of emotional design on multimedia learning. Acta Psychologica Sinica, 49(6), 771–782.

Arroyo, I., Woolf, B. P., Royer, J. M., & Tai, M. (2009). Affective gendered learning companions. In V. Dimitrova et al. (Eds.), Proceedings of the 2009 Conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling (pp. 41–48). IOS Press.

Ba, S., Stein, D., Liu, Q., Long, T., Xie, K., & Wu, L. (2021). Examining the effects of a pedagogical agent with dual-channel emotional cues on learner emotions, cognitive load, and knowledge transfer performance. Journal of Educational Computing Research. https://doi.org/10.1177/0735633121992421

Baylor, A. L., & Kim, S. (2009). Designing nonverbal communication for pedagogical agents: When less is more. Computers in Human Behavior, 25(2), 450–457.

Baylor, A. L., & Ryu, J. (2003). The effects of image and animation in enhancing pedagogical agent persona. Journal of Educational Computing Research, 28(4), 373–394.

Baumeister, R. F. (2014). Self-regulation, ego depletion, and inhibition. Neuropsychologia, 65, 313–319.

Beege, M., Schneider, S., Nebel, S., & Rey, G. D. (2020). Does the effect of enthusiasm in a pedagogical agent's voice depend on mental load in the learner's working memory? Computers in Human Behavior, 112, 106483. https://doi.org/10.1016/j.chb.2020.106483

Beege, M., Schneider, S., Nebel, S., & Rey, G. D. (2017). Look into my eyes! Exploring the effect of addressing in educational videos. Learning and Instruction, 49, 113–120.

Bringula, R. P., Fosgate Jr, I. C. O., Garcia, N. P. R., & Yorobe, J. L. M. (2018). Effects of pedagogical agents on students' mathematics performance: A comparison between two versions. Journal of Educational Computing Research, 56(5), 1–20.

Chen, G. D., Lee, J. H., Wang, C. Y., Chao, P. Y., Li, L. Y., & Lee, T. Y. (2012). An empathic avatar in a computer-aided learning program to encourage and persuade learners. Journal of Educational Technology & Society, 15(2), 62–72.

Davis, R. O., Vincent, J., & Park, T. (2019). Reconsidering the voice principle with non-native language speakers. Computers & Education, 140, 103605. https://doi.org/10.1016/j.compedu.2019.103605

de Koning, B., Hoogerheide, V., & Boucheix, J. M. (2018). Developments and trends in learning with instructional video. Computers in Human Behavior, 89, 395–398.

Endres, T., Weyreter, S., Renkl, A., & Eitel, A. (2020). When and why does emotional design foster learning? Evidence for situational interest as a mediator of increased persistence. Journal of Computer Assisted Learning, 36(4), 514–525.

Fiorella, L., & Mayer, R. E. (2015). Learning as a generative activity: Eight learning strategies that promote understanding. Cambridge University Press.

Fiorella, L., Stull, A. T., Kuhlmann, S., & Mayer, R. E. (2020). Fostering generative learning from video lessons: Benefits of instructor-generated drawings and learner-generated explanations. Journal of Educational Psychology, 112(5), 895–906.

Frechette, C., & Moreno, R. (2010). The roles of animated pedagogical agents' presence and nonverbal communication in multimedia learning environments. Journal of Media Psychology, 22(2), 61–72.

Graziano, P. A., Reavis, R. D., Keane, S. P., & Calkins, S. D. (2007). The role of emotion regulation in children's early academic success. Journal of School Psychology, 45(1), 3–19.

Guo, Y. R., & Goh, D. H. L. (2015). Affect in embodied pedagogical agents: Meta-analytic review. Journal of Educational Computing Research, 53(1), 124–149.

Guo, Y. R., & Goh, D. H. L. (2016). Evaluation of affective embodied agents in an information literacy game. Computers & Education, 103, 59–75.

Guo, Y. R., Goh, D. H.-L., & Luyt, B. (2014). Using affective embodied agents in information literacy education. Paper presented at the Digital Libraries Conference, London.

Guo, Y. R., Goh, D. H. L., Luyt, B., Sin, S. C. J., & Ang, R. P. (2015). The effectiveness and acceptance of an affective information literacy tutorial. Computers & Education, 87, 368–384.

Hatfield, E., Cacioppo, J. T., & Rapson, R. L. (1994). Emotional contagion. Cambridge University Press.

Hernández, Y., Sucar, L. E., & Conati, C. (2009). Incorporating an affective behavior model into an educational game. Paper presented at the FLAIRS Conference, Sanibel Island, FL.

Horan, S. M., Martin, M. M., & Weber, K. (2012). Understanding emotional response theory: The role of instructor power and justice messages. Communication Quarterly, 60(2), 210–233.

Horovitz, T., & Mayer, R. E. (2021). Learning with human and virtual instructors who display happy or bored emotions in video lectures. Computers in Human Behavior, 119, 106724. https://doi.org/10.1016/j.chb.2021.106724

Jaques, P. A., Lehmann, M., & Pesty, S. (2009). Evaluating the affective tactics of an emotional pedagogical agent. In Proceedings of the 2009 ACM Symposium on Applied Computing (pp. 104–109). ACM Press.

Kim, Y., Baylor, A. L., & Shen, E. (2007). Pedagogical agents as learning companions: The impact of agent emotion and gender. Journal of Computer Assisted Learning, 23(3), 220–234.

Krämer, N., Kopp, S., Becker-Asano, C., & Sommer, N. (2013). Smile and the world will smile with you—The effects of a virtual agent's smile on users' evaluation and behavior. International Journal of Human-Computer Studies, 71(3), 335–349.

Ku, J., Jang, H. J., Kim, K. U., Kim, J. H., Park, S. H., Lee, J. H., ... & Kim, S. I. (2005). Experimental results of affective valence and arousal to avatar's facial expressions. CyberPsychology & Behavior, 8(5), 493–503.

Lawson, A. P., Mayer, R. E., Adamo-Villani, N., Benes, B., Lei, X., & Cheng, J. (2021). Recognizing the emotional state of human and virtual instructors. Computers in Human Behavior, 114, 106554. https://doi.org/10.1016/j.chb.2020.106554

Lee, J. Y., Donkers, J., Jarodzka, H., Sellenraad, G., & van Merrienboer, J. J. (2020). Different effects of pausing on cognitive load in a medical simulation game. Computers in Human Behavior, 110, 106385.

Liew, T. W., Tan, S. M., Tan, T. M., & Kew, S. N. (2020). Does speaker's voice enthusiasm affect social cue, cognitive load and transfer in multimedia learning? Information and Learning Sciences, 121(3), 117–135.

Liew, T. W., Zin, N. A. M., & Sahari, N. (2017). Exploring the affective, motivational and cognitive effects of pedagogical agent enthusiasm in a multimedia learning environment. Human-centric Computing and Information Sciences, 7(1), 1–20.

Liew, T. W., Zin, N. A. M., Sahari, N., & Tan, S. M. (2016). The effects of a pedagogical agent's smiling expression on the learner's emotions and motivation in a virtual learning environment. The International Review of Research in Open and Distributed Learning, 17(5), 249–266.

Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge University Press.

Mayer, R. E. (Ed.). (2014). The Cambridge handbook of multimedia learning (2nd ed.). Cambridge University Press.

Mayer, R. E. (2020). Cognitive theory of multimedia learning. In R. E. Mayer & L. Fiorella (Eds.), The Cambridge handbook of multimedia learning (3rd ed.). Cambridge University Press.

Merkt, M., Lux, S., Hoogerheide, V., van Gog, T., & Schwan, S. (2020). A change of scenery: Does the setting of an instructional video affect learning? Journal of Educational Psychology, 112(6), 1273–1283.

Moreno, R., & Mayer, R. E. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19, 309–326.

Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19, 177–213.

Oker, A., Pecune, F., & Declercq, C. (2020). Virtual tutor and pupil interaction: A study of empathic feedback as extrinsic motivation for learning. Education and Information Technologies, 25, 3643–3658.

Park, B., Knörzer, L., Plass, J. L., & Brünken, R. (2015). Emotional design and positive emotions in multimedia learning: An eyetracking study on the use of anthropomorphisms. Computers & Education, 86, 30–42.

Pi, Z., Zhang, Y., Zhou, W., Xu, K., Chen, Y., Yang, J., & Zhao, Q. (2021). Learning by explaining to oneself and a peer enhances learners' theta and alpha oscillations while watching video lectures. British Journal of Educational Technology, 52(2), 659–679.

Plass, J. L., Heidig, S., Hayward, E. O., Homer, B. D., & Um, E. (2014). Emotional design in multimedia learning: Effects of shape and color on affect and learning. Learning and Instruction, 29, 128–140.

Roselyn Lee, J. E., Nass, C., Brave, S. B., Morishima, Y., Nakajima, H., & Yamada, R. (2007). The case for caring co-learners: The effects of a computer-mediated co-learner agent on trust and learning. Journal of Communication, 57(2), 183–204.

Saerbeck, M., Schut, T., Bartneck, C., & Janse, M. D. (2010). Expressive robots in education: Varying the degree of social supportive behavior of a robotic tutor. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1613–1622). ACM Press.

Schneider, S., Krieglstein, F., Beege, M., & Rey, G. D. (2022). The impact of video lecturers' nonverbal communication on learning—An experiment on gestures and facial expressions of pedagogical agents. Computers & Education, 176, 104350. https://doi.org/10.1016/j.compedu.2021.104350

Shangguan, C., Gong, S., Guo, Y., Wang, X., & Lu, J. (2020). The effects of emotional design on middle school students' multimedia learning: The role of learners' prior knowledge. Educational Psychology, 40(9), 1076–1093.

Sweller, J. (2005). Implications of cognitive load theory for multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 19–30). Cambridge University Press.

Terzis, V., Moridis, C. N., & Economides, A. A. (2012). The effect of emotional feedback on behavioral intention to use computer-based assessment. Computers & Education, 59(2), 710–721.

Tsai, J., Bowring, E., Marsella, S., Wood, W., & Tambe, M. (2012). A study of emotional contagion with virtual characters. In Y. Nakano, M. Neff, A. Paiva, & M. Walker (Eds.), Intelligent Virtual Agents (pp. 81–88). Springer.

Um, E. R., Plass, J. L., Hayward, E. O., & Homer, B. D. (2012). Emotional design in multimedia learning. Journal of Educational Psychology, 104(2), 485–498.

Uzun, A. M., & Yıldırım, Z. (2018). Exploring the effect of using different levels of emotional design features in multimedia science learning. Computers & Education, 119, 112–128.

van der Meij, H. (2013). Motivating agents in software tutorials. Computers in Human Behavior, 29(3), 845–857.

van der Meij, H., van der Meij, J., & Harmsen, R. (2015). Animated pedagogical agents' effects on enhancing student motivation and learning in a science inquiry learning environment. Educational Technology Research and Development, 63(3), 381–403.

Veletsianos, G. (2009). The impact and implications of virtual character expressiveness on learning and agent-learner interactions. Journal of Computer Assisted Learning, 25(4), 345–357.

Wang, Y., Liu, Q., Chen, W., Wang, Q., & Stein, D. (2019). Effects of instructor's facial expressions on students' learning with video lectures. British Journal of Educational Technology, 50(3), 1381–1395.