Abstract

This study explores the impact of ChatGPT, an AI tutor, on enhancing undergraduate learning in multimedia environments that combine verbal and visual information. Using a between-subjects design, it compares three groups: Pure Learning, Self-Study, and ChatGPT-assisted learning. The research assesses whether ChatGPT improves understanding, retention, and academic performance compared to traditional and self-study methods, with findings based on quantitative post-tests and qualitative feedback to evaluate the benefits and limitations of AI-assisted learning.

1 Introduction
1.1 Background and Significance

The advent of Artificial Intelligence (AI) has introduced innovative tools that are reshaping educational practices by offering personalized and adaptive learning experiences. Among these tools, large language models like ChatGPT have gained prominence for their ability to engage in human-like conversations, provide instant feedback, and assist in understanding complex concepts within multimedia learning environments. Multimedia learning, which integrates verbal explanations with visual aids, is grounded in cognitive theories that emphasize the importance of dual coding for effective comprehension and retention. By integrating ChatGPT into multimedia learning, educators can potentially enhance interactivity, personalize learning experiences, and support cognitive processes that facilitate deeper understanding. 

Furthermore, undergraduate education frequently encounters challenges related to varying student learning paces, diverse academic backgrounds, and the necessity for individualized attention. Traditional classroom settings may not adequately address these challenges, potentially leading to disparities in academic outcomes. AI tutors like ChatGPT present an opportunity to bridge this gap by offering tailored assistance, fostering deeper comprehension, and enhancing retention of academic knowledge.

1.2 Gaps in the literature

While AI tools have shown promise in educational settings, there is limited research on their effectiveness in enhancing undergraduate students' learning experiences and academic performance specifically within multimedia learning environments. Moreover, the comparative efficacy of AI-assisted multimedia learning versus traditional and self-study multimedia methods has not been thoroughly explored. This study seeks to fill these gaps by providing empirical evidence on ChatGPT's role in augmenting students' learning experiences and performance in multimedia learning contexts.

1.3 Problem Statement

Despite the growing adoption of AI tools in educational settings, there is limited empirical evidence regarding their effectiveness in enhancing undergraduate students' learning experiences and academic performance within multimedia learning environments. Specifically, questions persist about whether AI-assisted multimedia learning tools like ChatGPT offer significant advantages over traditional and self-study multimedia methods, and what specific roles these tools play in facilitating academic success.

2 Research Questions and Hypotheses

2.1 Research questions

1. How does the incorporation of ChatGPT as an AI tutor influence undergraduate students’ academic performance and learning experiences in multimedia learning environments?

2. In what ways does ChatGPT enhance student engagement and interaction with multimedia learning materials compared to pure learning and self-study methods?

3. What are the perceived benefits and challenges of using ChatGPT as a learning tool within multimedia learning contexts from the students' perspective?

2.2 Hypotheses

1. Students in the ChatGPT group will have significantly higher post-test scores in academic performance compared to those in the Pure Learning and Study Oneself groups.

2. Students in the ChatGPT group will report higher levels of engagement and satisfaction with the learning process compared to the other groups.

3. Students will perceive ChatGPT as a valuable tool for clarifying complex concepts and enhancing their overall learning experience.

3 Methods

3.1 Research Design

A between-subjects experimental design will be employed to compare the effects of three different learning conditions on students' performance and experiences. The study will involve random assignment of participants to one of three groups: Pure Learning, Study Oneself, and ChatGPT-assisted learning.

3.2 Participants

The study will recruit [x] undergraduate students from University of California, Santa Barbara, ensuring a diverse sample in terms of academic backgrounds and prior knowledge. Participants will be randomly assigned to one of the three experimental groups, with [x/3] students per group.

3.3 Materials

1. Multimedia Lectures: Developed based on foundational academic subjects or scientific phenomena such as Lightning, Airplane, Physics, Chemistry, Geography, etc. Each lecture will incorporate both verbal explanations and visual aids, adhering to Mayer’s principles of multimedia learning. Both the lecture slides and transcripts from the lectures will be prepared and provided to students for learning. 

2. ChatGPT Interface: Design and set up a unique agent powered by ChatGPT with specific guidelines according to the research requirements. Configured to interact with students regarding the lecture content, allowing for question-and-answer sessions, clarifications, and follow-up inquiries. 

3. Pre-Test: an assessment administered prior to the intervention to test baseline understanding.

4. Post-Test: A standardized assessment comprising multiple-choice and short-answer questions designed to evaluate comprehension and retention of the lecture materials.

5. Post-Questionnaire: A questionnaire designed to learn about students’ learning experiences, feedback, and levels of engagement and satisfaction within the learning process.

3.4 Procedure

1. Pre-Test

   1. All participants will complete a pre-test assessing their initial understanding of the selected academic subjects. This will serve as a baseline to control for prior knowledge and enhance the validity of the study.

2. Intervention

Participants will be randomly assigned to one of the following groups:

1. Pure Learning Group

   1. Receives only the multimedia lecture.

2. Study Oneself Group

   1. Receives a multimedia lecture.

   2. Generates three questions based on the lecture content.

   3. Independently studies the provided transcripts and slides to answer these questions within three minutes.

   4. Document their answers.

3. ChatGPT Group

   1. Receives a multimedia lecture.

   2. Generates three questions based on the lecture content.

   3. Interacts with ChatGPT using the transcripts and slides to answer these questions within three minutes, including follow-up questions and clarifications as needed.

   4. Document their answers.

3. Post-Test

   1. After the intervention, all participants will complete the post-test to measure their comprehension and retention of the lecture materials.

4. Post-Questionnaire

   1. Complete the questionnaire to learn about students’ learning experiences, feedback, and levels of engagement and satisfaction within the learning process.

4 Data Collection

1. Quantitative Data: Scores from the pre-test and post-test will be collected to measure learning gains.

2. Qualitative Data: Responses from the Study Oneself and ChatGPT groups will be analyzed to assess depth of understanding, engagement levels, and perceived usefulness of the learning methods.

5 Other Materials

1. Consent Form: the study’s purpose, procedures, risks, benefits, and participants rights.

2. ChatGPT Interaction Guide: guidelines for interacting with ChatGPT, including example prompts and expected types of responses…