Autar Kaw, Ph.D.^{1 ✉} , Ali Yalcin, Ph.D.² , Renee M. Clark, Ph.D.³ , Rafael B. Gomes¹ , Luis Serrano¹ , Andrew Scott, Ph.D.⁴ , Yingyan Lou, Ph.D.⁵

1. University of South Florida, Mechanical Engineering
2. Montana State University, Mechanical & Industrial Engineering
3. University of Pittsburgh, Industrial Engineering
4. Alabama A&M University, Electrical Engineering and Computer Science
5. Arizona State University, School of Sustainable Engineering and the Built Environment

1. Introduction

Since the publication of high-profile meta-analyses¹ of undergraduate STEM courses, active learning has become a standard in higher education pedagogy. This meta-analysis by Freeman and his colleagues showed an average effect size of 0.47 for an active learning class vs. a traditional class, an improvement of about half a standard deviation. The follow-up meta-study by Theobald et al.² in 2020 found that active learning also narrows the achievement gaps for underrepresented minorities and low-income groups.

Prince³ reviewed the research on active learning and defined it as an “instructional method that engages students in the learning process.” As opposed to the traditional lecture class, where students listen to the instructor passively, active learning involves student action to engage them in the learning process. Standard active learning methods are collaborative, cooperative, and problem-based.³ Under collaborative learning, one way to provide active learning is through flipped learning, which is defined as “a pedagogical approach in which direct instruction moves from the group learning space to the individual learning space, and the resulting group space is transformed into a dynamic, interactive learning environment where the educator guides students as they apply concepts and engage creatively in the subject matter”.^4,5

A typical flipped classroom involves pre-class, in-class, and post-class learning (Figure 1). The student’s pre-class learning is done individually and includes some combination of video lectures, textbook content, and online assessment but falls under the one-size-fits-all (i.e., non-personalized) approach. The pre-class learning prepares the student for the in-class segment, which involves well-thought-out conceptual and procedural exercises to improve students’ learning levels. The in-class component may also include minilectures to clarify student misconceptions and difficulties with the learning materials. It is followed by post-class learning, which includes completing the topic, solving problem sets from the textbook, and projects to improve students’ higher-level thinking skills.

Figure 1.The Three Components of a Flipped Classroom with And Without Adaptive Learning.

Flipped classes have been found to be more successful compared to the traditional lecture modality. A recent meta-analysis⁶ based on research articles from eight electronic reference databases showed an average effect size^[1] of d=0.24 for cognitive learning in favor of flipped classes over traditional ones. The average effect size on student satisfaction was lower at d=0.16. A meta-study of 63 papers for K-12 students from 2021 by Shao and Liu⁷ showed an average effect size of d=0.63, finding better results for classes smaller than 120 students and humanities courses. Also, a meta-study by Birgili et al.⁸ showed similar increases in student performance and affective outcomes in engineering.

Flipped classrooms are not without challenges. One significant challenge is finding suitable pre-class learning activities to improve student preparation and the subsequent classroom environment and engagement.^9–11 Many students come unprepared to the classroom, adversely affecting the group experience. These challenges were experienced by three of the authors of this paper, who teach a flipped course in Numerical Methods at three separate institutions. To address this challenge of under-preparation with pre-class learning materials and to remedy the one-size-fits-all approach to pre-class learning, we developed adaptive learning lessons using the RealizeIT commercial platform.¹² The in-class and post-class parts remain the same (Figure 1).

Adaptive lessons delivered via online platforms provide personalized and flexible learning by monitoring student progress and performance. Using learning algorithms, the platform subsequently provides an individualized learning path and motivates students optimally. Adaptive lesson platforms (ALPs) have shown their power on a large scale in undergraduate STEM education. For example, using ALPs, Georgia State University reduced the DFW (D and F grades and withdrawals) rate in college algebra from 43% to 21% in a sample of 7,500 students.¹³ In developmental mathematics courses, ASU reduced the DFW rate from 16% to 7% in a sample of 2,000 students.^14,15 Several universities have used one of the commercially available ALPs, called RealizeIT. The University of Central Florida and Colorado Technical University use RealizeIT for over 250 courses.¹⁶ They indicate that the use of the adaptive modality “stabilizes learning organization” in multiple disciplines, that there is a need for widespread collaborative work in adaptive learning, and that it is one possible solution to the shortage of resources in the higher education system.^16–18 More recently, in a mechanical engineering program, RealizeIT was used to support students in traversing the introductory mechanics sequence (i.e., Physics I, Statics, and Dynamics) by providing prerequisite support and elucidating conceptual connections.¹⁹ Performance scores from the ALP arose as significant predictors of subsequent scores on projects and exams.¹⁹

The use of adaptive lessons in flipped engineering classrooms is limited. Kakosimos²⁰ used adaptive learning in a flipped Chemical Engineering Fluid Operations course. However, the control group was from a different course, so a direct comparison of the effectiveness was not possible. The first and third authors of this paper conducted an exploratory study of adaptive learning in a flipped classroom in the Numerical Methods course. For the final examination, a positive effect size of d=0.12 was found for the flipped-with-adaptive lessons group over the flipped-without-adaptive lessons group.²¹ In addition, in a classroom environment inventory, there was a positive effect for flipped-with-adaptive over flipped-without-adaptive for each of the seven environmental dimensions.²² Araujo et al.²³ found that adaptive lessons in a flipped class improved test scores but without statistically significant results.

Given the limited research conducted on the use of adaptive learning in flipped classrooms and the success shown in the exploratory study²⁴ by the authors of this paper, a fuller and more diverse investigation of the effectiveness of adaptive learning for pre-class learning in flipped classrooms is being conducted with two other universities by measuring changes in cognitive and affective impacts on the student.²⁵

But, before the study could start, well-thought-out adaptive lessons had to be constructed. This paper discusses the development, implementation, and revising of the ALP lessons for pre-class learning in a Numerical Methods flipped course and briefly examines the behavior of selected students based on the ALP data.

2. Development of ALP Lessons

In the summer of 2020, three instructors from three universities began developing the ALP lessons for a course in numerical methods under an NSF-funded grant.²⁵ The universities included a large southeastern public university, a small HBCU from a southeastern state, and a large southwestern urban university. The courses were taught to Mechanical, Electrical, and Civil Engineering majors. The work was initially monitored by an external evaluator, who also provided an unbiased assessment of the process.

The first item was to enumerate all topics and break each into individual objectives. The eight topics of the course were the following:

1. Introduction to Scientific Computing

2. Numerical Differentiation

3. Numerical Solution of Nonlinear Equations

4. Simultaneous Linear Equations

5. Interpolation

6. Regression

7. Numerical Integration

8. Numerical Solution of ODEs

Each topic was broken down into chapters, called “objectives” by the ALP platform. There are a total of 30 objectives in the course. For example, for the topic of “Numerical Differentiation,” there are three “objectives” as follows:

1. Prerequisites to Numerical Differentiation

2. Numerical Differentiation of Continuous Functions

3. Numerical Differentiation of Continuous Functions Given at Discrete Points.

Each objective was then divided into individual lessons called nodes (Figure 2). There are 121 nodes for the whole course, out of which 70 fell in the pre-class learning portion of the flipped classroom. For example, we have two pre-class learning nodes for the “Numerical Differentiation of Continuous Functions” objective.

1. Numerical Differentiation of Continuous Functions – First Derivative

2. Numerical Differentiation of Continuous Functions – Second Derivative

Figure 2.Learning Map for a Typical Topic of Numerical Differentiation

If the nodes were being developed for the whole course, “Error Analysis of Numerical Differentiation Divided Difference Methods” would have been added to this objective. However, in our case, this topic is advanced and hence introduced as an in-class minilecture. Exercises related to this node are assigned for in-class and post-class work.

In a prior pilot study at a large public southeastern university, the first author had developed ALP lessons for the pre-class learning for four of the eight topics covered in a Numerical Methods course (i.e., Nonlinear Equations, Simultaneous Linear Equations, Regression, Integration). The lessons learned from the pilot study^21,22 informed the process for the expanded investigation, with lessons being developed for all topics.

The three instructors met biweekly to discuss the content of each node. The main discussion of the meetings centered on what a student would be expected to learn before coming to class, choosing appropriate content, agreeing on prerequisite nodes, and choosing and formulating new assessment questions. Lessons were then created by the first author and his student team using the commercially available platform RealizeIT. The content was tested by learning assistants and instructors. Notably, a significant percentage of the content, such as videos and textbook material, was available through previously funded work.^26–28 The development of new adaptive lessons and the revision of the existing ones from the pilot study were completed in December 2020.

The ALP RealizeIT can be used for any course, both STEM and non-STEM, allowing course developers to provide course content and assessment questions that can be adjusted dynamically to suit each student’s knowledge level and maximize the probability of students learning the material successfully. This platform is designed to help students of different abilities and readiness levels to master the material. Students don’t need to take pre-tests as they can be immediately provided with instructions and practice when they begin their tasks. The material they view is temporally tailored to the course’s learning objectives over the semester. Having completed prerequisite nodes, flipped classroom instructors are assured that their students are well-prepared to work on topics during and after class. In addition, when the student interacts with ALP lessons, gaps in their knowledge are identified, and corrective actions are taken to provide remediation, such as taking the student back to the prerequisites and relevant prior course topics. The ALP tries to imitate what an experienced tutor would do in a one-on-one session. The details of the framework for delivering the ALP lessons are given in Ref,²⁹ while some of the ALP’s intricacies are proprietary.

Each node of the ALP lessons includes five sections (i.e., overview, learning objectives, video lectures, textbook content, and assessment), as shown in Figure 3.

Figure 3.Overview Section, One Of The Five Sections of a Typical Node.

The introduction section includes a brief overview of the topic, while the learning objectives section delineates what the student should know by the end of the node.

The video section consists of relevant lectures (Figure 4). For example, for the “Numerical Differentiation of Continuous Functions – First Derivative” node, the student is presented with three video lectures³⁰ describing the three numerical differentiation methods: the forward–divided–difference method, backward-divided-difference method, and central-divided-difference method. These three videos have a total length of 33 minutes. Each video has a title, a summary, and a learning-objective section. Students can access transcripts of a video as well.

Figure 4.Video Lectures, One Of The Five Sections of a Typical Node.

The textbook content section includes corresponding text to the videos and is taken from the course’s open education resource (OER) textbook.³¹ If an OER is unavailable, an instructor can always assign page numbers from the required textbook for the course. This section serves as an alternative or additional resource to the lecture videos.

The last section of an ALP lesson is the assessment. The assessment design drives most of the success of any learning, including one that is adaptive in nature.^32,33 Since there are 70 nodes in the course just for pre-class learning, the number of questions asked of a student in each node was kept limited. However, there is a need to have a pool of questions available if a student struggles to answer the questions correctly. For example, in the Numerical Differentiation of Continuous Functions – First Derivative node, the question grouping for the assessment is given in Figure 5. One question from each of the three question blocks is presented randomly to the student. Two blocks have multiple-choice questions worth 1 point each, and one has algorithmic questions worth 3 points each. Algorithmic questions have a static template but have variables that are assigned random numbers, which the ALP generates within a range chosen by the developer. Most times, algorithmic questions require more time and a higher level of understanding, hence being assigned more points than the shorter questions. The answers to the algorithmic questions are considered correct if they match the correct integer answer or are within ±1% of the correct non-integer answer. Since the ALP lessons are used only for pre-class learning, algorithmic questions with lengthy solutions are limited. However, after the student has answered an algorithmic question, they can see the final answer and several intermediate answers when warranted.

Figure 5.Question Blocks for A Typical Node

Depending on the learning map, to attempt the next node for which the current attempted node is a prerequisite, a student must receive a minimum score of 59%. We chose a low 59% mark because 60% is a passing D grade for the course. The ALP determines the score and is calculated as content covered (amount of the lesson accessed) multiplied by how correctly the questions are answered. Some parts of these calculations are proprietary, but if it is the first attempt, the prerequisite nodes have been mastered, the number of points scored in the assessment is 59% or more, and the whole lesson is viewed, then the minimum score of 59% has been achieved. A score of 90% is deemed as proficiency and results in a 100% score reported to the LMS. Students can go through the lesson as often as they like. Still, unsuccessful attempts reduce the overall score to discourage guessing. The types of questions in this study’s ALP lessons were limited to static multiple-choice, matching and ordering, and algorithmic questions. The developer can restrict the number of attempts and the amount of feedback for all questions. Other types of questions available in the ALP but not used in this study include answers that are mathematical inputs such as formulas and attachments such as PDF or image documents. The latter must be manually graded, though. We did not use these types because pre-class learning was limited to lower-level skills, and manual grading delays the advantage of immediate feedback.

The questions are available in the most widely used and adopted Question and Test Interoperability (QTI) format.³⁴ Other materials used in the ALP lessons, such as introduction, learning objectives, textbook content, and video lectures, are available freely on the web^30,31 via the Creative Commons License. The materials can be shared, revised, and adapted as users wish, barring commercial use.

The ALP has several other available learning and question items that RealizeIT calls Bits.²⁹ Bits items available include interactive examples, summary, and review. Again, we did not include these because our ALP lessons were limited to pre-class learning.

Any reader who is an instructor of numerical methods can get the collection of questions for the whole course by contacting the first author.

3. Implementation of ALP Lessons

The adaptive lessons were implemented and initially tested in the classroom during Spring 2021 at the first author’s university before implementation at the other two universities. The student demographics included 20% female, 80% male, 28% who had transferred from a community college with an associate degree, 31% from underrepresented minority (URM) groups, 42% who work 20 hours/week or more, and 22% who were Pell Grant recipients (low socioeconomic group).

The lessons accounted for 15% of the student’s final course grade in the first author’s course. Each of the 30 objectives was presented as an assignment to students and was linked via the CANVAS learning management system. Each objective was released on a Thursday afternoon (after class meeting time for that week) and was due on a Tuesday afternoon, 11 days later and before the beginning of class time for that week. This 11-day period gave students two weekends in between, as 42% of our students work 20 hours or more per week, and the weekend is their time to catch up on schoolwork. They could ask questions during office hours, on the LMS discussion board, and via email.

The ALP automatically transferred the scores obtained on each objective to the CANVAS LMS an hour after the deadline. The ALP lessons remained accessible for all students until the end of the semester. The ALP lessons follow the W3C accessibility standards,³⁵ including transcripts for videos, alternative textbook content to replace and augment video content, use of LaTeX for readability of equations, and alternative text for figures.

During class, the in-class activities were based on extending the pre-class lessons due each Tuesday. In Figure 6, the dashboard is shown for a typical objective. The status shows how many students have finished the module and the number of students who have not started. Also, several students were repeating the course, and the dashboard pointed out if they were showing improved performance in the repeating semester. The knowledge state measures student ability via how many questions in the nodes a student can answer. The knowledge covered measures student progress via how many nodes a student has gone through. The time spent, the composite score based on the product of the knowledge state, and the knowledge covered are also reported. Anyone with a composite score of more than 90% gets a reported score of 100% on the LMS for the objective. This rule is used to avoid unnecessary extra attempts by the student to aim for a perfect score when that time could be used to solve textbook problem sets from the previously covered objectives.

Figure 6.Dashboard of a Typical Objective in The Adaptive Learning Platform.

The instructor looked at how students responded to ALP questions several hours before class. See Figure 7 as an example whereby a particular question had more incorrect answers than correct ones (i.e., 58 vs. 56). This subsequently informed the nature and content of the in-class minilectures and exercises.

Figure 7.Instances Of Correct and Incorrect Responses to Questions in A Node.

Other analytics available from the ALP include the names of students who have not started or are struggling, their knowledge state (i.e., how well they know the content), and the knowledge covered (i.e., how much content they had gone through). Those who are struggling or have not finished the lesson receive an email from the instructor encouraging them to seek help during office hours or, better, remain on schedule.

4. Revising of ALP Lessons

Before implementation in Spring 2021, the ALP lessons were tested by the three instructors and two undergraduate research assistants who had recently taken the course. The lessons were checked for content and accuracy. Feedback in the Spring 2021 semester was collected via the following.

1. Questions asked by students during office hours.
2. Posts by students on the CANVAS LMS discussion board.
3. Emails sent by students to the instructor regarding ALP lessons.
4. Open-ended questions asked about the adaptive lessons in the end-of-semester survey. Questions included the following.
   1. “What drawbacks did you perceive with this flipped classroom, and what suggestions do you have for improvement, including relative to the adaptive lessons?”
   2. “Discuss the adaptive lessons relative to your learning or understanding of course content, satisfaction as a student, and engagement with course content.”
5. Questions asked in two focus groups conducted by an independent assessment analyst. Questions included the following.
   1. “Did the adaptive lessons impact your learning or understanding more so than other methods or resources (that you use) for studying, learning, or reviewing content? Why do you feel this was the case?”
   2. “Were there other benefits or good outcomes related to the adaptive lessons in this course?”
   3. “Were there disadvantages, challenges, or negative outcomes related to the adaptive lessons in this course, and if so, do you have suggestions for changes?”

The above feedback was then used to revise the adaptive lessons during and after the Spring 2021 semester. Comprehensive revisions to the ALP lessons included the following.

1. Some students disliked the low resolution (240p) of the video lectures that had been recorded in the mid-2000s. All seventy-three videos in the ALP lessons were rerecorded with 1080p HD quality.
2. The textbook content format for each node was initially an embedded PDF file. This format was acceptable for a PC or a wide tablet but negatively affected font sizes and scrolling expectations (horizontal and vertical) for smaller tablets and mobile phones. Therefore, all original MS Word files of the textbook content were converted to a markdown language, allowing formats such as HTML to address the abovementioned issues, improve quality, and meet web accessibility standards.
3. Intermediate answers for more algorithmic questions were given as feedback so that students could identify where they went wrong. For example, if a question involves two iterations, intermediate solutions from the first and second iterations provide better feedback to the students in correcting their mistakes in the next attempt.
4. Some questions were perceived as challenging to understand by students. They were revised by breaking them into smaller sentences and clarifying what was being asked.
5. Some questions critical for pre-class learning and at a slightly advanced level were augmented with practical hints.
6. Without having to rewatch a video lecture, screenshots of a lecture in a single PDF file were added to provide quick access to its content when students are practicing, reviewing, or reattempting a node.

After these revisions, one undergraduate research assistant from two of the three universities tested all the lessons again in the Summer of 2021.

5. Case Study of Student Interactions with a Node

In this section, we demonstrate how we can begin to explore the student ALP data to study student behavior and course outcomes when engaging with the ALP lessons. The data is presented to explore the research question: How do students who earned an A, B, C, or D grade in the course differ in their behavior in approaching the ALP lessons? The four cases (i.e., students) were chosen to illustrate behaviors of interest amongst students who scored A, B, C, and D letter grades. These behaviors were not studied with inferential statistics, as that is not the intention of this paper. The four cases are meant to present the ALP data acquired, its potential interpretation and usage, and indicate prospective study areas.

We use the “Numerical Differentiation of Continuous Functions – First Derivative” node as an example and illustrate how a student from each group (i.e., A, B, C, and D final course grade) interacted with the node. This node was made available on January 15, 2021, and was due by January 26, 2021, for credit toward the final course grade. A graded test that included this node was administered on February 5, 2021. The node remained available to students until the end of the Spring 2021 semester on May 8, 2021. Our estimate regarding how long a student might spend on this node was as follows: Introduction – 2 minutes, Objectives – 4 minutes, Videos – 33 minutes, Textbook Instead of Videos – 20 minutes, Textbook and Videos – 45 minutes, and Assessment Questions – 20 minutes. These amount to a total time of 45-60 minutes to complete the node.

Two data types related to individual student engagement with nodes were collected: participation data and activity data at the more aggregate level. Participation data shows the duration of students’ engagement with the content within a node, such as the introduction, learning material, and questions. An activity constitutes one or more participations within the node. It may be viewed as a “sitting” or “attempt” at completing the contents and requirements of a node by an individual student. The ALP collects time data associated with participation, which is then summed to determine activity time. In addition, each activity is evaluated using a feature called “normresult,” which is the platform’s evaluation of the student’s performance for the node. The normresult score is scaled to a value between 0 and 1. A normresult of -1 indicates an abandoned activity or an activity for which there were no assessment questions; hence, no performance evaluation was possible. An activity left without attempting assessment questions does not negatively affect student scores. A NormScore of precisely 1 means that the learner scored full points in the node.

For the “Numerical Differentiation of Continuous Functions – First Derivative” node, the ALP recorded 237 distinct activities during the 2021 Spring semester for all the students in the course. Figure 8 shows a Box-and-Whisker plot of the activity durations. Compared to our time estimate of 45-60 minutes for this node, we noticed that the median time spent by students was lower at 34.2 minutes.

Figure 8.Distribution of Time of Activity Events for The Numerical Differentiation of Continuous Functions – First Derivative Node

However, it is essential to note that the reported individual activity times must be carefully interpreted. Many of these activities do not represent meaningful interactions between the student and the content. For example, students repeating the content in a node may quickly skip over the introduction and objectives sections and spend time on the text, videos, and questions, leading to times below our estimate for the node. Disproportionately long activity times may also be recorded if a student abandons the node but does not close the browser window.

The activity time for this node was broken down by the day before the due date, as shown in Figure 9. Considering the due date of January 26, 2021, these results align with the expectation that most students access and complete the content immediately before the due date.

Figure 9.Activity Time by Date for The Numerical Differentiation of Continuous Functions – First Derivative Node.

We will focus on the activity and participation data of four individual students we refer to as A, B, C, and D. The letters A, B, C, and D also correspond to the overall course grade they received at the end of the semester. The data collected by the ALP related to the activities of these students for this node are shown in Table 1. Similarly, the participation data for these students is shown in Table 2.

Student A has one activity recorded for this node, which has a duration of 46.7 minutes and a Normscore of 1.0. This record means the student completed the requirements of this node in one attempt with the maximum possible score. Within this activity, the student spent most of their time (approximately 40 minutes or 2324 seconds) on learning material and 7 minutes (380 seconds) correctly answering the three required questions on the first attempt. Note that the activity time is within our estimated 45-60 minutes range.

Student B has two activities reported for the node. The first activity was 41.0 minutes long and was completed with limited success (Normscore of 0.6) in answering the questions in the node. The student spent 6 minutes (359 seconds) on the questions. Of particular interest here is the time spent in the introduction part of the node, which is approximately 35 minutes (or 2092 seconds). However, the content in that part of the node should only take a few minutes.

In contrast, the student spent only 4 seconds on the learning materials in the node. Thus, the student’s behavior did not align with the instructor’s intent. The second activity started immediately after the first one, and the student spent 2.0 minutes (115 seconds) on the assessment questions in the node and completed the node’s requirements with Normscore=1.

Student C has four consecutive activities that are relatively short, namely, 2.7, 13.9, 5.6, and 4.3 minutes. The student abandoned the first three activities, as indicated by the Normscore of -1. The student spent almost no time on the learning materials in any activities, with most of their time on the questions. After three unsuccessful attempts, Student C completed the node requirements on the fourth attempt.

Student D has one activity that is short, around 14 mins, and completed the node with limited success (Normscore of 0.6) in answering the questions in this node and did not try again to improve his score. The student spent almost no time on the introductions and learning materials, spending virtually all their time on the questions.

The ALP does not capture the exact nature of what the students do while interacting with the nodes. However, looking at the duration of the various elements of the nodes, we see a distinct difference between these students as follows:

1. According to expectations and instructor intent, student A spent much more time on the learning material than Students B, C, and D.

2. Student A answered questions correctly on the first attempt.

3. Student C had several abandoned activities or unsuccessful attempts at answering the assessment questions.

4. Student C utilized a “trial and error” approach to getting the correct answers for the questions instead of exploring the learning material content.

5. Student D also did not spend any time exploring the learning material content and spent all their time on questions. Most interestingly, this student did not take the opportunity to improve their score.

These distinctions may show that it may be better for students to spend reasonable time on the learning material, such as the lecture videos and textbook content, before jumping to answer the assessment questions. These observations are being used to inform struggling students about how they should interact with the ALP lessons.

Studying individual student behaviors may become intractable as we have 70 ALP lessons and as many as 100 students enrolled in a typical semester. We will take the clustering³⁶ and principal component analysis³⁷ approach in future work. With clustering, we will identify homogenous groups of students whose learning behavior, as described by their interaction with the ALP, is associated with similar performance in the course as measured by the final course grade. With principal component analysis, we can identify which student behaviors are most influential in affecting their overall performance in the course. This data analysis can inform students how to improve their study habits for better performance. More importantly, we will begin using this analysis to support students as early as the first two weeks of the semester. This support would include one-on-one advising and tutoring sessions.

6. Conclusions

One way to provide active learning is through the flipped classroom. However, finding suitable pre-class learning activities to improve student preparation and the subsequent classroom environment, including student engagement, can challenge the flipped modality. To address this challenge, adaptive learning lessons were developed for pre-class learning for a course in Numerical Methods. This paper discussed developing, implementing, refining, and revising the adaptive learning platform (ALP) lessons for pre-class learning in a Numerical Methods flipped course.

Three instructors who teach Numerical Methods collaborated on developing the entire course’s adaptive lessons. The work began in the Summer of 2020 by enumerating the various chapters and breaking each into 30 individual objectives (assignments), which were then divided into individual nodes (lessons). Each lesson includes five sections: introduction, learning objectives, video lectures, textbook content, and assessment. The three instructors met twice a month to discuss the content that provided the basis for each lesson. The main discussion of the meetings centered on what a student would be expected to learn before coming to class, choosing appropriate content, agreeing on prerequisites, and choosing and making new assessment questions. Lessons were then created using the commercially available platform RealizeIT. The lessons were tested by learning assistants and instructors. The adaptive lessons were completed in December 2020.

The adaptive lessons were implemented and initially tested in the classroom during the Spring of 2021 at the first author’s university. Questions asked by students during office hours, on the LMS discussion board, and via emails while doing the lessons were used to update ALP content, clarify questions, and revise hints offered by the platform. Other revisions included video lectures rerecorded for HD quality, textbook content in HTML format for access via multiple platforms, and the addition of screenshot transcripts of videos.

ALP data of four individual students were discussed to show the unique ways students interact with an ALP lesson. For example, students with lower grades of C and D were found to spend less time on learning materials and more on assessment questions. The C student was making unusually more attempts to master the topics or even sometimes abandoning the lessons. In contrast, the D student did not take the opportunity to master the material and settled for a lower score after one attempt. Recognition of such unique student behaviors may inform the study habits of all students. Also, how all students responded to specific assessment questions in the ALP helped the instructor guide the in-class exercises and minilectures of a flipped classroom.

Acknowledgments

This material is based upon work supported partially by the National Science Foundation under Grant Number 2013271 and the Research for Undergraduates Program in the College of Engineering at the University of South Florida (USF). The authors would like to thank Colm Howlin, Sean O’Shea, and John Healy of RealizeIT for their help during the development and implementation of the adaptive lessons. We also thank Dr. Sarah Zappe of Zappe and Cutler Educational Consulting for her feedback on the manuscript. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. The building, implementation, and revisions of the adaptive lessons in a flipped classroom were first presented at the 2022 ASEE national conference.