Introduction

Imagine standing in a large lecture hall and glancing around to gauge whether students are grasping the lecture concepts. However, rather than observing students nodding in agreement or shaking their heads in confusion, the raised lids of open laptop computers greet you. Instead of garnering a quick comprehension check, you are left wondering, “Are students paying attention or are laptops hurting learning?” As large classes become more prevalent and schools increasingly implement college-wide computing initiatives, this is the reality for numerous instructors. From one-to-one initiatives (Hayhurst, 2018; Richardson et al., 2013) to Bring Your Own Device (BYOD) requirements (Siani, 2017) in both K-12 and higher education, personal computers are embedded into educational contexts. Personal computers have been heralded for enabling interaction and supporting technology-centered instructional activities such as electronic content delivery, interactive polling, course management, and interactive software mentoring (Campbell & Pargas, 2003; Tront, 2007). However, the advantages of laptops in the classroom are often accompanied by the disadvantage of student inattentiveness. Laptops allow students to engage in media multitasking, swapping between a myriad of distracting activities including social media, gaming, and email (Langan et al., 2016; Wammes et al., 2019). Additionally, students who engage in off-task laptop activities distract neighboring students (Hall, Lineweaver, Hogan, & Brien, 2020).

Over the years as technology has become increasingly embedded into classrooms, researchers have tried to provide instructors with data to understand how laptop usage in the classroom impacts learners. From personal digital assistants to laptops, tablets and cell phones, researchers have documented both positive learning effects (Barak, Lipson, & Lerman, 2006; Doolen, Porter, & Hoag, 2003; Lohani, Castles, Lo, & Griffin, 2007; Roth & Butler, 2016; Samson, 2010; Shaw, Kominko, & Terrion, 2015) and negative learning effects (Carter, Greenberg, & Walker, 2017; Fried, 2008; Hembrooke & Gay, 2003; Junco, 2012; Kraushaar & Novak, 2010; May & Elder, 2018; Wood et al., 2012; Zhang, 2015) related to personal technology usage in the classroom. With no clear consensus regarding the impact of personal technology on learning in the classroom, researchers continue to tease out details of how student computer usage impacts learning by, for example, quantifying the amount of off-task activity in classrooms (Ragan, Jennings, Massey, & Doolittle, 2014), examining how non-academic applications like Facebook are used by students during lectures (Judd, 2014), and investigating how laptop bans impact learning (Elliott-Dorans, 2018). One trend that has emerged is that, for classrooms where instructors structure students’ computer use, learning impacts are typically positive (Downs, Tran, Mcmenemy, & Abegaze, 2015; Kay & Lauricella, 2011). That is, when instructors design the course to incorporate purposeful and deliberate computer usage, the impact of computer usage on learning tends to be positive. When student computer use is unregulated, research results are both positive and negative with regard to learning impacts. This finding should motivate instructors to embrace technology in their classrooms and learn how to use technology for their advantage.

One powerful tool that would allow instructors to use student computers to their advantage is a system that would allow instructors to see the pulse of large lectures. By capturing data that is similar to but more accurate than glancing around the classroom in order to gauge who is engaged, instructors could react in real-time to encourage more participation from students. Instructors could use a learning pulse monitor to time instructional interventions to promote active, engaged learning. We hypothesize that the active window information from student computers could provide the requisite data for determining a real-time classroom learning pulse. This study uses observational research in two different large engineering lecture courses over one semester in order to quantify the amount of error for using active window as a proxy for student attention. Then, we capture active window data electronically to illustrate how a learning pulse monitor could provide actionable information to an instructor for both real-time intervention or post-lecture reflection in order to improve instructional practices.

Why Active Window?

In information processing theory, there is a strong, direct link between attention and learning. This direct link is very clear in a quote from Mackintosh (1975), “The probability of attending to a stimulus determines the probability of learning about that stimulus” (p. 294). More recent studies have reached similar conclusions: humans learn about items that they attend to (Mitchell & Pelley, 2010). Robert Gagne has been credited with shifting the information processing discussion from the research lab to the practical realm of instructional design with his introduction of the Conditions of Learning (Gredler, 1997). Gagne’s (1965) original theory stipulates that there are nine instructional events that must occur for learning to take place, the first of which involves obtaining the learner’s attention. The instructional events do not guarantee learning will occur, but rather they support the learner’s internal mental processes. That is, each event is a necessary condition for learning to take place. While the theory has evolved somewhat since its introduction (Gagne, 1965; Gagne, 1977), attention has remained an initial event.Fleming (1987) succinctly explains why: “Quite simply, without attention [the first event] there can be no learning” (p. 236). To support computer users, who are students in our contexts, some suggest a need to design attention-aware systems that delay interruption by deferring alerts unrelated to the task at hand (Bailey & Konstan, 2006). Instead, we focus on how attention-aware systems could support instructional design. Specifically, we hypothesize that a students’ top-most, active window can be used to make a determination of that student’s attention and provide real-time data for instructors’ decision making.

Current assessment strategies used to measure student computer use in classrooms are limited. Existing research studies have explored student attention through either self-reported survey data, internet activity monitoring or through the installation of Spyware software. Survey data, by far the most common data collection method, does not provide the data resolution needed to generate a real-time learning pulse. Internet monitoring provides an incomplete characterization due to missing data related to non-internet activity (e.g., local applications). Spyware would provide the requisite data; however, significant privacy concerns and installation issues have plagued studies attempting to utilize spyware (Kraushaar & Novak, 2010; Kraushaar, Chittenden, & Novak, 2008). By relying on active window data, particularly data captured as a binary on-task and off-task determination, we attempt to balance student expectations for privacy and the need for capturing real-time data.

In courses that use classroom learning technology in order to communicate with student laptops via a server (e.g., DyKnow Vision or Classroom Presenter), active window data could be captured directly through the software. Specifically, if a student’s top-most window contains the course material, the student is paying attention (i.e., on-task) (Figure 1). If any other application is the top-most, active window (e.g., Figure 2 , Figure 3 ), the student is not paying attention (i.e., off-task). The assumption that active window indicates attention to or distraction from lecture has been previously implied (Hembrooke & Gay, 2003; Kraushaar & Novak, 2010). However, the assumption has not been directly tested for reliability. It is clear there is error associated with the method. For example, consider the layout in Figure 3 where the classroom software and another application split the screen. While the active window (the window with the mouse focus) is not the course software, it is possible that the student could be paying attention to the lecture. Similarly, it is possible that in Figure 2 , the student has non-course software as the active window, but is viewing the slides on the classroom projector. There is a need to quantify the amount of error with the active window method to understand if active window data can provide actionable information. This study uses observational data of student computer usage within classrooms to quantify the error associated with the active window method.

Method

Observations

We used observations of students’ behavior to collect student attention data and information about active windows on student computers. Direct observations of student behavior are a frequent and recognized method for determining student attention in educational, behavioral, psychological research studies (Hoge, 1985; Rapport, Kofler, Alderson, Timko, & Dupaul, 2009). In determining attention, observations may focus on general behaviors, such as “on-task”, or specific behaviors, such as “playing with an object”. Focusing on general behaviors is recommended since significant and consistent evidence exists for the validity of general measures (Hoge, 1985). We used in-class, naturalistic observations, which are unobtrusive, covert observations during which the observer blends in with participants and does not affect behavior. Students were not informed that they were being observed in order to capture typical, unchanged student behavior.

Observations were conducted each week of the semester during FYE and S&D lectures. To increase validity of our estimates as a representation of total error rates, we selected students to observe using stratified random sampling. That is, we divided the class into sections (e.g., front, back, middle) and randomly sampled from each of these areas. Prior to the start of lecture, the observer would sit in a random location in the classroom, and select students whose computer screens were visible. To avoid data overlap due to neighboring students interacting, the selected students could not be sitting next to each other. Observations were conducted on a Tablet computer similar to students’ computers and the screen was shielded from nearby students. Throughout the semester, observers reported conversations with neighboring students that indicated their presence remained undetected (e.g., neighbors asked homework questions such as “What did you get for question 3?”).

Observers were trained and used an observation protocol to strengthen reliability. Figure 4 shows the observation protocol with sample data. The protocol guided observers to document student activity (Notes), the observer’s perception of student attention (A?), and the students’ top-most, active window (Window) at every minute during the lecture. Generally, a student was considered attentive if they were looking at course content or the instructor, discussing course content, working on instructor-assigned tasks, or listening to the instructor. In other words, a student was classified as on-task or attentive if they were participating in teacher-sanctioned activities (Hoge, 1985). For validity and reliability purposes, after each observation was completed, the protocol “Notes” field and the judgement of attention columns were reviewed by the research team.

Electronic Active Window Monitoring for Classroom Pulse

There are a variety of ways active window monitoring could be implemented. In our case, we approached the developer for the interactive learning software used at our study site and asked them to generate the data. They incorporated a visual attention widget into the instructor panel. The widget provided instructors with a visual representation of student attention by monitoring students’ active, top-most window (Figure 5 ). The software assumed that, similar to the measured student attention from the observation protocol, if the active window on a student’s computer was the learning software, then the student was on-task. All other active windows indicated off-task behavior. We created a record of the widget’s output with screen capture software. We then processed the recordings with MATLAB’s image processing toolbox to create a spreadsheet file for analysis.

Active windows were measured every 12 seconds for the entire lecture. For each time, average class attention was calculated by dividing the total number of attentive students (e.g., course software as top-most, active window) by the total number of students logged into the course software (Equation 1 ). The average class attention timeline was supplemented with information from observation notes and an audio recording of lecture in order to create a descriptive class timeline (i.e., start of class, start of homework review, start of new lecture material, start of practice problems related to new lecture material).

1 $Average Class Attention= \frac{total students with DyKnow active}{total students logged into DyKnow} *100 \left(1\right)$

Results

Estimate of Mean Error Rates

Based on the bootstrap analysis of the FYE data (Figure 6 , left), the mean percent error is 4.28% and the estimate of standard error is 0.82. The 95% confidence interval for FYE percent error is [2.81%, 6.04%]. The bootstrap analysis of the S&D data (Figure 6 , right) produced a mean percent error and standard error of 6.89% and 1.51. The 95% confidence interval for S&D percent error is [4.42%, 10.17%].

Real-time Electronic Classroom Pulse

Active window records were electronically captured from 135 students in one 75-minute Statics lecture every 12 minutes. The percentage of class time spent in the course software was calculated for each of the students and the frequency distribution for all students is shown in Figure 7 . The average percentage of on-task time varies across the entire frequency range. Twenty-eight students were in the 90-100% category indicating they remained in the course software nearly the entire course. Fourteen students were in the 0-10% category indicating they were logged into, but not using the course software for nearly the entire course. The remaining 93 students engaged in multitasking (i.e., switching between application windows).

The average class attention for the Statics lecture is plotted in Figure 8 . The timeline is annotated based on a recording of the course. The instructor reviewed homework problems starting at 3:43pm and worked practice problems starting at 4:18pm. During the homework review and practice problem sessions, gray shading is used to indicate the start and end of different problems. By annotating the average class attention time line in Figure 8 we see a clear indication that instructors effect student attention. When new material was presented, there were peaks (e.g., local maximums) in attention. Furthermore, instructor statements such as “Pay attention” also promoted attention, but the duration was short lived. Randomly calling on students while working on practice problems may be a method of returning students to lecture, but another method must be used to prolong the increased engagement, as students would return to off-task activities when they realized that they were not selected.

Discussion

The primary purpose of this study was to examine the validity of using students’ top-most, active window as a proxy for attention. Asserting that there is an average of 4.28% or 6.89% error depending on course type, this study provides strong evidence that active window can be a valid proxy for average classroom attention. Obviously, the final determination of acceptable error rates for future contexts should be made in consideration with the specific research or pedagogical questions under investigation. However, to give the reader perspective, until now, instructors primarily obtain student computer usage with surveys. Even with a high response rate, survey data can be inaccurate because student’s memories may not match reality (Brener, Billy, & Grady, 2003) or grade-oriented students may underreport negative behavior 40 . Kraushnaar and Novak’s 19 investigation directly comparing students’ self-reported computer use to computer use monitored by Spyware established that students underreported instant messaging use by 40%. Only 25% of students reported using instant messaging programs during class, but the Spyware record captured instant messaging use by 61% of the class. The error rates established in our investigation for the active window method are significantly less than the error rate for instant messaging self-reports established by Kraushnaar and Novak.

While active window error rates are acceptable for an application producing a general classroom pulse, they may not be acceptable for applications requiring less error. For example, the active window method may not be appropriate for assigning participation grades. Data collection in this study occurred across two distinct types of computer-infused classrooms. Student characteristics given in Table 1 and sources of error in Table 2 allow for informed decisions regarding the appropriateness of using the active window technique in classrooms that differ greatly from the study context. For example, if an instructor has observed classroom behavior similar to activities in Table 1, Table 2 , the active window method may be considered appropriate. In the future, researchers and instructors can consider how technology is used in their classroom contexts to estimate the error for their particular situation.

Our decision to treat FYE and S&D as two different groups due to the use of technology was supported by the sources of error observed in the two groups. The primary and secondary reasons for mismatches in one group were not the primary or secondary reasons for the other group. In FYE courses (structured use), students were more likely to leave the course software active and use a secondary device. In S&D (semi-structured use), students were more likely to log into the course software and then switch to off-task activities on the same device. Then, S&D students would “check-in” with lecture by glancing up at the instructor and projector; students appeared to use the projector as a second monitor, and would only switch back to the course software if they decided to re-engage with the lecture by, for example, taking notes on the instructor distributed slides. The distinction in error types between FYE and S&D courses did not appear to be related to content or instructor, as reasons for error were similar across different sections, course timing, instructors, and weeks. Instead, the differences in error appear to be related to the instructor’s use of technology. Our results provide some evidence that instructional methods produce different student behaviors in computer-infused classrooms. Future studies could investigate whether this behavior is a natural change that occurs as students progress through degree programs, or if it is more directly related to technology policies for a course.

To further explore our hypothesis that an active window monitor could serve as a classroom pulse to generate actionable feedback for the instructor, we paired near real-time active window data (i.e., data collected every 20s) with recordings of courses lectures. Our results suggest an active-window based classroom pulse could provide insight for both real-time intervention or post-lecture reflection. For example, a real-time intervention based on the timeline in Figure 8 could be related to waning attention during the middle of the lecture, approximately 4:07pm to 4:18pm. The pulse could alert the instructor that an active exercise should be executed to reengage students with lecture content. As another example, upon reflection of the course at the end of the semester, the instructor could be motivated to reduce homework review time (low attention approximately 3:43pm to 3:54pm) in future semesters. The extra time could be dedicated to working practice problems as students appear to be more engaged during that portion of the lecture. Based on our initial analysis conducted as part of the study and reported in this paper, a classroom pulse generated from active window data can show an instructor how their pedagogical techniques directly affect student engagement. We are conducting additional studies to determine how instructors respond to the data and how that subsequently effects student engagement and learning.

There are two primary limitations of the active window method. First, the active window method only provides information on whether a student is in course software or not. Second, the active window method can only be used in courses where there is a clear datum for on-task. Essentially, these two limitations combine to mean that a researcher cannot use the active window method without knowing the context of the computer use. As an example, if a portion of the lecture required students to complete an exercise on paper, then the computer active window would not be an indication of the classroom pulse. However, the instructor would be aware of this activity and surely not consider the pulse as an indication of attention.

Conclusion

As evident from the popularity of studies examining student computer use, instructors want to understand how students are using their computers in classes. We examined the appropriateness and application of monitoring the active window on student computers as a means for providing a pulse of classroom attention. To quantify error for the active window method, we observed students in two course types, structured computer use and semi-structured computer use. We quantified both false-positive and false-negative error for active window monitoring through observations of unmanipulated student behavior. The observations provided a listing of behaviors that observers classified as attentive or inattentive and a listing of behaviors that were associated with error. These listings will provide evidence to inform decisions as to whether the active window method is appropriate for alternate contexts.

In courses where students are required to use interactive learning software, electronically captured active window data has the potential to produce a real-time attention record for every student, as well as the average class attention, essentially creating a pulse for the classroom. By implementing data collection through existing interactive learning software, the method was much less invasive than spyware installations – data were only recorded during class times and no additional software was required. Active window monitoring has the potential to inform the timing of real-time instructional intervention and to help instructors improve their practice through post-lecture reflection.

Appendix