Innovative VR-Based Research to Develop Intuitive Human Computer Interaction

Abstract— This paper explains the design of a prototype desktop and augmented Virtual Reality (VR) framework as a medium to deliver instructional materials to the students in an introductory computer animation course. To be of use to other instructors or researchers interested in implementing a similar framework like this, the paper provides information on the hardware, software, and the concept inventory components of this framework. This framework was developed as part of a Teaching Innovation Grant at a Midwestern University to propose some cost-effective and innovative instructional frameworks to engage and stimulate students. This paper is an extended version of the paper presented at the CoED division of the ASEE conference and it presents VR modules and assessments with some modified techniques to the earlier version presented at the annual conference. This paper also shows the relevance of the methods used in the context of other STEM curriculum in addition to graphics and modeling courses. PC-based Desktop VR has been chosen as a medium for this study due to the ease-of-access and affordability; this framework can be visualized and accessed with the available computers in PC labs available on university campuses. Besides modeling and animation, various STEM concepts including manufacturing education, nanotechnology, chemistry, and biology are presented in an interactive manner on a desktop display. This framework allows the users to interact with the objects on the display not only via the standard mouse and keyboard, but also using multiple forms of Human Computer Interaction (HCI) such as Touchscreen, Touchpad, and 3D Mouse. Hence, the modules were developed from scratch for access via regular desktop PCs. As the author has been teaching this course for many years now, this framework has been designed with due consideration to structure of the course, ‘Introduction to Animation’. Finally, the framework has been tested on a range of VR media to check its accessibility. On the whole, this proposed framework can be used to not only teach basic modeling and animation concepts such as spatial coordinates, coordinate systems, transformation, and parametric curves, but also to teach basic graphics programming concepts.

Keywords—Virtual Reality, Interactive Instruction; STEM Education; Computer Graphics

I. INTRODUCTION AND BACKGROUND

In today’s higher educational institutions, classrooms are often overwhelmed by challenges including lack of enthusiasm and lessening attention span, which may be caused by classroom practices that fail to remedy the cognitive overload, specifically when teaching new information. Interactive VR-based tools can be quite effective in motivating curiosity and reducing cognitive load [1] [2]. Designing the instructional elements in a fun-based interactive VR setting can motivate instructors as well as students to explore new technologies and facilitate overcoming the inherent resistance to change. Over the years, VR tools and methods have been successfully employed in various educational disciplines [2] [3] [4].

Captivating students’ attention necessitates connecting to the participant (student) in the learning process and VR is a proven tool that can engage learners effectively. [3] [4] [5] [6]. According to Sherman and Craig [7], VR can be described as “… a medium composed of interactive computer simulations that sense the participant’s position and actions and replace or augment the feedback to one or more senses, giving the feeling of being mentally immersed or present in the simulation.” Several notable studies have successfully demonstrated the use VR in learning and their effectiveness in instruction [1] [3] [4] [5]. The features of VR such as interaction and navigation (Figure 1) facilitate actively engaging with the learning materials [4] [5]. The multiple modes in which contents can be delivered via VR include immersive VR, augmented VR, mixed VR, and desktop VR allow for the ability to select the correct mode as required by budgetary, infrastructure, and space constraints [3] [5]. These features are especially important in case of computer graphics (CG) materials because of the creative components involving storyboarding and production pipeline.

Figure 1. VR Characteristics & Advantages to STEM

Based on the observations in classrooms in multiple computer graphics technology (CGT) classes with students from a variety of disciplines, especially in the initial stages, students tend to be excited to create and see 3D digital objects. Often times, it is challenging for students to patiently wait to learn the underlying basics before seeing the end result. Especially, with the availability of software applications that can swiftly create 3D objects and cool software gizmos, it is hard for young minds to resist the temptation to explore a lot at once. However, succeeding in the CGT program as well as in their careers, requires a firm grasp of the fundamentals of the production pipeline and the processes therein. To overcome this challenge, it is important to balance courses with activities to allow latitude for their creativity whilst boosting critical-thinking and problem-solving skills. The lab assignments typically include a part where they would summarize their understanding of the theory and includes a creative part where they select a theme of their own choice to demonstrate their graphics skillsets. A strong understanding of the fundamental mathematics, geometric, trigonometric, and physics fundamentals plays a crucial role in determining the career-success of computer graphics (CG) students. Figure 2 shows virtual  reality 3D worlds developed to demonstrates a range of concepts such as digital lighting, mechanical components, interior design, and robotics. These 3D representations (Figure 2) facilitate understanding the theoretical knowledge behind the corresponding disciplines. This program puts forth a novel PBL-based approach wherein an interactive portable desktop Virtual (dVR) framework is used to methodically organize and present foundational information. The framework uses a sequential learning process, beginning with simple concepts, gradually advancing towards more complex ideas. The medium used is desktop VR and the computers in the computer labs can be used ‘as is’. An Open Source plug-in can be used on the standard browsers such as Internet Explorer and Mozilla Firefox to view and interact with the virtual worlds.

II. VR AS AN INSTRUCTIONAL TRAINING TOOL IN STEM & NON-STEM CONTEXTS

Virtual reality (VR) and computer graphics are highly interrelated. The evolution of VR has been aided by the advancements in 3D graphics, visualization, and interactive user interfaces. Evidently, CG has tools and techniques that tremendously influence and impact the capabilities of VR and define the limitations as well. In this study, multiple modes are used to overcome space and cost limitations. Laboratories in various engineering and technology disciplines are not always able to meet the demands of today’s advanced curriculum in their conventional form with the traditional experiments. Revamping physical laboratories with equipment to specifically tailor them for pedagogical training in specialized areas is not only labor-intensive, but also costly. Virtual training laboratories, can be designed and virtual modules can be implemented across multiple platforms so that students can easily access them over standard desktop PC. Interaction with these VR worlds can be performed with standard I/O (Input/Output) devices such as a mouse and a keyboard. The interactions can be programmed to performed with other affordable input devices including a stylus^©, a touchpad^©, etc. to understand processes, assemble components, and perform trial-and-error procedures without risking equipment damage.

Figure 2. (Clockwise from Top-Left) Interactive Applications:

Lighting, Trigonometry, Interior Design, Robotics The virtual laboratories and training modules, once built,  can be replicated for use on multiple machines without any additional costs. This involves copying the required software and training modules to other computers. During the training for manufacturing processes, VR tools serve as a viable alternative offering a cost and material-efficient solution by replacing the need for actual physical materials. Although virtual laboratories may not entirely replace physical laboratories, VR practices can lead to considerable cost and time savings by limiting the use of physical lab/equipment to only when actual implementation or manufacturing is required. De Jong, Linn, & Zacharia [8] state: “Virtual experiments delivered with computer technology add value to physical experiments by allowing students to explore unobservable phenomena; … to conduct multiple experiments in a short amount of time;” Virtual training has immense potential in terms of courses using alternative delivery modes (distance education) with guided voice-over and visual cues to allow students to complete tasks in a paced manner.

III. METHODOLOGY: CONCEPTS, HARDWARE & SOFTWARE

Computer animation skills are extremely important for CGT students to excel in their academic pursuit and to succeed in their career. Computer graphics are used in numerous applications including web design, print media, logo design, game design and development, 3D modeling and animation, etc. While this offers numerous opportunities for students, it inherently involves the challenge of preparing students for these opportunities. Students, especially those at the beginner level, encounter difficulties when learning to understand basic computer animation concepts and the lack of efficient tools to overcome such difficulties can affect students’ motivation. The methods described are for augmented and desktop VR. Active learning and project-based learning are proven tools to facilitate learning and retention [9] [10] as they can motivate students and facilitate applying the knowledge and skills gained in the classroom in the real-world. Instructional practices used in the course CGT 241 (Introduction to Animation) enabled the students to apply the 3D design knowledge to solve a real-world problem involving 3D printing design [11]. Pedagogical practices integrating virtual environments allow the learner to interact with the 3D representations (models). The modules are designed for access via multiple modes such as desktop VR (dVR) and augmented VR (aVR). VR training simulations have been used in a variety of fields for training outside of engineering [4] [6].

In developing VR modules, the first step involves creating a concept inventory of aspects to be covered using the 3D scenes. For example, with respect to computer graphics, a concept inventory study was carried out to determine the core concepts that would be covered using the functionalities of the VR worlds. The concepts for graphics identified included the following:

• What are the references for building 3D objects- origin, coordinates, dimensions, & coordinate systems?
• What are polar and Cartesian coordinate systems?
• How to build shapes using points, lines, and polygons?
• What are translation, rotation, and scaling?
• How is uniform scaling distinct from non-uniform scaling?
• What is the role of triangles & quadrilaterals in modeling?
• What is poly-count and what role does it play in modeling?
• How to represent in absolute and relative values?

The above represent the concept inventory questions generated for the primary stage of computer graphics learning. Similar set of questions are generated for intermediate and advanced stages of computer graphics instruction as well. The explanations to the above concepts instead of merely being provided as a book based or conventional lab-based approach are presented in the form of interactive VR worlds.

The hardware used in this research (Figure 3) consists of the following major components:

• HP Z800 machine,
• Oculus
• Graphics Drawing Tablet
• Xbox One Kinect

Figure 3. Hardware Devices Used for the Framework

The HP Z800  is a computer  workstation  well-known  for high fidelity graphics and computations performance. The Samsung display is capable of generating binocular vision using its own proprietary glasses. The Razer Hydra is built specifically for the game industry. MS Kinect is a gaming interface designed to function with Microsoft Xbox. However, owing to the possibility of Kinect being extended (without Xbox), this research uses Kinect as the tracking interface. More  specifically,     we use Kinect for head-tracking of the users. It is not always required to use immersive systems like CAVE systems as affordable augmented VR (aVR). With a well-designed human computer interaction (HCI) and UI (User Interface) design can accomplish the immersion  and navigation required for a specific application. It needs to be noted that not all applications that use CAVE necessarily require such high specifications. Frequently, the ultimate objective of an application can be attained using the system proposed in this research without resorting to CAVE. The display system can also use a HMD display instead of a HDTV. Head mounted displays have built features for audio, navigation controls, and tracking. Figure 4 demonstrates the essential elements of the framework developed for this study.

Figure 4. Three-Tiered Framework

The proposed framework is used as follows:

1. The weekly theory lessons, instead of being presented primarily in the textual format or as PDF or Power Point files, are presented via the interactive browser interface (Figure 2 – Image on top right – Interaction with trigonometry module).
2. The modules introduce the concepts in an interactive manner to the students. Students can practice using the interactive modules delivered in the form of 3D virtual. Then they will be led to another screen that presents the students with questions (in the form of visual demonstrations) to help them review the learning materials.
3. Next, the students will be led to the subsequent chapter. The modules are divided into conceptual categories such as mathematical concepts, physics concepts, trigonometric concepts, and graphics programming concepts (that help them to access as a reference if needed. The VR-based framework designed from a computer graphics perspective include the following:

• A VR laboratory capable of delivering conceptual (theoretical) and practical CG training
• Extensible VR modules designed to support immersion, navigation, and interaction
• Coursework materials and laboratory exercises delivered in a paced manner to support face-to-face and distance- learning curriculum

The VR modules developed were also tested on multiple modes (immersive, desktop, and augmented VR). The steps involved in the process of designing and implementing the framework have been summarized into three major procedures (Figure 5):

1. The 3D models are created using a software (Autodesk 3ds Max or Maya) or programming language such as Java3DL
2. Export to a compatible format such as .OBJ (Object) or. FBX (Film box) to be compatible with Unity
3. Program and customize objects in Unity

Figure 5. Major Steps in Tier II Model Creation

Figure 6 below demonstrates the interactive delivery of a module to explain programming concepts [1]. There are nine icons showing various elements of the learning module. The icon on the bottom right (x=3) is used to demonstrate the concept of data types in programming.

Figure 6. Step-by-Step Interaction Using VR

The laptop (Figure 6) shows the actual module being executed. The diagram on the right shows the step-by-step interactive game where the data items such as integer, float, char, and string are matched using drag and drop into the correct cylinders. As the student selects the correct data types and dropping them into the correct cylinders, they start moving down and finally when all the items are correctly matched the cylinders are finally all the way down showing the module has been completed.

IV. RESULTS AND DISCUSSION

This section discusses the hardware and the software issues in addition to the actual results from the 3D interactive VR displays. Selecting the correct VR system is a multifaceted problem. The system must be able to meet the instructional, graphics-based, immersive, and interactive aspects necessary for users to receive necessary instruction while being immersed in the simulation. This is to balance educational necessity with the goal of motivating learners with interaction and graphics [12]. Factors considered include:

1. Hardware (CPU) and graphics requirements: System requirements were considered to decide if available computers are compatible with the system or if they will require a better graphics cards, CPU, etc. As the visual learning style is critical, the system requires the necessary tools for learners to interact with the simulation [1].
2. Cost: Economical systems are vital to accomplish the intended goal, as staying within budget is necessary.
3. Standalone Capability: can function free of costly additional hardware/devices without server support.
4. Software compatibility/Support: Software compatibility/support refers to the support from the company/community that the system is associated with.

Unity was used as the development platform due to the support of this platform from companies and online communities, and is recognized as a common development language. Unity works very well with VR due to the Unity VR package and compatibility with HMD and desktop VR. The options that were considered for the VR system were the Samsung Odyssey, Google Card Board Headset (GCBH), and Dell Visor (Figure 7). In order to determine the best system for the study, a comparative analysis was created using the Oculus/HTC as the basis to compare the other systems.

Figure 7. VR systems used for Testing VR modules

Table 1 shows the summary of the comparative analysis of the VR systems while comparing characteristics including cost, standalone capability and software requirements.  Using a virtual workflow, basic 3D operations were explained through interacting with the objects in the 3D environment.

Table 1. Comparative Analysis of VR Devices

	COST	STANDALONE	SOFTWARE
Oculus/ HTC Vive	$500 Vive $400 Oculus Laptop Needed	Need additional laptop	High graphic capability, Interaction
Samsung Odyssey	$400 system Phone $300-500 plan	Smartphone with plan Needed	Accurate controls, Needs smart phone
Google Cardboard	$15 Cardboard, Phone $300-500, plan varies	Requires a modern smartphone with a plan	Limited interaction, most affordable
Dell Visor	$300 Visor, $15 dongle, $10 adaptor	Standalone (low- cost Adaptor and dongle)	Accurate programmable controls.

Figure 8. Virtual Workflow to facilitate Interactive Learning of 3D Transformations

To explain transformational concepts, a simple and practical example for chosen and students interact with this in the VR scene. Students in a university are typically well- acquainted with the library setup and are familiar with the library environment. Hence, a 3D VR-based common example of stacking books and arranging the layout is used to explain the fundamental concepts of transformation (Figure 8). A complete library of 3D objects has been built to help students help students understand the basic concepts demonstrated earlier has been developed so that students can use such real-world examples to understand and practice the above theoretical aspects (Figure 9).

Figure 9. Virtual Library –Showing a part/section

Tables 2 and 3 demonstrate the beginner and intermediate/advanced VR scenes generated to explain the computer graphics concepts to the students. These virtual worlds are capable of interaction using the standard devices and those mentioned above Wacom © and Dell © Touchpad. These are relatively much cheaper and can be procured and installed without much difficulty. Other additional features of interaction were using affordable devices including the following:

• Standard Gaming Mouse/Keyboard
• Wacom © Tablet/Stylus
• Dell © Touchpad
• Razer Hydra ©

Table 2 provides an overview some of the CG concepts. Table 3 provides an overview of some of the intermediate to advanced CG concepts that have been covered within the VR framework that can be accessed via multiple modes. This system can be used to facilitate instruction of STEM concepts through interactive learning exercises        and active learning in Engineering and Technology curriculum. In addition, students in various ET disciplines can use this framework to apply CG concepts in their discipline-specific applications. Various examples were discussed earlier demonstrating components that were assembled together to form a scene assembly thereby enabling students to understand the concept of transformation. The salient features of the framework and how these are aligned with the course learning outcomes is depicted in Table 1.

In summary, the salient features and the advantages of the framework are as follows:

• Simple approach aiding coherent flow
• VR examples to explain CG concepts
• Active learning via interaction to facilitate underlying mathematical concepts
• An easy to understand, VR- based approach for explaining advanced concepts

V. CONCLUSION

This framework allows the users to interact with the objects on the VR display not only via the standard mouse and keyboard, but also using multiple forms of HCI such as Touchscreen, Touchpad, and 3D Mouse. Hence, the modules were developed from scratch for access via regular desktop PCs. As the author has been teaching this course for many years, this framework has been designed with consideration to structure of the course, ‘Introduction to Animation’. This project employs a pedagogical approach using multi-modal VR for CG instruction specifically targeted at 4-year degree programs. In addition to CG (computer graphics), 3D modeling and animation courses are also taught in various other engineering and technology disciplines. In this paper, the framework discussed is primarily from the point of view of a technology curriculum with more emphasis on the application. However, some basic theoretical aspects of modeling and animation are inevitable in order to create 3D models and animate them. This work represents an extended version of the paper presented at the CoED division in the ASEE conference. This framework has improved in the following ways:

• More streamlined approach for the design and implementation of the VR modules
• Wide range of examples in different STEM disciplines
• Enhancing the framework with Unity platform interaction and programming Customization for specific Head Mounted Displays

ACKNOWLEDGEMENTS

The author expresses gratitude to the PNW Teaching Innovation Grant and the PNW Internal Grants that facilitated this research. This effort was also partly funded by the NSF Grant # 1700674.

VI. LIMITATIONS

There were some important limitations and/or constraints that the authors experienced with this prototype study. The vast majority of the efforts in the study were dedicated towards designing the framework, development, and implementation with consideration of specific CG curriculum.

Studies were conducted to develop and test applications in diverse disciplines such as other branches of ET, Interior Design, Manufacturing, Civil Engineering etc. However, as part of future studies, the authors endeavor to conduct tests and include data related to student performance. These studies would specifically target measuring the accomplishment of course learning outcomes within the ET disciplines.

As this is a work in progress, the framework needs to be further developed and then these modules will be tested in CGT courses with Institutional Review Board (IRB) approval. Also, the basic concepts in computer graphics span over a wide range of topics that vary according to the software languages and modeling platforms. Based on the literature review this study has narrowed down the very basic concepts that are common to different CG courses across the nation. Detailed feedback will be obtained from the participants and comprehensive quantitative/qualitative evaluations will be carried out. Furthermore, focus groups of participants will be included during the workshops to actively engage other instructors as well and solicit their critiques on methods implemented thus far and for future enhancement.

REFERENCES

[1] Chandramouli, M., & Heffron, J. (2015, March). A desktop vr-based HCI framework for programming instruction. In 2015 IEEE Integrated STEM Education Conference (pp. 129-134). IEEE.

[2] Dias, P., Sousa, T., Parracho, J., Cardoso, I., Monteiro, A., & Santos, B. S. (2014). Student projects involving novel interaction with large displays. IEEE computer graphics and applications, 34(2), 80-86.

[3] Toth, E. E., Ludvico, L. R., & Morrow, B. L. (2014). Blended inquiry with hands-on and virtual laboratories: the role of perceptual features during knowledge construction. Interactive Learning Environments, 22(5), 614-630.

[4] Chandramouli, M., Zahraee, M., & Winer, C. (2014, June). A fun-learning approach to programming: An adaptive Virtual Reality (VR) platform to teach programming to engineering students. In IEEE International Conference on Electro/Information Technology. IEEE

[5] Jen, Y. H., Taha, Z., & Vui, L. J. (2008). VR-Based robot programming simulation system for an industrial robot. International Jl. of Industrial Engineering, 15(3), 314-322.

[6] Pan, Z., Cheok, A. D., Yang, H., Zhu, J., & Shi, J. (2006). Virtual reality and mixed reality for virtual learning environments. Computers & graphics, 30(1), 20-28.

[7] Sherman, W. R., & Craig, A. B. (2003). Understanding virtual reality. San Francisco, CA: Morgan Kauffman.

[8] De Jong, T., Linn, M. C., & Zacharia, Z. C. (2013). Physical and virtual laboratories in science and engineering education. Science, 340(6130), 305-308.

[9] Jou, M., & Wang, J. (2013). Investigation of effects of virtual reality environments on learning performance of technical skills. Computers in Human Behavior, 29(2), 433- 438.

[10] Monahan, T., McArdle, G., & Bertolotto, M. (2008). Virtual reality for collaborative e-learning. Computers & Education, 50(4), 1339-1353.

[11] Houstex International 3D Design Competition houstexonline.com/3d-printing-student-competition-winners/

[12] Chandramouli, M., Takahashi, G., & Bertoline, G. R. (2014). Desktop VR centered project-based learning in ET courses using a low-cost portable VR system. In Proceedings of the American Society of Engineering Education.