1. Introduction

Since the spring of 2020, when institutions around the globe shut down classrooms and design studios in response to COVID-19 safety concerns, remote learning has been an expanding area of research. During the early stages of the pandemic, remote work was required for students and many professionals due to public safety concerns, and this accelerated the development and adoption of numerous tools for remote collaboration. While most public health concerns have now subsided, and students and many professionals are back in-person, teams in industry and academia have continued to research ways to leverage remote collaboration tools to improve the performance of collaborative teams. The proliferation of new tools and increased comfort with remote work creates an opportunity to assemble teams that are more diverse because they can include members that are geographically separated. To examine this opportunity, we attempted to execute senior capstone design projects in mechanical engineering with students and faculty advisors from geographically separated universities, with sponsors also scattered throughout the US. This arrangement allowed us to increase the pool of candidate students for these projects and include a wider variety of expertise and personal backgrounds. Though we focused on teams built from students at two universities who were physically located at their respective primary campus, this effort is highly applicable to situations imposed by semesters abroad, distance learners, or satellite campuses. It is also useful to consider for students who are not able to attend class in person due to injury, illness, travel visa restrictions, or other reasons. More broadly, this effort can provide guiding principles for remote collaboration within industry where participants on a team are not co-located. As many businesses and government offices contemplate policies surrounding remote work, this effort will help inform their decision making process by providing context-specific experiential data linked to specific phases and activities of an engineering design project.

Most engineering education programs include capstone design courses for many reasons. Capstone experiences unite a student’s engineering coursework into a holistic, project-based experience exposing them to real-world challenges they will face after graduation. The capstone design experience fulfills program accreditation needs and approaches learning through a different pedagogical model by making the learning more hands-on, interdisciplinary, and purpose-driven. Indeed, engineering capstones stem from the need to provide a culminating experience to engineering students. This experience directly contributes to the ABET “General Criterion 3: Student Outcomes 1-7”¹ in an integrated approach. The following objectives are used in the capstone engineering course that is the subject of this research:

1. Given a statement of customer need, students design a system to satisfy that need based on commercial product development best practices.

2. Students will demonstrate the ability to effectively communicate their design.

3. Students will demonstrate the ability to fabricate a functioning prototype of their design.

4. Students will demonstrate the ability to be effective interdisciplinary team members and leaders.

5. Student designs will comply with a realistic level of engineering codes and standards and shall include considerations such as environment, economics, manufacturability, sustainability, health and safety.

These outcomes illustrate the importance of executing a complete design process in capstone from initial need definitions all the way through to a functioning prototype. Thus, for the multi-university capstone teams that are the subject of this study, it is important to evaluate the teams’ effectiveness during all phases of the design process. Additionally, one must consider how all five of the outcomes are achieved by multi-university collaborative teams.

2. Related Research

During the past 20 years as technology’s march has brought individuals closer together virtually, the idea of geographically disbursed design capstone teams working together has taken off. Indeed, the internet was invented primarily to facilitate collaboration between geographically separated researchers.² Initial studies highlighted the technological challenges of the early 2000s including limited abilities to video conference or share files conveniently.^3,4 These works highlighted challenges of coordinating schedules across schools, between daytime and nighttime, courses as well as aligning semesters and quarters. They also emphasized the high financial cost of bringing students together for in-person meetings as well as the opportunity cost of lengthy travel. Despite these difficulties, the authors highlighted the benefits of multi-university collaboration including shared facilities, student learning focused on concurrent engineering and system integration, product realization of something complex given larger, more capable teams, and students learning how to work on diverse teams. Despite these pathfinder activities, remote collaboration for physical design teams has remained the exception, not the norm, in the past two decades. Efforts have focused instead on virtual collaboration through teams working on shared CAD models⁵ or have used online tools to develop a marketplace where students from multiple universities could find a collaborative capstone experience advertised by a shared sponsor.⁶ This study also confirmed the additional faculty burden to bring these teams together and to advise them, but highlighted the benefits of obtaining students with a more diverse set of skills.

As online design efforts have matured, studies have started to investigate features that make remote collaboration effective. In their research, Brisco et al.⁷ identified key characteristics for effective remote design processes to function, such as ensuring software is in place for collaborative document editing, electronic whiteboards, and task lists with each team member’s responsibilities clearly identified. These findings echo those determined earlier by Carter⁸ and Motoya et al.⁹ including high-bandwidth video conferencing, shared whiteboard, application sharing, and data management tools.

Recent research on remote pedagogy has tended to focus on a university’s ability to continue classes remotely, spurred by the COVID-19 pandemic, including how to remotely teach or assess learning.^10–17 Contemporary efforts have stopped short of investigating how faculty can conduct design capstones with students not collocated. Early research did, however, start to emphasize the importance of teaching remote collaboration in an engineering design curriculum.¹⁸ A few pre-pandemic efforts did focus on examining how capstone design teams can work together from geographically separated locations, usually motivated by the prospect of students finding themselves on such teams once moving to industry.^19–21 These efforts tended to focus specifically on what enabled the teams to succeed. For instance, Stone et al. divided the design effort into three stages – early, middle, and late – focusing on customer needs and ideation, then detailed design, then prototyping and testing, respectively. For instance, for each stage they recommended specific communication tools appropriate to that stage such as face-to-face, web conferencing, video conferencing, shared databases, and text messages. Since that effort, many of the online collaboration tools have evolved to include multiple of these features operating simultaneously, blurring the lines of when to turn to a specific, isolated tool. In general, these efforts have concluded that students benefit from rich communication as defined by Daft and Lengel²² early in the process to establish team camaraderie and a shared sense of purpose. To accomplish this end and encourage rich communication, the teams in this study met each other and their sponsors in-person within the first month.

Within an academic setting, geographic diversity can emerge due to several reasons. For instance, health concerns may lead students to miss classroom attendance or faculty may seek out diverse skill sets on teams by recruiting students from academic programs uniquely offered at other institutions. For this perspective, industry has a lot to share based on recent lessons learned after the massive shift to remote collaboration necessitated by COVID-19. Despite the pandemic, design firms were still required to meet their customers’ needs and create working software and hardware. Ferguson et al. discuss this transition in depth.²³ They identify nine challenges (such as “shallow and inefficient communication” or “building rapport between teammates virtually is difficult”) that design teams face across the three phases of design – intangible design, tangible design, and communication and management. Each of these phases align well with a traditional capstone effort starting with intangible design efforts in the fall (e.g. ideation, customer needs analysis), tangible design efforts in the spring (e.g. prototyping, testing) and management and communication throughout the entire year. Additionally, each of the nine challenges they discuss should be addressed, if not at least considered, in a remote collaboration environment. In a separate effort,²⁴ researchers further expanded upon the difficulty of prototyping in the design phase. The existing research to date highlights a need to revisit remote collaborative engineering design teams in an academic setting to elucidate how teams can be successful through the application of current tools.

3. Research Approach

To conduct this research, the investigators formed three senior capstone design teams composed of students from the United States Air Force Academy (USAFA) and Brigham Young University (BYU). The following descriptions provide context about each school relevant to the composition of each team.

1. The United States Air Force Academy (USAFA) is an undergraduate only institution with a strong focus on teaching. The university is an accredited institution of higher learning that has approximately 4,000 students, all of whom graduate with a Bachelor of Science degree. USAFA is located in Colorado Springs, Colorado and has the mission “to educate, train, and inspire men and women to become leaders of character, motivated to lead the U.S. Air Force and U.S. Space Force in service to our nation.”

2. Brigham Young University (BYU) has both undergraduate and graduate students accomplishing both first class teaching and research. The university is an accredited institution of higher learning that has approximately 35,000 students, 90% of whom are at the undergraduate level. BYU is located in Provo, Utah, and has the mission “to assist individuals in their quest for perfection and eternal life.”

This effort highlights unique opportunities and challenges of a military academy (USAFA) working with a civilian institution (BYU); this collaboration is rare within academic engineering programs, but prolific within the larger DoD industrial complex. Typically, there are many companies that work closely with DoD acquisition professionals to field systems such as F-35 aircraft or communications satellites; in these projects, hundreds of companies collaborate to build products using inputs from around the country (or even around the world). This unique collaboration opportunity provided a glimpse into the motivations of students who have different outcomes after their college experience. For example, upon graduation and successfully earning a commission as an officer, USAFA cadets are guaranteed a job in the military for at least 5 years, whereas BYU students need to find employment after graduation and often use their capstone experiences as stepping stones and resume builders toward that end.

Since previous efforts have shown that an iterative approach towards a capstone design experience may improve learning for the students,²⁵ each team started with a mini design project related to their overall project addressing many of the major design activities, then repeated the entire process for the rest of the course focused on their main project. The capstone engineering course in the subject research is taught over two semesters following the design and creation of a project, as desired by a sponsor, with design theory woven into both semesters of the course to fit the appropriate design stage (e.g., ideation techniques, analysis methods, failure modes). Throughout the academic year, teams worked remotely on a day-to-day basis through online videoconferencing, then came together in person for key milestones and test events.

Each team was given a unique design challenge based on the needs of various Department of Defense (DoD) sponsors (see Table 1 for a complete list of projects studied in this effort). In research such as this, there are numerous variables, many of which cannot be controlled. It would be preferable to assign the same design task to all of the teams in the investigation and to include “control” teams, with all members collocated at a single university. This would help discern whether observed differences are due to individual team members and teams, or if they were consistent across multiple teams. However, practical concerns, such as the limited number of students, faculty advisors, and financial resources, prevented this. Each team had a sponsor from the DoD that had worked with homogeneous USAFA capstone teams previously; this allowed them to provide feedback on how the unique collaborative, multi-university teams compared to previous single-university teams.

Table 1.This table describes the projects the teams worked on as well as the student, advisor, and sponsor details.

While this construct is specific to two geographically separated universities, it is much more broadly applicable to any collaborative design engineering effort where participants are separated due to other reasons such as: students working remotely because they are abroad or on exchange for a semester, distance learning students who attend remotely full time, students who are unable to attend in person due to injury, or illness (for example, as was experienced significantly during the COVID-19 lockdowns of 2020). This work also serves as an example of how to conduct remote collaboration in more broad remote collaborative engineering efforts within industry, for instance when two companies work together or even two segments of a single company work together from different locations.

Once the multi-university teams were established, they set to work going through the engineering design process adopted from Mattson and Sorensen²⁶ as shown in Figure 1. As the students worked on these year-long projects, faculty provided them various collaboration tools; some intended to be used throughout the effort, as well as a few that were intended for only specific phases or activities. For the duration of the year, students primarily used Microsoft Teams for group chat, file exchange and archive, and video meetings each class. However, other collaborative tools were also promoted. As an example of a tool used extensively by the teams is the collaborative white board app, ConceptBoard, which was found in previous work to be the most-effective online collaborative ideation tool.²⁷ It was introduced for use in ideation during the “Explore” phase. During the “Define the design” and “Test the design” phases, which include modeling and prototyping, faculty introduced and encouraged students to use Fusion 360 (Autodesk), a computer aided design (CAD) and analysis software package which has online collaborative features.

Figure 1.Mechanical Engineering Basic Design Process adapted from Mattson and Sorensen[@400019] and updated to match our specific course and show major milestone reviews.

This investigation seeks to answer two primary research questions:

1. How does an undergraduate engineering capstone design team perform when composed of students from two geographically separated universities, as compared to teams from a single university?

2. How beneficial or detrimental is remote collaboration to each phase of the design process? In other words, can all design phases/activities be completed effectively with remote collaboration, or are some phases more or less amenable to remote collaboration?

To answer these questions, surveys were administered to students and project sponsors after three key milestones: (1) Preliminary Design Review (PDR), (2) Critical Design Review (CDR), and (3) Final Out-brief. The students answered the surveys to provide insight into how the teams worked together during each phase of the project as well as how effectively their tools enabled collaboration. The sponsors responded to the surveys at these milestones to help answer how the collaborative teams compared to previous single-university teams, as well as to track how well they performed in the time period leading up to each of the three milestones. The surveys consisted of both quantitative and qualitative questions. Since the surveys were anonymous, voluntary, and not tied to a grade, student participation rates varied during each phase from 88% during PDR, to 47% during CDR and final presentation. Thus, the effort presented here is squarely in the “discovery and description” phase of theory building about what works well or not in collaborative design efforts as detailed by other researchers.²⁸ Furthermore, while some quantitative data exists, this work relies heavily upon qualitative data, since this is a relatively new approach.²⁹ During the course, assessment of student learning is completed formally via faculty comments at PDR, CDR, and final out brief from sponsors, and assessment via grades is used to measure attainment of course objectives³⁰ indicates grades are a valid approach to assess student learning). Data to suggest one is better vs the other (i.e. co-located vs geographically separate teams) unfortunately cannot be uncoupled from individual student and team performance, but for the geographically separated teams we see added learning and development of skills for dealing with various approaches to problem solving, in terms of diversity and inclusion for various behavioral and technical approaches to problem solving. An additional quantitative measure that researchers considered was the productivity of geographically separated teams compared with collocated teams of previous years; this was assessed using the metrics of patent applications and papers produced. The use of these metrics is not meant to reduce the emphasis on the growth and learning of students, which is paramount in our program. Patents and papers are quantifiable and external validation of the novelty and usefulness of the teams’ work, thus they are evidence of how the performance of these teams compares to those of prior years.

4. Discussion and Results

The results from the surveys are separated by student feedback and then sponsor feedback. The students surveys help determine how well remote collaboration worked at each design phase, how the tools enabled design activities and overall. The sponsor surveys provided key insights into how the remote collaboration teams performed compared to previous years’ teams consisting of students who were all at the same location. Additional metrics are also evaluated to determine the productive output of the remote collaboration teams compared to previous collocated teams.

4.1. Student Feedback

Student feedback about the collaborative experience was obtained through a series of surveys administered at three separate times during the design project - after the PDR, the CDR, and the final out-briefing. This feedback was constrained to only consider the current academic year as the students only take a single year of capstone in the program at each participating school. Student data provided detailed information about the progress of the projects during the design phases detailed in Figure 1 experienced throughout the entire year. The information obtained was quantitative in Likert-scale questions as well as qualitative in open-ended questions.

Quantitative results for student feedback are shown in Figure 2, as well as Tables 2 - 5. Figure 2 shows a paraphrased version of each survey question along with the mean user rating for that question. The shading corresponds to the elements of collaboration and serve to group similar questions. Summative questions asked students to assess how effective remote collaboration was for successfully completing the project as a whole. Design activity effectiveness questions focus on how well students were able to achieve each of the design phases through remote collaboration. Tool-specific questions focus on the specific tools the faculty provided for them to accomplish the tasks at hand. Finally, the dislocation element describes how effective it was to physically bring team members together at a single location (thereby dislocating part of, or the entire, team). Note, mean response values are plotted for each of the three milestone events (PDR, CDR, and Final) to facilitate analysis of how success measures changed throughout the design experience.

Figure 2.Student survey results from all three phases of the design. Responses are color coded by phase with a unique symbol used for each. Additionally, the four elements of collaboration are alternately shaded in the table to help differentiate how remote collaboration went when viewed through different lenses. Questions had a range of 1-5 with higher scores corresponding to positive responses as indicated at the arrow at the bottom of the chart.

Table 2.This table of student feedback describes the first element of collaboration: Summative feedback. Each question was asked after the three key milestones in the course, namely PDR, CDR, and the final presentation. Questions had a range of 1-5 with higher scores corresponding to positive responses.

Table 3.This table of student feedback describes the second element of collaboration: Design Activity Effectiveness. These questions elicit student responses concerning how well the various phases of design went in a remote collaborative environment. Each question was asked after the three key milestones in the course, namely PDR, CDR, and the final presentation (except as indicated by the grey boxes; these questions were not asked at a respective milestone as they were not applicable to one or multiple phases). Questions had a range of 1-5 with higher scores corresponding to positive responses.

Table 4.This table of student feedback describes the third element of collaboration: Tools. Each question was asked after the three key milestones in the course, namely PDR, CDR, and the final presentation except as indicated by a grey box with no score; these questions were not asked at a particular milestone as they were not applicable. Questions had a range of 1-5 with higher scores corresponding to positive responses.

Table 5.This table of student feedback describes the fourth element of collaboration: Dislocation. Each question was asked after the three key milestones in the course, namely PDR, CDR, and the final presentation except as indicated by a grey box with no score; these questions were not asked at a particular milestone as they were not applicable. Questions had a range of 1-5 with higher scores corresponding to positive responses.

From Figure 2, it is apparent that the student experience was generally positive, with the only question rating below a score of 3 being about the assessment of prototyping and testing during the final phase of the project. This is not surprising, as it is expected that physical prototyping would be difficult to perform collaboratively from distant locations, especially as compared to other, less-tangible, design tasks. This echoes previous research findings^23,24 that synthesizing a physical design remotely is difficult and not as easy to accomplish remotely as other tasks. Overall, there is more scatter for CDR vs. the PDR and final phases, indicating a convergence of results during PDR, then divergence during CDR, and concluding with more converged responses for the final period of time. Specific question-by-question comments are found in each respective table (i.e. Tables 2 - 5), one table for each element of collaboration.

4.1.1. Results Based on Student Feedback after the Preliminary Design Review (PDR)

Regarding the activities and time leading up to the PDR, which includes understanding the need and exploring solutions and ideas, as shown in Figure 1, the students generally had a positive experience with the remote collaboration. Though there was some learning required to familiarize themselves with the tools (e.g., MS Teams and ConceptBoard for these design phases), once the students had working accounts and knew how to use the features of the tools, they developed an effective rhythm to understand the need and explore solutions and ideas. To help accelerate their progress and motivate their purpose, the students visited their sponsors early in the semester. This trip was very well received by the students (as seen in Figure 2, question 22). It allowed the students to meet each other for the first time, accelerating their team formation. The visit also allowed the team to interact with customers first-hand, explore existing testing facilities and capabilities, and gain a much stronger appreciation for the context in which their systems would be used. These meetings, while relatively expensive compared to the overall team budgets, greatly increased student understanding of customer needs, accelerated team formation, and motivated students by exposing them to the customers’ missions.

The time period leading up to the PDR was not without issues related to remote collaboration, however, despite having positive feedback overall. Though the majority of students were up and running quickly in the semester, some had persistent technical difficulties connecting to MS Teams and were thus left out of conversations or unable to contribute to team products. It is important to note that the network constraints of a military academy are very stringent with substantial cybersecurity controls in place, thus other universities may not experience similar connectivity difficulties. Through serendipity, the two schools’ capstones occur at the same time allowing synchronous team meetings 2-3 days per week to work together for two hours; outside of this window, however, it was often difficult to align schedules between the two schools, as has been found in previous multi-university studies.^3,4 We should note that this is not a unique problem, however, as even when students are collocated their schedules outside of class are misaligned due to other classes, work, or other extracurricular engagements. Students commented that during this period, it was difficult to track each team member’s contributions since there was not clear accountability for each document produced. This would become more pronounced later in the project as team members worked more independently on subsystems.

Students generally requested more hands-on mentoring early in the design process to help them understand the project and develop a plan to meet the customer needs. For most of these students, this capstone course is their first experience with a formal engineering design process, whereas later in the project, when students are analyzing concepts and building prototypes, they are applying skills learned previously in other courses. Therefore, it should be expected that the students would seek more guidance for these early design activities. The students who were not collocated with the faculty advisors commented that not having the advisors on the video calls left them feeling neglected and without sufficient guidance. Related, students expressed dissatisfaction in their inability to participate in sidebar conversations during video calls. This led to students feeling excluded and that their “opinions and contributions are inferior” and that they “don’t know as much about the project.” One solution that has strong advocates in commercial practice,³¹ is that meetings should either be all remote or all in person, so each member has a common experience. When part of the team is remote and part is in person, it may result in excluding the remote participants. An alternative, which blends both approaches, is for participants to be grouped together as much as possible, then each group connected remotely.

4.1.2. Results Based on Student Feedback after the Critical Design Review (CDR)

During the period between PDR and CDR, design teams work to Define the Design by adding detail to components and interfaces while performing analysis, modeling and experimentation to demonstrate that the design will meet the design requirements while avoiding failure. Remote collaboration during this phase of the design process was generally rated by the students as being less successful, however students still had a positive experience. Though technical issues persisted, by this time most students had worked through those wrinkles and settled into an effective meeting cadence and work rhythm. With overarching designs defined, teams were able to decompose their proposed solutions into subsystems and effectively divide design and analysis tasks among the students. Thus, the highly synchronous work that occurred pre-PDR moved into asynchronous tasks that the teams could spread among themselves and work remotely with greater ease.

Again, despite having positive feedback overall, the students encountered challenges with remote collaboration during the phase preceding CDR. As team members started to work individually on delegated tasks, communication became less frequent, and, according to one student; “making it difficult to gauge the progress of certain subsystems.” This phase increases the demands on students (compared to the conceptual design phase prior to PDR) as it requires them to deliver a detailed design that is more tangible than the various PDR deliverables. Related, team members often found it difficult to track the assignment, progress, and completion of tasks. All of the teams utilized spreadsheets to track tasks, but these proved ineffective. Task management tools are needed that can provide daily progress checks, reminders, and an easy way to assess the progress of the whole project at a glance. Thus, it may be beneficial for project leaders (faculty in this case) to investigate and incorporate dedicated management tools such as JIRA or Asana to handle these functions. Ideally, the chosen tools would have low start-up costs and low overhead so that they can be used by student teams. Additionally, the issue of some remote students feeling left out of conversation persisted from the pre-PDR phase into the pre-CDR phase. Students also commented on the frequency of team meetings, with some stating that 2-3 times per week was too infrequent, but others stating that every day was excessive. Thus, a baseline recommendation is to meet every class period (every other day, 2-3 times per week) to ensure all students are on the same page in terms of team expectations and progress, but also meet outside of those times if needed to make progress.

4.1.3. Results Based on Student Feedback after the Final Presentation

As the teams worked toward their final presentations, remote collaboration continued to be effective overall, however evidence emerged that the experiences varied by team. Some team members expressed great satisfaction, while others had large struggles during the last phase of the project, as evidenced by the spread in results for question 3 in Figure 2. Some teams reported having substantial success integrating code and hardware developed separately; one stated, “I was surprised at how well the collaboration and software development process went despite the physical separation and differences in hardware between us and our BYU counterparts.” Other students describe satisfaction in integrating hardware built at separate locations for different subsystems into one effective system, reporting “everything came together nicely.” Other students, however, had much less positive experiences stating, “motivation of the team overall decreased rapidly after CDR” and “[it is] hard to build a single physical model when teams are in other states” echoing the quantitative results seen in question 12. Thus, it is possible that students with highly modular systems, i.e. those that could be broken down into independent subsystems, and then brought together using careful interface controls, had more success in the final phase of the project which is very software and hardware dependent. Another challenge cited is that the team meetings, that are supposed to keep everyone synchronized and on track, occurred when students needed to be working in the lab, hindering individual progress. This is a constant balancing effort for all teams, however, whether they are remote or not – how much to communicate versus how much to be working independently to accomplish each task. Thankfully all members reported much success working together to produce final briefings and reports (as seen in question 13 which was most highly rated for final phase), another key deliverable at the end of the project.

4.1.4. Results Based on Student Feedback Throughout the Entire Project

Throughout the entire year, several themes emerged based on student feedback. The experience as a whole was viewed positively and students were thankful to be part of the collaboration. Helpful technological solutions were found during each stage of the project to aid in the collaboration including voice, video, and text communication tools, online-whiteboards, and computer aided design. Students also highly valued in-person meetings when available. Initial meetings accelerated team formation, while subsequent meetings allowed intensely creative work sessions and physical hardware integration. Less favorable findings persisted as well. Students found it difficult to make sure all voices were heard. Some conversations occurred on the sidelines excluding remote participants. System integration and physical prototyping were difficult to accomplish efficiently. Accountability was difficult to achieve since at times it was unclear what certain members of the team were accomplishing and contributing to the team.

4.2. Question-Based Results

The previous sections focused on the temporal dimension, i.e. how the project progressed throughout the three phases. Another approach to evaluate the effectiveness of the collaboration is to consider the experience in terms of the elements of collaboration; these are the different ways to quantify the activity and determine how effective it was. Here, we will consider the elements of collaboration in four categories: summative, design activity effectiveness, tools, and dislocation. Summative questions were related to how well remote collaboration worked for the overall project; these questions and results are found in Table 2. Design activity effectiveness questions focused on how well students were able to achieve each of the design activities in a remote collaboration environment; these questions and results are found in Table 3. The tools element focuses on the specific tools the faculty provided for them to accomplish various tasks; these questions and results are found in Table 4. Finally, the dislocation element describes how effective it was to physically bring team members together at a single location (thereby dislocating some or all of the team); these questions and results are found in Table 5.

4.3. Sponsor Feedback

Alongside the student feedback, evaluations were solicited from representatives of the various capstone projects’ sponsors. These are generally engineers or program managers in the government who are subject matter experts in the particular field of the capstone project, and they have several years of experience working with military academy capstone design teams. Like the student feedback, sponsor feedback was obtained through three surveys throughout the project - after the PDR, the CDR, and the final presentation. Because it was impractical to implement anything resembling a proper scientific control in this work, the sponsor feedback was collected to provide an un-biased assessment of how the teams performed relative to prior instances of capstone projects, when team members were all collocated. Thus, the sponsor’s role is to evaluate how the teams’ products compared to those of previous, collocated, teams. They also provided feedback relevant to how the teams performed relative to the objectives for the capstone enumerated in Section 1. Results are shown in Figure 3 and Table 6.

Figure 3.Sponsor survey results from all three phases of the design. Responses are color coded by phase with a unique symbol used for each. Questions had a range of 1-5 with higher scores corresponding to positive responses as indicated by the arrows at the bottom of the chart.

Table 6.This table of sponsor feedback provides their feedback as well as conclusions derived from their ratings and comments. Each question was asked after the three key milestones in the course, namely PDR, CDR, and the final presentation except as indicated by a grey box with no score; this question was not asked at a particular milestone as it were not applicable. Questions had a range of 1-5 with higher scores corresponding to positive responses.

Before starting the projects, the sponsors had widely varying views on the ability of remotely collaborating student design teams to succeed. As seen in individual survey responses to questions 10 and 11, two of the sponsor representatives were not very concerned and quite excited, while the third was quite concerned and not very excited. Thus, even among seasoned government technologists, there is a lack of consensus on the effectiveness of remote collaboration, especially when physical products are being produced. Otherwise, the results were nearly constant between the sponsors across all three of the different time periods, indicating that, from their perspectives, the teams performed similarly regardless of the design phase they were completing. A key result from the sponsors’ feedback is that the remote teams performed consistently with previous collocated teams – neither better nor worse. This is promising, since, as one sponsor pointed out, “because this team is a hybrid team with members from two physically separated universities, they have had a number of different additional challenges to address that previous teams did not.” Thus, despite the additional logistical work required, the students still performed at a consistently high level with previous years, and throughout all phases of the design process. Considering specific performance areas, the results were unanimous that the teams performed on par or better than collocated teams, with slightly higher quality and creativity (this result should not be unexpected because it is well-known that increasing team diversity leads to an increase in innovation.^32–34 Sponsors also indicated that communication between the sponsor and the team was slightly better than previous years. Perhaps the necessity for the students to work online every day amongst themselves lowered the barrier to communicating remotely with team sponsors. By the projects’ ends, a sponsor wrote, “They learned how to work effectively and efficiently under those circumstances [i.e. being physically separated].” This indicates the methods the teams used to communicate and accomplish their design process was equivalent to collocated teams.

4.4. Team Productivity Metrics

The outcomes of the senior design capstone course are focused on student development and learning. However, patents are convenient metrics that are externally recognized and validated as products of an innovative design process; the design process is an integral focus of the course. Published papers are also externally validated sources of a team’s collective effort. Thus, each team’s performance can be measured based on its productivity in terms of published work enabling a historical analysis of how the teams compare to previous years. Considering all three of the remote-collaboration teams of this research, each team’s work resulted in an academic conference paper and a submitted invention disclosure for a patent application. This is consistent with previous collocated teams and hence their outputs are comparable based on these independent metrics.

5. Limitations

The results of this effort must be considered relative to a few limitations. First, the project occurred over a single academic year. Next, only three teams were included with a maximum participation rate of 87% of 17 students, so, while trends can be considered and evaluated, the results do not have strong statistical significance. On the sponsor side, there were three sponsors total (one for each team) and not all three responded to each of the surveys (participation rates ranged from 33% to 100%). Future work should look at innovative methods to increase survey participation rates such as incentives.³⁵ This collaboration occurred between two schools only, so it does not add in the complexity of a third (or more) location, and additional locations could bring additional complications. Thus, the overall amount of data considered is limited. This collaborative approach is ongoing, so additional data can be collected in subsequent years to strengthen the quality of the data set. Survey data was anonymous so it is not possible to infer factors that could have led some students to rate the experience well or poorly such as grade point average, location at the host school vs the remote school, etc. Another consideration is that each team had an independent advisor; this could lead to differences in how the remote collaboration was executed on a daily basis as each advisor brought in unique expertise for how to run capstone teams and how to work effectively in a remote setting. Finally, the sponsors’ ratings of this year’s teams were relative to their past experiences with fully collocated teams; thus, if a sponsor has only worked with a few teams then their comparison was limited.

6. Conclusions and Considerations for Implementation

This work indicates that a collaborative capstone engineering class can be successfully executed with students who are geographically separated. Such separation may result from separate schools working together, students who are not able to join due to sickness, or from semester exchange programs at other universities or in different countries. As evidenced by the sponsor feedback falling largely close to a rating of 3, the results of remotely collaborating teams were on par with the collocated teams of previous years. Furthermore, students generally enjoyed and learned from the experience stating, “overall, this was a unique opportunity and I am glad that I was part of this collaborative experience” and “the collaboration has gone really well.” If students are not able to be continually collocated, leadership needs to recognize the importance of occasionally bringing the team members together at key events in the process, such as initial team formation, major presentations, and during hardware integration and testing. Leadership in charge of remote collaboration design teams must also recognize, as did the students in the present study, that there is additional logistical overhead in managing these projects: “I have really appreciated all the efforts of leaders both at BYU and USAFA to make this collaboration happen.” Another enduring benefit of these efforts is that it exposes students at one location to other students with diverse backgrounds, experiences, and cultures, which supports key diversity and inclusion initiatives across the nation. Indeed, “Increased exposure to people who differ on various attributes can cause individuals to question their beliefs and assumptions about the world and correct any negative biases they may possess about unfamiliar others.”.³⁶ Beyond this cultural imperative, “mere exposure to foreign cultures in and of itself increases creativity… similar benefits may be found when people interact extensively with others who come from different cultures, backgrounds, and philosophies”³⁶ – this is very desirable for a design team.

Below is a list of final considerations faculty or design team leads would need to examine if implementing remote collaboration for design efforts.

1. Funding requirements may be higher with two geographically separated teams. This can result from building prototypes at separate locations, shipping of materials between locations and additional travel costs to bring team together for team building and specific design activities.

2. Certain projects may benefit from remote collaboration more than others such as those with less hardware focus, as hardware manufacturing and integration is one of the more difficult parts of remote collaboration.

3. Remote collaboration can add strengths to a project by bringing in students with additional classes and backgrounds from schools that may have additional capabilities, such as software availability or manufacturing facilities and tools.

4. Each location may have different purchasing systems, as is the case between separate universities. This can slow down progress and increase accounting burden.

5. Universities must decide how to allocate and share costs between schools.

6. Project leads must figure out a plan to effectively share supplies and equipment (e.g., test configurations, sponsor-provided hardware) between universities.

7. Team advisors and leads must think carefully about how to ensure equitable participation and buy-in between universities.

8. Faculty must carefully consider academic calendar differences between schools such as semester start and end dates as well as grade reporting timelines. Related, faculty must also consider overlaps or conflicts in student schedules to ensure they have an opportunity to meet synchronously.

9. Faculty must consider expectations for how much students from one school travel vs. students from the other school.

10. When working on projects with classified information or security considerations (e.g. U.S. citizens only), faculty must exercise caution and ensure any collaborators have appropriate security protocols in effect.

11. Faculty should consider and be aware of team and cultural dynamics between different universities to help students work together effectively.

12. Faculty should consider student constraints that can make working together outside of the assigned class times (to include traveling from one location to another to be collocated for select design activities) challenging such as athletic requirements, family obligations, work, cultural differences, etc. Ensure students are willing and available to travel as needed.

13. When members of a team are traveling for a collocated event, it may be beneficial to try to extend the collocation duration beyond the minimums of the event requirements to increase focused work time and face to face interactions.

14. Planning should be in place to determine the intellectual property ownership and assignment process should papers, patents, or similar products manifest.

Though online collaboration was effective, in-person still seems to be preferred for this kind of design work; according to one student, “The online collaboration has not been bad… but the team does so much better when we are physically in the same place in my opinion.”

Conflict of Interest

The authors declare no conflict of interest.