Learning Analytics of Outcomes-Based Engineering Programs' Data

Anwar Ali Yahya

Paper Views 5,909

DOWNLOAD

DOI COMING SOON

Anwar Ali Yahya ✉ ¹

1 College of Computer Science and Information Systems, Najran University, Saudi Arabia

2 Faculty of Computer Science and Information Systems, Thamar University, Thamar, Yemen

Abstract

In the recent years, learning analytics is attracting attention in tertiary education sector. This paper presents a case study of applying learning analytics approaches to discover knowledge from Outcome Based (OB) engineering programs’ data. More specifically, Association Rule Mining approach is applied to a dataset extracted from the Self-Study Reports of 152 engineering programs accredited by American Board of Engineering and Technology (ABET). In doing so, the dataset has been processed and transformed into a suitable representation. Apriori algorithm is then applied to generate rules involving PEOs and ABET SOs. The generated rules are filtered, and the filtered rules are used to draw a set of generic rules for mapping each PEO to ABET SOs and to discover the correlations among ABET SOs. Finally, the practical benefits of the discovered insights to the engineering programs’ academicians, decision makers, and ABET are discussed.

Keywords

Association rule mining, learning analytics, program educational objectives, student outcomes, ABET accreditation

INTRODUCTION

In response to the pressing needs of reforming traditional educational systems, many attempts have emerged 1. Outcome Based Education (OBE) 2 is the latest paradigm shift sweeping in the tertiary education (education at university or college level) sector. It emerged in the 1950’s to equip graduates with ability to accept the challenges, adopt to technological changes, and translate their knowledge to new contexts for the benefit of the society. Generally, OBE is based on developing a set of outcomes around which educational activities are focused and establishing the conditions and opportunities that enable achieving these outcomes. In tertiary education, OB academic programs develop two main types of outcomes: Program Objectives (POs) and Program Learning Outcomes (PLOs) 3. Whilst PLOs define what the students would be able to do after the completion of the program, POs identify the reasons or purpose of the programs.

In practice, the POs of an OB academic program must meet the requirements of employers and other stakeholders and correspond with the institution mission 4. The PLOs specify the competencies the graduates must demonstrate, based on POs. In this sense, a mapping relationship should be established between program mission, POs, and PLOs. Moreover, other program components such as curriculum, teaching and learning strategies, and assessment strategies must be designed in such a manner that students ultimately gain knowledge and develop skills stated in the PLOs. In this hierarchical structure of OB program, shown in Figure 1 , the curriculum is viewed as a set of courses aim to attain certain Course Learning Outcomes (CLOs) that map to PLOs which themselves map to POs which in turn map to the mission of the institution.

https://s3-us-west-2.amazonaws.com/typeset-prod-media-server/fb183f33-4f7f-432d-9fab-94eca1522575image1.png — **Figure 1: Structure of OB Academic Program**

In engineering education, the professional bodies responsible for accrediting professional engineering such as the Accreditation Board of Engineering and Technology 5, plays a key role to hasten the shift towards OBE paradigm. In fact, since the Washington accord 6, an agreement of mutual recognition of programs between accreditation bodies of professional engineering programs, adopted OBE as compulsory for accreditation, the signatories accreditation bodies have started developing their approaches of OB engineering programs and implemented them for program accreditation. ABET, a signatory of Washington accord moved in the direction of OB since it introduced “Engineering Criteria 2000” (EC2000). In ABET approach of OBE, the terms Program Educational Objectives (PEOs) and Student Outcomes (SO) are adopted to refer to the POs and PLOs respectively 7. While SOs represent the knowledge, skills, and capabilities that students should possess by the time of graduation, PEOs represent the achievements graduates should attain few years (3 to 5 years and more) after graduation 8. In addition, ABET developed the following set of SOs for accrediting Engineering programs.

(a) An ability to apply knowledge of mathematics, science, and engineering.
(b) An ability to design and conduct experiments, as well as to analyze and interpret data.
(c) An ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability.
(d) An ability to function on multidisciplinary teams.
(e) An ability to identify, formulate, and solve engineering problems.
(f) An understanding of professional and ethical responsibility
(g) An ability to communicate effectively.
(h) The broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context.
(i) A recognition of the need for, and an ability to engage in life-long learning.
(j) A knowledge of contemporary issues
(k) An ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.

Despite the crucial role of POs and PLOs and the importance of the consistency between them for the design and accreditation of OB engineering programs, there is a consensus among academicians on the lingering confusion on these terms and their relationship to each other9 10 8. In case of ABET OB approach of engineering programs, this concern has been raised earlier in several events such as the fall 2005 ABET Summit 11. A possible consequence of this concern is a poor design of curriculum and teaching strategies, a misleading assessment of PEOs, and ultimately inaccurate corrective plan 9 10 8. As a result, a need for progressive clarification changes in ABET definitions of PEOs and SOs, the accreditation policy, and procedure manual has been always insistent 12. The most recent wave of these changes was introduced in ABET engineering accreditation criteria 13. It was motivated, among other factors, by the feedback from educators and programs evaluators on the difficulties related to the understanding the ABET SOs and their mapping to PEOs.

In the recent years, Leaning Analytics (LA) is being used actively for wide range of purposes in tertiary education, to enhance the learning process, to evaluate efficiency, to improve feedback, to enrich the learning experience, to support decision-making 14 15. One of the effective approaches of LA is Association Rule Mining (ARM) 16 17. It has been used to discover the relationships in educational content in various contexts18 19. In this paper ARM is proposed to resolve the above-introduced confusion surrounding PEOs and SOs in engineering context. More concretely, this paper proposes using ARM approach to discover a set of rules that govern the mapping between PEOs and SOs and a set of correlations among SOs by inductively interrogating PEOs-SOs mapping data of number of ABET accredited engineering programs. Although, the current work addresses the problem in ABET-specific context, the proposed approach and the obtained results are extendable to other OB approaches such as Engineers Australia 20 21.

The remainder of this paper describes the association rule mining techniques, positions the current work within the relevant literature, describes how the ARM technique is applied to this dataset, presents the obtained results, discusses the results, and finally concludes this work.

Association Rules Mining

Association rules mining (ARM) is one of the most important and well researched techniques of data mining 22. It was first proposed to identify significant purchasing pattern from a large database of consumer transactions 23. Since then, it has been widely used in various areas such as telecommunication networks, market and risk management, inventory control, etc. 16, and more recently in education. To understand the ARM approach, the following mathematical preliminaries are necessary.

Let $I = \{I_{1}, I_{2}, \dots, I_{3}\}$ be a set of items. Let X be a set of transactions in a task–relevant data, where each transaction T is a set of items such that $T \subseteq I$ . An association rule is an implication of the form $A \Rightarrow B$ where $A \subset I, B \subset I$ , and $A \cap B = \emptyset$ ,where both A and B are a set of items, which is referred to as an itemset. An itemset that contains k items is a k-itemset. For example, the set $\{I_{2}\}$ is a 1-itemset, and the set { $\{I_{2}, I_{5}\}$ is a 2-itemset. The occurrence frequency of an itemset is the number of transactions that contain the itemset 16. Typically, there are two common measures used in association rule mining: support and confidence. Support is defined as the percentage of transactions in X that contain $A \cup B$ :

S u p p o r t (A \Rightarrow B) = \frac{C o u n t (A \cup B)}{N}

where count ( $A \cup B$ ) is the number of transactions that contain both A and B, and N is the total number of transactions in the whole dataset. A low value of Support ( $A \Rightarrow B$ ) suggests that association rule $A \Rightarrow B$ may occur simply by chance and is not interesting. Confidence is defined as the percentage of transactions in X containing A that also contain B:

C o n f i d e n c e (A \Rightarrow B) = \frac{S u p p o r t (A \cup B)}{S u p p o r t (A)}

Confidence determines the extent to which the appearance of A implies the appearance of B. Based on these two measures, an association rule could be identified if both of its support and confidence values exceed a predetermined threshold. Given the total number of transactions in the whole dataset, we can obtain Support ( $A \Rightarrow B$ ) and Confidence( $A \Rightarrow B$ ) by calculating the number of transactions containing both A and B or count ( $A \cup B$ ), and the number of transactions containing A or count(A).

The algorithm of extracting association rules from a given dataset work by dividing the problem into two parts: mining frequent itemsets and rules discovery from the frequent itemsets. A frequent itemset is a set of items with frequency more than a threshold. The procedure of finding frequent itemsets is simple but very time consuming, because of the large number of the possible combinations. Once they have been discovered, the rules production is a simple process. A widely used algorithm for the association rules mining is the Apriopri algorithm 22. It is based on the following intuition: All sub-itemsets of a frequent itemset must also be frequent. Using this rule, Apriori algorithm prunes a huge amount of itemsets examinations since it is certain that they are not frequent. Frequent sub-itemsets are extended one item at a time (candidate generation), and groups of candidates are examined. It terminates when no further extensions are found. In other words, Apriori algorithm generates candidate itemsets of length l from itemsets of length l−1 and then it prunes the candidates, which have a non- frequent sub-itemset. Thus, it keeps only the frequent item sets among the candidates.

Related Work

The huge amount of data available in a digital form has motivated the emergence of data analytics to analyze these data in an automated manner 16. Data analytics has already achieved significant success in many areas including medicine, business, robotics, and computer vision, to name just a few. By the same token, the explosive growth of data in educational institutions has led to the emergence of three research fields, namely Educational Data Mining (EDM), Learning Analytics (LA), and Academic Analytics, that are concerned with applying computerized methods to analyze large collections of educational data that would otherwise be hard or impossible to analyze 24 25 14. Whilst the three fields share the common goal of improving educational practice using data-driven approaches, several differences between them in their focus and the scale of analysis have been emphasized 26 27. While LA focuses on improving educational outcomes and applied to the data at course, subject, program, and department levels, AA focuses on improving educational results and applied to the data at institution, region, national, and international levels. EDM techniques, on the other hands, are applied to the data at any level because it focuses on extraction of useful insights from of learning related data.

To get an overview of the relevant literature, regardless of the LA/EDM distinction, the following are worthy examples. In each work, program-relevant data of different type are analyzed using different data analytics techniques to extract knowledge on different aspects of the program. For instance, a dataset of student learning outcomes is collected through survey and analyzed using a combination of neural network and experts’ prior knowledge to predict and evaluate student-learning outcomes of an academic program and ultimately enhance teaching quality 28. K−means clustering algorithm is applied to investigate the relationship between skills taught in business programs and the title of the program using a dataset extracted from program catalogue 29. The analysis shows that, with very limited exceptions, the labels of programs match the skills one would expect to learn. In 30, data analytics methods are used to identify the similarities between course content at a learning object, module and program level.

A dataset consisting of student – job interview pairs is used to build a weighted directed graph (vertices are programs and edge denote the % of jobs that interviewed at least one student from both programs), to which a graph mining method is applied 31 to carry out different analysis: finding communities, finding vertices connected to many communities, and finding vertices strongly connected to their neighbors.

A dataset collected from undergraduate students of engineering programs at a large Canadian university in a form of survey responses is analyzed using regression and classification techniques to investigate the effect of academic program type on the mental health of students 32. Interestingly, the results show that the more competitive a program is the lower mental health becomes. Moreover, the results show that the stronger classmate relationships and flexible curriculum are the reasons of higher mental health score.

Furthermore, the academic programs’ assessment data have been analyzed in several works. For instance, a data-driven approach is applied to course assessment and program assessment data to quantify the level at which program’s curriculum meet the program outcomes. This involves the use of clustering and prediction techniques for question evaluation, topic clustering, predicting missing scores, and clustering with partial topic information to construct score matrix and relevancy matrix. Using these matrices with course matrix the numeric value for each program outcome can be determined by simply find the average score for each topic and multiply that vector by the course matrix.

On the other hands, the analysis of program’s curriculum data has been receiving an increasing attention 33 34 as well. For instance, Synthetic Control Method is applied to analyze based on student performance data and build a linear model describing the relation between courses 35. The model can be also used to predict students’ grades in a specific course based on their previous grades. In another work 36, the students’ behavior data is analyzed to assess the curriculum. First, the students are categorized according to their learning paths using aggregate profile clustering, and then sequence mining approach is applied to assess the sequence of learning path in conformity with the prior curriculum guideline.

Concerning ARM technique, which is of focus in this work, its estimated that its use to analyze educational data represent 14% of the previous works 37. It has been used for a variety of purpose 25 38 39 such as discovery of learning rules based on students’ characteristics and competencies, promoting collaborative learning, predicting student’s performance (final grades) based on features extracted from logged data in an e-learning environment, monitoring and evaluation of academic performance, to mention but a few. Nonetheless, the use of ARM to analyze data at program level has not been reported yet, even in most recent relevant survey 37.

Based on the above concise review of the field, the current work can be positioned at the intersection of the Learning analytics and Educational data mining fields, because it applies a well-known data mining technique, ARM, to analyze data at program level to improve the educational outcomes of academic programs. More concretely, from LA perspective, it has been reported that most of the previous works deal only with data at course level, whereas very little works have been done to analyze educational data at program level 24. Meanwhile, the current work is an EDM application and to the best of our knowledge it is the first application of ARM to analyze the correlation between PEOs and SOs and the correlations among SOs.

Methodology: Mining Association Rules From ABET Accredited Engineering Programs’ Data

The methodology of applying ARM to extract useful insights from PEOs-SOs mapping data is a customized variant of the general methodology of knowledge discovery process 18. It involves raw data collection, data selection, data pre-processing, data transformation, data mining, and evaluation. Figure 2 depicts schematically the steps of the methodology of applying ARM to ABET accredited engineering programs’ data. Although the proposed approach is applied to engineering programs, it can be applied to any OBE academic programs regardless of its discipline.

In the data collection step, raw data are collected and used to create a target dataset on which the discovery will be performed. The target data is cleaned and pre-processed to obtain consistent data in the pre-processing step. The transformation step transforms the data using dimensionality reduction or other transformation methods. The step of ARM application applies procedures to search for patterns of interest in a particular representational form. Finally, in the interpretation/evaluation step, the mined patterns are interpreted and evaluated. In the following subsections, the details of ARM application to ABET accredited programs PEOs-SOs data is presented.

https://s3-us-west-2.amazonaws.com/typeset-prod-media-server/fb183f33-4f7f-432d-9fab-94eca1522575image2.png — **Figure 2: Methodology of applying ARM to ABET accredited engineering programs’ data**

Raw Data Collection

The raw data is a collection of PEOs-SOs mapping data of engineering programs accredited by ABET. The PEOs-SOs mapping data have been extracted from programs’ Self-Study Reports (SSRs). A program’s SSR is the primary document a program seeking for accreditation must prepare to demonstrate its compliance with all applicable criteria and policies developed by accreditation agency. As per ABET accreditation requirements, a program seeking for accreditation must develop its PEOs and map them to a specific set of SOs developed by one of four commissions responsible for accrediting programs in four disciplines in engineering and technology. In case of engineering discipline, ABET developed the following eleven (a to k) SOs. In this research, the SSRs of a 152 engineering academic programs previously accredited by ABET between 2000 and 201740 are the source of PEOs-SOs mapping data. Figure 3 shows the distribution of the SSRs over the years.

https://s3-us-west-2.amazonaws.com/typeset-prod-media-server/fb183f33-4f7f-432d-9fab-94eca1522575image3.png — **Figure 3: SSRs distribution over years**

Data Selection

The program’s SSR contains detailed information about the program with respect to the following: program integrity and capacity, including program mission, objectives, outcomes, faculty and resources, and curriculum plan, evaluation, content, and outcomes. However, since this research is concerned with analyzing PEOs and SOs data, only data of PEOs-SOs mapping have been drawn from the collected SSRs. These data have been extracted from sub-section B of the third criteria (Student Outcomes) of each SSR and consolidated in a table form as shown in Figure 4 , where symbols ✓ and × indicate, respectively, the presence or absence of the SO in the mapping with the PEO.

https://s3-us-west-2.amazonaws.com/typeset-prod-media-server/fb183f33-4f7f-432d-9fab-94eca1522575image4.png — **Figure 4: Snippet of the raw PEOs-SOs dataset**

Dataset Pre-processing

In the pre-processing step, the selected data is cleaned and pre-processed in order to obtain consistent data. Moreover, the pre-processing of the selected data involves substantial verification and validation of the content, attempts to remove spurious or duplicated objectives, fulfilling the objectives and outcomes format, etc. Moreover, since PEOs in the SSR are represented in textual form, they are unsuitable for data ARM algorithms, therefore, a transformation procedure to transform each PEO into labels representing graduate accomplishments expressed in its text is proposed. To this end, a set of PEOs labels is developed and used to annotate each data instance with a single or multiple PEOs labels. Based on the desired analysis, target datasets are extracted from the annotated dataset.

Due to the manual extraction of the raw PEOs-SOs mapping data from the SSRs, mistakes might occur such as duplicate extraction of data instances, extraction of unnecessary content, missing part of the content etc. Therefore, the pre-processing of the raw dataset involves substantial verification and validation of the content to remove any spurious or duplicated PEOs-SOs data and to ensure each data instance fulfill the objectives and outcomes format. Table 1 shows some statistical aspects of the pre-processed dataset.

Table 1: Statistical aspects of PEOs-SOs dataset.

Statistical Aspect	Value
No. of instances	667
No. of programs	152
Max. No. of textual PEOs in a program	17
Min. No. of textual PEOs in a program	2
Avg. No. of textual PEOs in a program	4.39 (667)
Most frequent SO	(e) = 305
Least frequent SOs	(b, d) = 266
Avg. frequency of SO	287.7

A transformation procedure is proposed to transform every PEOs text into one or more PEOs codes that represent the graduate attributes expressed in its text. The details of the transformation procedure are given in the following subsections.

PEOs Code Set Identification

According to ABET’s definitions, PEOs describe, in a broad sense, the expected career and professional achievements of a graduate after few years of graduation. The expected achievements fall in the following dimensions: Technical, professional, ethical, and communicative, management and leadership, life-long learning and continuous education, advanced and graduate studies pursuing, and other 20. Academic programs should develop a certain set of graduate attributes, and each PEO can correspond to one or more of these attributes. Based on PEOs wordings of a number of engineering programs, a set of common PEOs attributes has been identified and coded as shows in Table 2 .

Table 2: PEOs Code Set

No	PEO	PEO Code	Frequency
1	Life-long Learning	LL	130
2	Communication	C	82
3	Leadership	L	62
4	Teaming	T	98
5	Ethical Conduct	EC	81
6	Professionalism	P	137
7	Social and Community	SC	78
8	Career Success	CS	127
9	Technical Competency	TC	212
10	Knowledge Competency	KC	96
11	Graduate Studies	GS	64
12	Others	O	29

An interesting observation from Table 2 is the variation in the PEOs frequencies. Obviously, some PEOs (Life-long Learning, Professionalism, Career Success, and Technical Competency) have high frequencies, whereas others (Leadership, Graduate Studies, Social and Community) have low frequencies. This reflects a bias in how academic programs craft their PEOs, where it seems that certain PEOs are preferred instead of others. An interesting reason behind the bias of PEOs crafting is the obligatory requirement of ABET from academic program to assess their PEOs. Due to this, academic programs tend to craft their PEOs solely based on the easiness of their assessment. Since Life-long Learning, Professionalism, Career Success, and Technical Competency PEOs can all be assessed by number of graduates that are proceeding to licensure or have been promoted (both are relatively easy to assess), their frequencies in the dataset are high. Similarly, due to the difficulty of assessing Leadership, Social and Community, Ethical Conduct, and Communication PEOs, their frequencies are low. It should be mentioned that due to the bias in PEOs crafting, starting from 2016, ABET has removed the obligatory requirement of assessing PEOs, giving programs the freedom to craft PEOs that truly represented their missions.

PEOs Coding

The developed PEOs code set is used to code the data instances in the pre-processed PEOs-SOs mapping dataset by replacing the PEO text in each data instance with the most appropriate PEOs codes from the PEOs code set. This task has been accomplished by three coders who initially coded the data set individually and then at a later time, they met to resolve the conflicting cases of coding. It should be mentioned that in this process some PEOs text might be coded with more than PEOs code, because it happens that the PEOs text describes multiple attributes. Figure 5 shows a snippet of the data instances after PEOs coding.

https://s3-us-west-2.amazonaws.com/typeset-prod-media-server/fb183f33-4f7f-432d-9fab-94eca1522575image5.png — **Figure 5: Snippet of the coded PEOs-SOs dataset**

Multi-codes to Single-code Data Projection

In this step, each multi-code data instance, in the coded PEOs-SOs mapping dataset, is projected into a set of single-code data instances as shown in Figure 6. This results in an enlarged dataset represented as a 1196×13 matrix.

https://s3-us-west-2.amazonaws.com/typeset-prod-media-server/fb183f33-4f7f-432d-9fab-94eca1522575image6.png — **Figure 6: Snippet of the projected PEOs-SOs dataset**

Association Rules Mining Application

In this step, Apriori algorithm is applied under WEKA framework 41 as described in Section 2. To implement the Apriori algorithm in WEKA, the minimum support and minimum confidence parameters must be specified. In this research due to the imbalanced nature of the PEOs-SOs dataset, and the need to generate rules for each PEO, the minimum support and minimum confidence are set to low values, 0.1. In a subsequent step the generated rules are filtered to obtain the desired rules. Figure 6 shows snippet of the generated rules. As it can be observed, each rule involves rule antecedent and rule consequent (a combination of PEOs and SOs), the count value of the rule antecedent, the count value of the combined set of antecedent or consequent, and the confidence value, which is the ratio between the two count values.

Rules Filtering and Evaluation

The application of Apriori algorithm results in a very large number of rules, which must be filtered to focus on two sets of rules: PEOs-SOs mapping rules, and SOs correlations rules. The filtered rules are then used to calculate the correlation among PEOs and among SOs according to the following formula. Assuming x and y are two PEOs or SOs, the correlation between them is defined as the confidence in their equivalency as follows.

C o r r e l a t i o n (x, y) = C o n f i d e n c e (x ⟺ y)

= C o n f i d e n c e ((x y) (\bar{x} \bar{y}))

Since the confidence is defined in terms of the conditional probability, it can be expressed as follows.

= P ((x y) (\bar{x} \bar{y}))

As they are mutually exclusive sets, it can be written as

= P (x y) + P (\bar{x} \bar{y})

= P (x | y) \times P (y) + P (\bar{x} (\bar{y}) \times P (\bar{y})

= C o n f i d e n c e (x ⟹ y) \times P (y) +

C o n f i d e n c e (\bar{x} ⟹ \bar{y}) \times P (\bar{y})

The following section describes how these rules are filtered and interpreted to extract relevant insights.

Results

This section presents the results of applying ARM to PEOs-SOs dataset. The results are two sets of rules on the PEOs-SOs mapping and correlations among SOs.

PEOs-SOs Mapping Rules

The generated rules are filtered to extract PEOs-SOs mapping rules. These rules are characterized by having a particular PEO in it antecedent and a combination of SOs in its consequent. For example, in Figure 7 rule No. 4105 and 4106 are among the filtered rules for the Knowledge Competency (KC) PEO. The filtered rules for each PEO are then sorted based on their confidence and the top-10 rules are presented in Table 3, where each column contains the best-10 rules for each PEO. The number above the arrow of each rule represents the confidence value of the rule as calculated by Eq. 2. A first look at these rules show that in most of them a single SO appears as an indicator to a particular PEO and for most PEOs the absence of the SO is the key indicator to the PEO. It is also obvious that as the confidence of the rule declines more SOs appears in the consequent part of the rule.

https://s3-us-west-2.amazonaws.com/typeset-prod-media-server/fb183f33-4f7f-432d-9fab-94eca1522575image7.png — **Figure 7: Snippet of projected PEOs-SOs dataset**

Table 3: Top-10 PEOs-SOs mapping rules

Life-Long Learning	Communication	Leadership	Teaming	Ethical Conduct
$L L \overset{0.85}{\Rightarrow} i$ $L L \overset{0.73}{\Rightarrow} \bar{d}$ $L L \overset{0.72}{\Rightarrow} \bar{b}$ $L L \overset{0.70}{\Rightarrow} \bar{e}$ $L L \overset{0.68}{\Rightarrow} \bar{c}$ $L L \overset{0.68}{\Rightarrow} \bar{g}$ $L L \overset{0.65}{\Rightarrow} \bar{a}$ $L L \overset{0.64}{\Rightarrow} \bar{b} ˄ \bar{c}$ $L L \overset{0.64}{\Rightarrow} \bar{b} ˄ \bar{e}$ $L L \overset{0.64}{\Rightarrow} \bar{d} ˄ i$	$C \overset{0.90}{\Rightarrow} g$ $C \overset{0.80}{\Rightarrow} \bar{i}$ $C \overset{0.82}{\Rightarrow} \bar{b}$ $C \overset{0.79}{\Rightarrow} \bar{a}$ $C \overset{0.79}{\Rightarrow} g ˄ \bar{i}$ $C \overset{0.77}{\Rightarrow} \bar{f}$ $C \overset{0.76}{\Rightarrow} \bar{a} ˄ \bar{b}$ $C \overset{0.74}{\Rightarrow} \bar{b} ˄ g$ $C \overset{0.74}{\Rightarrow} \bar{b} ˄ \bar{i}$ $C \overset{0.73}{\Rightarrow} \bar{a} ˄ g$	$L \overset{0.76}{\Rightarrow} d$ $L \overset{0.71}{\Rightarrow} \bar{b}$ $L \overset{0.69}{\Rightarrow} \bar{a}$ $L \overset{0.69}{\Rightarrow} \bar{k}$ $L \overset{0.66}{\Rightarrow} g$ $L \overset{0.66}{\Rightarrow} \bar{i}$ $L \overset{0.66}{\Rightarrow} \bar{a} ˄ \bar{k}$ $L \overset{0.66}{\Rightarrow} \bar{b} ˄ \bar{k}$ $L \overset{0.65}{\Rightarrow} \bar{c}$ $L \overset{0.65}{\Rightarrow} \bar{a} ˄ \bar{b}$	$T \overset{0.85}{\Rightarrow} d$ $T \overset{0.73}{\Rightarrow} \bar{a}$ $T \overset{0.72}{\Rightarrow} \bar{b}$ $T \overset{0.7}{\Rightarrow} \bar{a} ˄ \bar{b}$ $T \overset{0.68}{\Rightarrow} \bar{i}$ $T \overset{0.68}{\Rightarrow} \bar{c}$ $T \overset{0.65}{\Rightarrow} \bar{e}$ $T \overset{0.64}{\Rightarrow} g$ $T \overset{0.64}{\Rightarrow} \bar{b} ˄ \bar{c}$ $T \overset{0.64}{\Rightarrow} \bar{k}$	$E C \overset{0.89}{\Rightarrow} \bar{b}$ $E C \overset{0.86}{\Rightarrow} \bar{k}$ $E C \overset{0.85}{\Rightarrow} \bar{a}$ $E C \overset{0.83}{\Rightarrow} \bar{e}$ $E C \overset{0.83}{\Rightarrow} \bar{a} ˄ \bar{b}$ $E C \overset{0.81}{\Rightarrow} f$ $E C \overset{0.81}{\Rightarrow} \bar{b} ˄ \bar{e}$ $E C \overset{0.81}{\Rightarrow} \bar{b} ˄ \bar{k}$ $E C \overset{0.80}{\Rightarrow} \bar{a} ˄ \bar{k}$ $E C \overset{0.78}{\Rightarrow} \bar{a} ˄ \bar{e}$	$P \overset{0.80}{\Rightarrow} \bar{b}$ $P \overset{0.73}{\Rightarrow} \bar{a}$ $P \overset{0.72}{\Rightarrow} \bar{c}$ $P \overset{0.72}{\Rightarrow} \bar{e}$ $P \overset{0.70}{\Rightarrow} \bar{b} ˄ \bar{c}$ $P \overset{0.64}{\Rightarrow} \bar{d}$ $P \overset{0.64}{\Rightarrow} f$ $P \overset{0.64}{\Rightarrow} \bar{b} ˄ \bar{b}$ $P \overset{0.64}{\Rightarrow} \bar{a} ˄ \bar{b}$ $P \overset{0.68}{\Rightarrow} \bar{g}$	$S C \overset{0.90}{\Rightarrow} \bar{b}$ $S C \overset{0.87}{\Rightarrow} \bar{a}$ $S C \overset{0.87}{\Rightarrow} \bar{a} ˄ \bar{b}$ $S C \overset{0.85}{\Rightarrow} \bar{k}$ $S C \overset{0.81}{\Rightarrow} \bar{e}$ $S C \overset{0.81}{\Rightarrow} \bar{b} ˄ \bar{k}$ $S C \overset{0.79}{\Rightarrow} \bar{a} ˄ \bar{k}$ $S C \overset{0.90}{\Rightarrow} \bar{b} ˄ \bar{e}$ $S C \overset{0.90}{\Rightarrow} \bar{a} ˄ \bar{b} ˄ \bar{k}$ $S C \overset{0.78}{\Rightarrow} \bar{a} ˄ \bar{e}$	$C S \overset{0.74}{\Rightarrow} e$ $C S \overset{0.71}{\Rightarrow} a$ $C S \overset{0.70}{\Rightarrow} k$ $C S \overset{0.69}{\Rightarrow} c$ $C S \overset{0.69}{\Rightarrow} b$ $C S \overset{0.66}{\Rightarrow} a ˄ e$ $C S \overset{0.65}{\Rightarrow} b ˄ e$ $C S \overset{0.65}{\Rightarrow} a ˄ b$ $C S \overset{0.65}{\Rightarrow} a ˄ c$ $C S \overset{0.64}{\Rightarrow} b ˄ c$	$T C \overset{0.74}{\Rightarrow}$ a $T C \overset{0.71}{\Rightarrow} c$ $T C \overset{0.71}{\Rightarrow} e$ $T C \overset{0.71}{\Rightarrow} \bar{f}$ $T C \overset{0.71}{\Rightarrow} \bar{i}$ $T C \overset{0.68}{\Rightarrow} b$ $T C \overset{0.68}{\Rightarrow} k$ $T C \overset{0.67}{\Rightarrow} \bar{j}$ $T C \overset{0.65}{\Rightarrow} \bar{g}$ $T C \overset{0.65}{\Rightarrow} \bar{h}$	$K C \overset{0.75}{\Rightarrow} \bar{d}$ $K C \overset{0.71}{\Rightarrow} \bar{g}$ $K C \overset{0.68}{\Rightarrow} \bar{i}$ $K C \overset{0.65}{\Rightarrow} \bar{f}$ $K C \overset{0.64}{\Rightarrow} \bar{d} ˄ \bar{g}$ $K C \overset{0.58}{\Rightarrow} \bar{k}$ $K C \overset{0.57}{\Rightarrow} \bar{g} ˄ \bar{i}$ $K C \overset{0.55}{\Rightarrow} \bar{h}$ $K C \overset{0.54}{\Rightarrow} \bar{d} ˄ \bar{i}$ $K C \overset{0.53}{\Rightarrow} \bar{b}$	$G S \overset{0.77}{\Rightarrow} i$ $G S \overset{0.72}{\Rightarrow} a$ $G S \overset{0.67}{\Rightarrow} e$ $G S \overset{0.66}{\Rightarrow} k$ $G S \overset{0.59}{\to} a ˄ e$ $-$ $G S \overset{0.58}{\Rightarrow} \bar{d}$ $G S \overset{0.63}{\Rightarrow} b$ $G S \overset{0.56}{\Rightarrow} \bar{f}$ $G S \overset{0.56}{\Rightarrow} j$ $G S \overset{0.56}{\Rightarrow} a ˄ b$

Table 4: Recommended PEOs-SOs mapping rules

PEO

ABET SOs

✓

From the top-10 rules of PEOs-SOs mapping, a generic representation can be obtained as shown in Table 4. More specifically, the rules in each column of Table 3 have been merged to compose a single rule of PEO in each row of Table 4 . For example, the recommended rule for Life-long Learning (LL) PEO in Table 4 is the result of merging the top-10 rules of that PEO in Table 3 . In the recommended rules, the symbols ‘✓’, ‘×’, and ‘?’ are used to indicate the presence, absence, and undetermined state of a given SO in the consequent parts of the PEO top-10 rules given in Table 3.

The recommended PEOs-SOs mapping rules in Table 4 reveals some interesting observations on the correlation between PEOs and SOs. For example, it can be observed that the Social and Community (SC) and Knowledge Competency (KC) PEOs are not dependent on the presence of any SOs. They mainly depend on the absence of different combinations of SOs. The Lifelong Learning (LL), Communication (C), Ethical Conduct (EC), and Professionalism (P) PEOs depend on the presence of a single SO, which indicates a one-to-one mapping between the graduate attribute of the PEO and the skills of the SO. More interestingly, a closer look at Table 4 show that the two PEOs Leadership(L) and Teaming(T) depend on the presence of the same SOs (d and g), and on the absence of almost the same combination of SOs. This suggests a correlation between the two graduate attributes. Similarly, the PEOs Career Success (CS) and Technical Competency (TC) depend on the presence of the same combination of technical skills SOs (a, b, c, e, and k), yet differ in their dependence on the soft skills SOs (d, f, g, h, i, and j) and this suggests that they are somewhat correlated. As for PEO Knowledge Competency (KC), it does not show dependency on the presence of any SOs; however, it depends on absence of seven SOs. It should be mentioned that obtained PEOs-SOs mapping rules, shown in Table 4 , are self-explanatory in the sense that the PEO of each rule matches conceptually and linguistically the SOs skills. For example, in the first rule, Life-Long learning (LL) PEO, the PEO attribute of life-long learning has a conceptual and linguistic match with the skill of life-long learning stated in SO (i). Similarly, the communication(C) PEO, matches the communication skill stated in SO (g). The same interpretation goes for the rules of PEOs Ethical Conduct (EC), and Professionalism (P). Even in those PEOs that are dependent on multiple skills such as Career Success (CS), Technical Competence (TC), and Graduate Studies (GS), the mapping between these PEOs and the corresponding SOs is linguistically and conceptually intuitive. For example, it intuitively makes sense to map the success in career (CS) PEO with the skills stated in SOs (a, b, c, e, and k), the technical skills of SOs, because the technical success is essential to achieve success in career.

On the other hand, the recommended PEOs-SOs mapping rules in Table 4 reveal some interesting observations regarding the use of SOs in the PEOs-SOs mapping. More precisely, some SOs are used more frequently than others in the PEOs-SOs mapping. This is observable from the number of ‘✓’ symbols in the column of each SO. For example, it is obvious that SOs (a, b, e, g, and k) are mapped more frequently to PEOs, while SOs (h and j) are least frequently mapped. Again, the way in which academic programs craft their PEOs, particularly prior 2016, plays a role in the variation of using SOs in PEOs-SOs mapping. As pointed out in the earlier ‘PEOs Code Set Identification’ subsection (4.3.1), due to ABET obligatory requirement, prior 2016, from academic program to assess their PEOs, they tend to map PEOs to the SOs based on their easiness of assessment. Considering this, SO (h), understanding global, societal, environmental, and economic impacts, has least frequency in mapping to any PEOs, because its assessment in graduates three to five years is extremely difficult and thus academic programs tend to exclude it from PEOs-SOs mapping to avoid this difficulty.

Finally, the recommended PEOs-SOs mapping rules, shown in Table 4, distinguish between PEOs based on the clarity of their mapping to SOs, which can be measured by counting the number of ‘?’ symbols in their rows. In this manner, since the Technical Competency (TC) has only one ‘?’ symbol, it has the clearest mapping to SOs. However, the most ambiguous PEOs is Social and Community (SC) with seven ‘?’ symbols. By the same token, the SOs also can be distinguished based on their clarity in the PEOs-SOs mapping by counting the number of ‘?’ symbols in their columns. Following this manner, the SOs b is the clearest one, whereas h and j are the most ambiguous. Moreover, a distinction between SOs can be observed based on their contribution to PEOs-SOs mapping by counting the number of ‘✓’ symbols in their columns. In this manner, the SOs (a), (b), (e), and (g) have the highest contribution, while (h) has the lowest contribution.

SOs Correlation Rules

In order to discover correlations among ABET SOs, the generated rules are filtered to extract rules of the form $S O_{x} \Rightarrow S O_{y} a n d \bar{S O_{x}} \Rightarrow \bar{S O_{y}}$ .Table 5 shows extracted rules along with their confidence values.

Table 5: Confidence ( $S O_{x} \Rightarrow S O_{y}$ ) and Confidence ( $S O_{x} \Rightarrow S O_{y}$ )

$a \overset{c o n f}{\Rightarrow} S O y$ $\bar{a} \overset{c o n f}{\Rightarrow} \bar{S O y}$	$b \overset{c o n f}{\Rightarrow} S O y$ $\bar{b} \overset{c o n f}{\Rightarrow} \bar{S O y}$	$c \overset{c o n f}{\Rightarrow} S O y$ $\bar{c} \overset{c o n f}{\Rightarrow} \bar{S O y}$	$d \overset{c o n f}{\Rightarrow} S O y$ $\bar{d} \overset{c o n f}{\Rightarrow} \bar{S O y}$	$e \overset{c o n f}{\Rightarrow} S O y$ $\bar{e} \overset{c o n f}{\Rightarrow} \bar{S O y}$	$f \overset{c o n f}{\Rightarrow} S O y$ $\bar{f} \overset{c o n f}{\Rightarrow} \bar{S O y}$	$g \overset{c o n f}{\Rightarrow} S O y$ $\bar{g} \overset{c o n f}{\Rightarrow} \bar{S O y}$	$h \overset{c o n f}{\Rightarrow} S O y$ $\bar{h} \overset{c o n f}{\Rightarrow} \bar{S O y}$	$i \overset{c o n f}{\Rightarrow} S O y$ $\bar{i} \overset{c o n f}{\Rightarrow} \bar{S O y}$	$j \overset{c o n f}{\Rightarrow} S O y$ $\bar{j} \overset{c o n f}{\Rightarrow} \bar{S O y}$	$k \overset{c o n f}{\Rightarrow} S O y$ $\bar{k} \overset{c o n f}{\Rightarrow} \bar{S O y}$
$a \overset{0.76}{\Rightarrow} b$ $\bar{a} \overset{0.90}{\Rightarrow} \bar{b}$	$b \overset{0.85}{\Rightarrow} a$ $\bar{b} \overset{0.83}{\Rightarrow} \bar{a}$	$c \overset{0.75}{\Rightarrow} a$ $\bar{c} \overset{0.83}{\Rightarrow} \bar{a}$	$d \overset{0.52}{\Rightarrow} a$ $\bar{d} \overset{0.64}{\Rightarrow} a$	$e \overset{0.80}{\Rightarrow} a$ $\bar{e} \overset{0.87}{\Rightarrow} \bar{a}$	$f \overset{0.40}{\Rightarrow} a$ $\bar{f} \overset{0.55}{\Rightarrow} \bar{a}$	$g \overset{0.48}{\Rightarrow} a$ $\bar{g} \overset{0.61}{\Rightarrow} \bar{a}$	$h \overset{0.41}{\Rightarrow} a$ $\bar{h} \overset{0.56}{\Rightarrow} \bar{a}$	$i \overset{0.42}{\Rightarrow} a$ $\bar{i} \overset{0.56}{\Rightarrow} \bar{a}$	$j \overset{0.41}{\Rightarrow} a$ $\bar{j} \overset{0.55}{\Rightarrow} \bar{a}$	$k \overset{0.74}{\Rightarrow} a$ $\bar{k} \overset{0.82}{\Rightarrow} \bar{a}$
$a \overset{0.78}{\Rightarrow} c$ $\bar{a} \overset{0.81}{\Rightarrow} \bar{c}$	$b \overset{0.86}{\Rightarrow} c$ $\bar{b} \overset{0.86}{\Rightarrow} \bar{c}$	$c \overset{0.74}{\Rightarrow} b$ $\bar{c} \overset{0.90}{\Rightarrow} \bar{b}$	$d \overset{0.49}{\Rightarrow} b$ $\bar{d} \overset{0.69}{\Rightarrow} \bar{b}$	$e \overset{0.73}{\Rightarrow} b$ $\bar{e} \overset{0.90}{\Rightarrow} \bar{b}$	$f \overset{0.35}{\Rightarrow} b$ $\bar{f} \overset{0.59}{\Rightarrow} \bar{b}$	$g \overset{0.46}{\Rightarrow} b$ $\bar{g} \overset{0.68}{\Rightarrow} \bar{b}$	$h \overset{0.40}{\Rightarrow} b$ $\bar{h} \overset{0.63}{\Rightarrow} \bar{b}$	$i \overset{0.41}{\Rightarrow} b$ $\bar{i} \overset{0.63}{\Rightarrow} \bar{b}$	$j \overset{0.37}{\Rightarrow} b$ $\bar{j} \overset{0.60}{\Rightarrow} \bar{b}$	$k \overset{0.69}{\Rightarrow} b$ $\bar{k} \overset{0.86}{\Rightarrow} \bar{b}$
$a \overset{0.52}{\Rightarrow} d$ $\bar{a} \overset{0.64}{\Rightarrow} \bar{d}$	$b \overset{0.55}{\Rightarrow} d$ $\bar{b} \overset{0.64}{\Rightarrow} \bar{d}$	$c \overset{0.54}{\Rightarrow} d$ $\bar{c} \overset{0.66}{\Rightarrow} \bar{d}$	$d \overset{0.56}{\Rightarrow} c$ $\bar{d} \overset{0.64}{\Rightarrow} \bar{c}$	$e \overset{0.78}{\Rightarrow} c$ $\bar{e} \overset{0.83}{\Rightarrow} \bar{c}$	$f \overset{0.45}{\Rightarrow} c$ $\bar{f} \overset{0.56}{\Rightarrow} \bar{c}$	$g \overset{0.52}{\Rightarrow} c$ $\bar{g} \overset{0.61}{\Rightarrow} \bar{c}$	$h \overset{0.50}{\Rightarrow} c$ $\bar{h} \overset{0.60}{\Rightarrow} \bar{c}$	$i \overset{0.44}{\Rightarrow} c$ $\bar{i} \overset{0.55}{\Rightarrow} \bar{c}$	$j \overset{0.46}{\Rightarrow} c$ $\bar{j} \overset{0.56}{\Rightarrow} \bar{c}$	$k \overset{0.71}{\Rightarrow} c$ $\bar{k} \overset{0.76}{\Rightarrow} \bar{c}$
$a \overset{0.83}{\Rightarrow} e$ $\bar{a} \overset{0.84}{\Rightarrow} \bar{e}$	$b \overset{0.86}{\Rightarrow} e$ $\bar{b} \overset{0.80}{\Rightarrow} \bar{e}$	$c \overset{0.79}{\Rightarrow} e$ $\bar{c} \overset{0.82}{\Rightarrow} \bar{e}$	$d \overset{0.54}{\Rightarrow} e$ $\bar{d} \overset{0.62}{\Rightarrow} \bar{e}$	$e \overset{0.52}{\Rightarrow} d$ $\bar{e} \overset{0.64}{\Rightarrow} \bar{d}$	$f \overset{0.55}{\Rightarrow} d$ $\bar{f} \overset{0.67}{\Rightarrow} \bar{d}$	$g \overset{0.75}{\Rightarrow} d$ $\bar{g} \overset{0.83}{\Rightarrow} \bar{d}$	$h \overset{0.53}{\Rightarrow} d$ $\bar{h} \overset{0.65}{\Rightarrow} \bar{d}$	$i \overset{0.46}{\Rightarrow} d$ $\bar{i} \overset{0.59}{\Rightarrow} \bar{d}$	$j \overset{0.52}{\Rightarrow} d$ $\bar{j} \overset{0.65}{\Rightarrow} \bar{d}$	$k \overset{0.50}{\Rightarrow} d$ $\bar{k} \overset{0.62}{\Rightarrow} \bar{d}$
$a \overset{0.45}{\Rightarrow} f$ $\bar{a} \overset{0.50}{\Rightarrow} \bar{f}$	$b \overset{0.44}{\Rightarrow} f$ $\bar{b} \overset{0.50}{\Rightarrow} \bar{f}$	$c \overset{0.48}{\Rightarrow} f$ $\bar{c} \overset{0.53}{\Rightarrow} \bar{f}$	$d \overset{0.60}{\Rightarrow} f$ $\bar{d} \overset{0.62}{\Rightarrow} \bar{f}$	$e \overset{0.43}{\Rightarrow} f$ $\bar{e} \overset{0.48}{\Rightarrow} \bar{f}$	$f \overset{0.40}{\Rightarrow} e$ $\bar{f} \overset{0.51}{\Rightarrow} \bar{e}$	$g \overset{0.56}{\Rightarrow} e$ $\bar{g} \overset{0.64}{\Rightarrow} \bar{e}$	$h \overset{0.48}{\Rightarrow} e$ $\bar{h} \overset{0.57}{\Rightarrow} \bar{e}$	$i \overset{0.46}{\Rightarrow} e$ $\bar{i} \overset{0.56}{\Rightarrow} \bar{e}$	$j \overset{0.45}{\Rightarrow} e$ $\bar{j} \overset{0.55}{\Rightarrow} \bar{e}$	$k \overset{0.78}{\Rightarrow} e$ $\bar{k} \overset{0.82}{\Rightarrow} \bar{e}$
$a \overset{0.50}{\Rightarrow} g$ $\bar{a} \overset{0.59}{\Rightarrow} \bar{g}$	$b \overset{0.54}{\Rightarrow} g$ $\bar{b} \overset{0.61}{\Rightarrow} \bar{g}$	$c \overset{0.52}{\Rightarrow} g$ $\bar{c} \overset{0.61}{\Rightarrow} \bar{g}$	$d \overset{0.78}{\Rightarrow} g$ $\bar{d} \overset{0.80}{\Rightarrow} \bar{g}$	$e \overset{0.56}{\Rightarrow} g$ $\bar{e} \overset{0.64}{\Rightarrow} \bar{g}$	$f \overset{0.55}{\Rightarrow} g$ $\bar{f} \overset{0.64}{\Rightarrow} \bar{g}$	$g \overset{0.58}{\Rightarrow} f$ $\bar{g} \overset{0.61}{\Rightarrow} \bar{f}$	$h \overset{0.67}{\Rightarrow} f$ $\bar{h} \overset{0.70}{\Rightarrow} \bar{f}$	$i \overset{0.58}{\Rightarrow} f$ $\bar{i} \overset{0.61}{\Rightarrow} \bar{f}$	$j \overset{0.64}{\Rightarrow} f$ $\bar{j} \overset{0.67}{\Rightarrow} \bar{f}$	$k \overset{0.42}{\Rightarrow} f$ $\bar{k} \overset{0.48}{\Rightarrow} \bar{f}$
$a \overset{0.45}{\Rightarrow} h$ $\bar{a} \overset{0.52}{\Rightarrow} \bar{h}$	$b \overset{0.49}{\Rightarrow} h$ $\bar{b} \overset{0.54}{\Rightarrow} \bar{h}$	$c \overset{0.52}{\Rightarrow} h$ $\bar{c} \overset{0.57}{\Rightarrow} \bar{h}$	$d \overset{0.57}{\Rightarrow} h$ $\bar{d} \overset{0.61}{\Rightarrow} \bar{h}$	$e \overset{0.50}{\Rightarrow} h$ $\bar{e} \overset{0.55}{\Rightarrow} \bar{h}$	$f \overset{0.66}{\Rightarrow} h$ $\bar{f} \overset{0.71}{\Rightarrow} \bar{h}$	$g \overset{0.57}{\Rightarrow} h$ $\bar{g} \overset{0.61}{\Rightarrow} \bar{h}$	$h \overset{0.55}{\Rightarrow} g$ $\bar{h} \overset{0.64}{\Rightarrow} \bar{g}$	$i \overset{0.49}{\Rightarrow} g$ $\bar{i} \overset{0.59}{\Rightarrow} \bar{g}$	$j \overset{0.53}{\Rightarrow} g$ $\bar{j} \overset{0.63}{\Rightarrow} \bar{g}$	$k \overset{0.53}{\Rightarrow} g$ $\bar{k} \overset{0.61}{\Rightarrow} \bar{g}$
$a \overset{0.44}{\Rightarrow} i$ $\bar{a} \overset{0.54}{\Rightarrow} \bar{i}$	$b \overset{0.48}{\Rightarrow} i$ $\bar{b} \overset{0.56}{\Rightarrow} \bar{i}$	$c \overset{0.45}{\Rightarrow} i$ $\bar{c} \overset{0.54}{\Rightarrow} \bar{i}$	$d \overset{0.49}{\Rightarrow} i$ $\bar{d} \overset{0.57}{\Rightarrow} \bar{i}$	$e \overset{0.46}{\Rightarrow} i$ $\bar{e} \overset{0.55}{\Rightarrow} \bar{i}$	$f \overset{0.56}{\Rightarrow} i$ $\bar{f} \overset{0.64}{\Rightarrow} \bar{i}$	$g \overset{0.50}{\Rightarrow} i$ $\bar{g} \overset{0.58}{\Rightarrow} \bar{i}$	$h \overset{0.55}{\Rightarrow} i$ $\bar{h} \overset{0.64}{\Rightarrow} \bar{i}$	$i \overset{0.57}{\Rightarrow} h$ $\bar{i} \overset{0.62}{\Rightarrow} \bar{h}$	$j \overset{0.71}{\Rightarrow} h$ $\bar{j} \overset{0.76}{\Rightarrow} \bar{h}$	$k \overset{0.49}{\Rightarrow} h$ $\bar{k} \overset{0.55}{\Rightarrow} \bar{h}$
$a \overset{0.46}{\Rightarrow} j$ $\bar{a} \overset{0.50}{\Rightarrow} \bar{j}$	$b \overset{0.46}{\Rightarrow} j$ $\bar{b} \overset{0.50}{\Rightarrow} \bar{j}$	$c \overset{0.49}{\Rightarrow} j$ $\bar{c} \overset{0.52}{\Rightarrow} \bar{j}$	$d \overset{0.58}{\Rightarrow} j$ $\bar{d} \overset{0.59}{\Rightarrow} \bar{j}$	$e \overset{0.48}{\Rightarrow} j$ $\bar{e} \overset{0.52}{\Rightarrow} \bar{j}$	$f \overset{0.65}{\Rightarrow} j$ $\bar{f} \overset{0.67}{\Rightarrow} \bar{j}$	$g \overset{0.57}{\Rightarrow} j$ $\bar{g} \overset{0.59}{\Rightarrow} \bar{j}$	$h \overset{0.73}{\Rightarrow} j$ $\bar{h} \overset{0.74}{\Rightarrow} \bar{j}$	$i \overset{0.71}{\Rightarrow} j$ $\bar{i} \overset{0.70}{\Rightarrow} \bar{j}$	$j \overset{0.67}{\Rightarrow} i$ $\bar{j} \overset{0.75}{\Rightarrow} \bar{i}$	$k \overset{0.47}{\Rightarrow} i$ $\bar{k} \overset{0.56}{\Rightarrow} \bar{i}$
$a \overset{0.76}{\Rightarrow} k$ $\bar{a} \overset{0.80}{\Rightarrow} \bar{k}$	$b \overset{0.80}{\Rightarrow} k$ $\bar{b} \overset{0.78}{\Rightarrow} \bar{k}$	$c \overset{0.71}{\Rightarrow} k$ $\bar{c} \overset{0.77}{\Rightarrow} \bar{k}$	$d \overset{0.52}{\Rightarrow} k$ $\bar{d} \overset{0.61}{\Rightarrow} \bar{k}$	$e \overset{0.77}{\Rightarrow} k$ $\bar{e} \overset{0.83}{\Rightarrow} \bar{k}$	$f \overset{0.39}{\Rightarrow} k$ $\bar{f} \overset{0.51}{\Rightarrow} \bar{k}$	$g \overset{0.52}{\Rightarrow} k$ $\bar{g} \overset{0.62}{\Rightarrow} \bar{k}$	$h \overset{0.46}{\Rightarrow} k$ $\bar{h} \overset{0.57}{\Rightarrow} \bar{k}$	$i \overset{0.46}{\Rightarrow} k$ $\bar{i} \overset{0.57}{\Rightarrow} \bar{k}$	$j \overset{0.43}{\Rightarrow} k$ $\bar{j} \overset{0.54}{\Rightarrow} \bar{k}$	$k \overset{0.47}{\Rightarrow} j$ $\bar{k} \overset{0.50}{\Rightarrow} \bar{j}$

By applying Eq. 8 on data of Table 5 Table 6, the correlations between SOx and SOy are calculated as shown in Table 6. Moreover, a correlation graph of the ABET SOs is depicted in Figure 8, where 0.70 is assumed as a threshold value for drawing a link between a pair of SOs. This threshold value is the minimum value that allows every SO to have a link with at least one SO in Figure 8. In this figure, a thicker link indicates stronger correlation among SOs. Obviously, three SOs clusters appear in the correlation graph. The first cluster involves technical skills SOs (a, b, c, e, k). The second cluster involves soft skills SOs (d, g), and the third cluster involves professional skills SOs (f, h, i, j).

Table 6: Confidence (SOx, SOy)

$a \overset{c o n f}{\Leftrightarrow} S O y$	$b \overset{c o n f}{\Leftrightarrow} S O y$	$c \overset{c o n f}{\Leftrightarrow} S O y$	$d \overset{c o n f}{\Leftrightarrow} S O y$	$e \overset{c o n f}{\Leftrightarrow} S O y$	$f \overset{c o n f}{\Leftrightarrow} S O y$	$g \overset{c o n f}{\Leftrightarrow} S O y$	$h \overset{c o n f}{\Leftrightarrow} S O y$	$i \overset{c o n f}{\Leftrightarrow} S O y$	$j \overset{c o n f}{\Leftrightarrow} S O y$
$a \overset{0.84}{\Leftrightarrow} b$	$b \overset{0.83}{\Leftrightarrow} c$	$c \overset{0.61}{\Leftrightarrow} d$	$d \overset{0.59}{\Leftrightarrow} e$	$e \overset{0.46}{\Leftrightarrow} f$	$f \overset{0.60}{\Leftrightarrow} g$	$g \overset{0.60}{\Leftrightarrow} h$	$h \overset{0.60}{\Leftrightarrow} i$	$i \overset{0.71}{\Leftrightarrow} j$	$j \overset{0.49}{\Leftrightarrow} k$
$a \overset{0.79}{\Leftrightarrow} c$	$b \overset{0.60}{\Leftrightarrow} d$	$c \overset{0.81}{\Leftrightarrow} e$	$d \overset{0.61}{\Leftrightarrow} f$	$e \overset{0.60}{\Leftrightarrow} g$	$f \overset{0.69}{\Leftrightarrow} h$	$g \overset{0.54}{\Leftrightarrow} i$	$h \overset{0.74}{\Leftrightarrow} j$	$i \overset{0.52}{\Leftrightarrow} k$
$a \overset{0.59}{\Leftrightarrow} d$	$b \overset{0.82}{\Leftrightarrow} e$	$c \overset{0.51}{\Leftrightarrow} f$	$d \overset{0.79}{\Leftrightarrow} g$	$e \overset{0.53}{\Leftrightarrow} h$	$f \overset{0.60}{\Leftrightarrow} i$	$g \overset{0.58}{\Leftrightarrow} j$	$h \overset{0.52}{\Leftrightarrow} k$
$a \overset{84}{\Leftrightarrow} e$	$b \overset{0.48}{\Leftrightarrow} f$	$c \overset{0.57}{\Leftrightarrow} g$	$d \overset{0.59}{\Leftrightarrow} h$	$e \overset{0.51}{\Leftrightarrow} i$	$f \overset{0.66}{\Leftrightarrow} j$	$g \overset{0.57}{\Leftrightarrow} k$
$a \overset{48}{\Leftrightarrow} f$	$b \overset{0.58}{\Leftrightarrow} g$	$c \overset{0.55}{\Leftrightarrow} h$	$d \overset{0.53}{\Leftrightarrow} i$	$e \overset{50}{\Leftrightarrow} j$	$f \overset{0.45}{\Leftrightarrow} k$
$a \overset{55}{\Leftrightarrow} g$	$b \overset{0.52}{\Leftrightarrow} h$	$c \overset{0.50}{\Leftrightarrow} i$	$d \overset{0.59}{\Leftrightarrow} j$	$e \overset{80}{\Leftrightarrow} k$
$a \overset{49}{\Leftrightarrow} h$	$b \overset{0.53}{\Leftrightarrow} i$	$c \overset{0.51}{\Leftrightarrow} j$	$d \overset{0.57}{\Leftrightarrow} k$
$a \overset{50}{\Leftrightarrow} i$	$b \overset{0.49}{\Leftrightarrow} j$	$c \overset{0.74}{\Leftrightarrow} k$
$a \overset{48}{\Leftrightarrow} j$	$b \overset{0.78}{\Leftrightarrow} k$
$a \overset{78}{\Leftrightarrow} k$

Interestingly, the discovered three clusters are consistent with the categorization of the SOs reported in the literature, where ABET SOs are intuitively divided into technical skills SOs (a, b, c, e, and k), soft skills SOs (d and g), and professional skills SOs (f, h, i, and j). Moreover, the SOs correlations depicted in Figure 8 are consistent with the findings of ABET that were based on feedback of programs’ educators and feedback, collected via surveys and meetings, and with the corrective action recently taken by ABET, particularly to develop a new SOs for engineering programs. In the new ABET SOs, the SOs (a) and (e) have been combined into a single SO (1), the SOs (f, h, and j) are also combined into a single SO (4), and the SO (k) has been implied in the new SOs (1,2, and 6) that map to (a, b, c, and e) 42.

https://s3-us-west-2.amazonaws.com/typeset-prod-media-server/fb183f33-4f7f-432d-9fab-94eca1522575image8.png — **Figure 8: Correlations among ABET SOs**

Discussion

This paper presents a data analytics approach to analyze the data of two core components, POs and PLOs, of OB engineering programs. It does so by applying ARM technique to a dataset extracted from the SSRs of a number of ABET accredited engineering programs. The aim is to discover the insights regarding the mapping between PEOs and SOs and the correlations among SOs. Generally, two sets of insights have been discovered: the recommended mapping between PEOs and ABET SOs, shown in Table 4 , and the correlations among ABET SOs, depicted in Figure 8 . The recommended mapping between PEOs and ABET SOs is useful as a practical guideline for the academicians and educators of the engineering academic programs while designing a new program or reviewing existing ones. Consider for example the scenario of developing a new engineering program, where an essential requirement is to establish PEOs- ABET SOs mapping. A common practice is to solicit the required mapping from the tenured program faculty. However, arriving at consensus among them is often difficult due to the variations in their level of understanding and interpretation of the PEOs and SOs. Obviously, with the recommended PEOs-SOs mapping, shown in Table 4 , the task of establishing a PEOs-SOs mapping would be much easier and robust. Moreover, the recommended PEOs-SOs mapping, shown in Table 4 , are useful for programs evaluators, who are responsible for evaluating a program seeking for accreditation, to improve the validity of their evaluation. In addition, the recommended PEOs-SOs mapping are informative to ABET decision makers. In fact, it provides them with insights on how the developed ABET SOs are being used in the real context. Finally, the PEOs-SOs mapping can be utilized in developing computer-based systems that could contribute to computer-assisted academic programs design or accreditation.

In addition, the mapping rules can be used to get insights on the correlation among PEOs themselves. For example, by observing that L (Leadership) and T (Teaming) are mapped to the same set of SOs, a correlation between them can be suggested. Similarly, the PEOs Career Success and Technical Competency (CS and TC) depend on the presence of the same combination of technical skills SOs (a, b, c, e, and k), yet differ in their dependence on the soft skills SOs (d, f, g, h, i, and j) and this suggests that they are somewhat correlated. As for PEO Knowledge Competency (KC), it does not show dependency on the presence of any SOs; however, it depends on absence of seven SOs. These insights are useful when developing the PEOs of the academic programs which is an essential step for the design of academic programs.

The second type of insights is the correlations among ABET SOs, shown in Figure 8 . These insights are informative for a better understanding these SOs and how they can be utilized to improve the design of program’s assessment plans. For example, instead of developing assessment plan for each SO a unified assessment plan can be developed for each cluster of SOs. On the other hands, understanding ABET-EAC SOs and their correlation is very useful for designing the curriculum, because in the courses of the curriculum should be mapped to the SOs through COs as shown in Figure 1 . Certainly, the insights on ABET-EAC SOs correlations would lead to improving understanding of the relationship between courses in the curriculum and consequently the sequence and structure of the curriculum. In addition, the ABET SOs correlations are useful for the decision makers of ABET to identify the existing redundancy and overlapping among SOs and consequently take corrective actions to resolve them. In this regard, it’s worth to point out that the ABET SOs correlations, discovered in this research, are consistent with the corrective actions recently taken by ABET, particularly to develop a new SOs for engineering programs. More specifically, in the new ABET SOs, the SOs (a) and (e) have been combined into a single SO (1), the SOs (f, h, and j) are also combined into a single SO (4), and the SO (k) has been implied in the new SOs (1,2, and 6) that map to (a, b, c, and e) 42. Interestingly, all these corrective actions are consistent with the findings presented in this paper. This provides evidence on the validity and utility of the proposed approach to engineering discipline and an indication on the potentiality of applying it to academic programs in other disciplines.

Conclusion

Association rules mining, as a learning analytics method, is proposed to discover useful insights on POs and PLOs of the OB engineering programs. Apriori algorithm, in particular, is applied to a PEOs-SOs mapping dataset extracted from the SSRs of 152 ABET accredited engineering programs. Tow set of insights have been drawn, a recommended mapping rules between PEOs and ABET SOs and correlations among ABET SOs. The recommended mapping rules between PEOs and ABET SOs provides a practical guideline and a systematic method for academicians of engineering programs to better fine-tune the mapping between the PEOs and the ABET SOs. Moreover, the correlation among ABET SOs is very useful to empower the decision-making at program, as well as accreditation agencies level.

In conclusion, besides the simplicity and straightforward application of the proposed approach, it provides evidence on the utility and validity of the learning analytics approaches for distilling academicians’ expertise from a large sample of academic programs distributed all over the world.

[1] Callahan, J . 2016. Objectives and Outcomes Jabberwocky. JOM: The Journal of the Minerals, Metals & Materials Society (TMS) 68(8).

[2] Accreditation Board for Engineering and Technology, Inc. 2019. EAC Mapping C3: A-K to C3 1-7. ABET. https://www.abet.org/wp-content/uploads/2018/03/C3_C5_mapping_SEC_1-13-2018.pdf

[3] Hall, Mark, Frank, Eibe, Holmes, Geoffrey, Pfahringer, Bernhard, Reutemann, Peter & Witten, Ian H. . 2009. The WEKA data mining software. ACM SIGKDD Explorations Newsletter. Association for Computing Machinery (ACM) 11:10–18 https://dx.doi.org/10.1145/1656274.1656278

[4] Osman, A, Yahya, A A & Kamal, M B . 2018. A Benchmark Collection for Mapping Program Educational Objectives to ABET Student Outcomes: Accreditation. In: 5th International Symposium on Data Mining Applications. Advances in Intelligent Systems and Computing.

[5] Baker, S R & Yacef, K . 2009. The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining 1(1):3–17.

[6] Romero, C & Ventura, S . 1995. Educational data mining: A survey from. Expert systems with applications 33(1).

[7] Aldowah, Hanan, Al-Samarraie, Hosam & Fauzy, Wan Mohamad . 2019. Educational data mining and learning analytics for 21st century higher education: A review and synthesis. Telematics and Informatics 37:13–49.

[8] Priyambada, S, Mahendrawathi, E R & Yahya, B . 2017. Curriculum Assessment of Higher Educational Institution Using Aggregate Profile Clustering. In: 4th Information Systems International Conference.

[9] Barbosa, A M, Neto, A N, Santos, E & Gomes, J P . 2017. Using learning analytics and visualization techniques to evaluate the structure of higher education curricula. In: Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE) , 1. 28:1297.

[10] Pechenizkiy, M, Trcka, N, Bra, P De & Toled, P . 2012. CurriM: curriculum mining. In: International Conference on Educational Data Mining. 216–217.

[11] Trcka, N & Pechenizkiy, M . 2009. From local patterns toFrom local patterns to global models: Towards domain driven educational process mining. International Conference on Intelligent Systems Design and Applications .

[12] Deziel, M, Olawo, D, Truchon, L & Golab, L . 2013. Analyzing the mental health of engineering students using classification and regression. 6th International Conference of Educational Data Mining .

[13] Jiang, Y H & Golab, L . 2016. On Competition for Undergraduate Co-op Placements: A Graph Mining Approach. 9th International Conference of Educational Data Mining .

[14] Goslin, K & Hofmann, M . 2013. Identifying and visualizing the similarities between course content at a learning object, module and program level. 6th International Conference on educational data mining .

[15] R & Crues, Wes . 2017. Untangling The Program Name Versus The Curriculum: An Investigation of Titles and Curriculum Content. 10th International Conference on Educational Data Mining .

[16] Zhong-Mei, S, Qiong-Fei, Q & Lu-Qi, F . 2014. Educational data mining and analyzing of student learning outcome from the perspective of learning experience. 7th International Conference on Educational Data Mining .

[17] Soltanpoor, Reza . 2018. An Integrated Framework For Learning Analytics. RMIT University thesis

[18] Siemens, G & Long, P . 2011. Penetrating the fog: analytics in learning and education. Educause review 46(5):30–32.

[19] Baker, Ryan S.J.d. . 2011. Data Mining for Education. In: McGaw, B, Peterson, P & E Baker , eds. International Encyclopedia of Education (3rd). Oxford, UK: Elsevier

[20] Mendez, Gonzalo, Ochoa, Xavier, Chiluiza, Katherine & De Wever, Bram . 2014. Curricular Design Analysis: A Data-Driven Perspective. Journal of Learning Analytics 1(3):84–119.

[21] Agrawal, Rakesh, Imieliński, Tomasz & Swami, Arun . 1993. Mining association rules between sets of items in large databases. ACM SIGMOD Record 22(2):207–216.

[22] Agrawal, R & Ramakrishnan, S . 1994. Fast algorithms for mining association rules in large databases. In: the 20th International Conference on Very Large Data Bases, VLDB.

[23] 2020. Engineers Australia. https://www.engineersaustralia.org.au/

[24] Australia, Engineers . 2007. Engineers Australia Policy on Accreditation of Professional Engineering Programs. Engineers Australia Accreditation Board .

[25] Viberga, O, Hatakkab, M, Bältera, O & Mavroudi, A . 2018. he current landscape of learning analytics in higher education. Computers in Human Behavior 98–110.

[26] Ougiaroglou, S & Paschalis, G . 2012. Association Rules Mining from the educational data of ESOG web-based application. In: IFIP Advances in Information and Communication Technology. (Vol. 382) Springer

[27] Dunham, Margaret H. . 2003. Data mining: Introductory and advanced topics. India: Prentice Hall

[28] Han, J, Kamber, M & Pei, J . 2011. Data Mining: Concepts and Techniques. The Morgan Kaufmann Series in Data Management Systems .

[29] Pe ̃na-Ayala, A . 2018. Learning analytics: A glance of evolution, status, and trends accordingto a proposed taxonomy. Wiley Interdisciplinary Reviews: Data Mining and KnowledgeDiscovery .

[30] Leitner, P, Khalil, M & Ebner, M . 2017. Learning Analytics in Higher Education-A Literature Review. In: Learning Analytics: Fundaments, Applications, and Trends. Studies in Systems, Decision and Control. (Vol. 94) Springer

[31] Haddad, J R, Kalaani, Y & Shahat, A El . 2016. An Optimal Mapping Framework for ABET Criteria 3 (a-k) Student Outcomes into the Newly Proposed (1-7) Student Outcomes. In: IAJC/ISAM International Conference.

[32] Karimi, A . 2017. An Examination of the Proposed Changes to ABET-EAC Criteria. In: ASEE Annual Conference & Exposition.

[33] ABET 2005. 2005 ABET Summit. .

[34] Estell, J & Williams, S . 2011. Program educational objectives: What constitutes sufficient assessment? 2011 ASEE Annual Conference & Exposition .

[35] Petersen, O, Williams, S & Durant, E . 2007. Understanding ABET Objectives and Outcomes. In: 2007 ASEE Annual Conference.

[36] Shivakumar, R., Usha, H. S., Nayak, Chetan A., Sainath, K. & Maitra, Samita . 2015. Establishing Program Educational Objectives. Journal of Engineering Education Transformations 29(2):53.

[37] 1989. The Washington Accord . https://www.ieagreements.org/accords/washington/

[38] 2020. ABET .

[39] Subbaraman, S, Jagtap, R R & Shinde, S S . 2013. Outcome Based Learning: A Case Study. IEEE International Conference in MOOC, Innovation and Technology in Education (MITE) .

[40] Rashid, M H . 2013. The process of outcome-based education-implementation, assessment and evaluations. In: 2013 ASEE International Forum. (pp. 21–67)

[41] Spady, W . 1994. Outcome-Based Education: Critical Issues and Answers. Arlington Virginia: American Association of School Administrators

[42] Thomas, C, Davies, I, Openshaw, D & Bird, J . 2007. Programmed learning in perspective : a guide to program writing. London: AldineTransaction

Learning Analytics of Outcomes-Based Engineering Programs’ Data