Literate Programming (LP)

Donald Knuth proposed LP 11 in 1984. In the literate programming paradigm, code and comments are equally important. The literate program mandate that a programmer’s creative output consist of a document which contains intermingled code and prose differs radically from typical software development models that view the creative output as code in which scattered comments are not required to connect into a coherent whole. A quick glance at the current state of computer programming paradigms will reveal that Knuth’s efforts in LP were not widely adopted. It has been suggested that the failure of adoption of LP for pedagogical use is due, in part, to the lack of sophistication of early LP tools10 . This last point is substantiated by noting that most education-focused research using LP tools took place in the 1990s. Efforts in this area include using LP tools to grade homework submissions 12 , teach programming13 , or write better comments 14 . The work in 14 and 15 reports some success, but both cite student complaints about the difficulty of using LP tools.

Knuth’s initial LP implementation, called WEB, incorporated ideas of LP but had major weaknesses that limited adoption. The first weakness was related to language support. Pascal was the only supported language. The second weakness was related to writing and formatting. WEB used macros and preprocessor directives to create a meta-language “source document” that would render a WEB file into two separate outputs: a human-readable document and source code for compilation. The WEB source document consisted of a series of cryptic formatting instructions (Figure 1(a) ), some of which allowed the inclusion of Pascal code. Knuth’s choice of an additional source document meant that the formatted human- readable document (Figure 1(b) ) and the source code (Figure 1(c) ) had to be created by WEB. Neither the rendered text nor the source code could be edited directly and minor textual edits become major chores. Making minor edits to the formatted document in Figure 1(b) required finding the text in the source document in Figure 1(a) which produced it, editing the offending lines, then regenerate the formatted document to verify that the correct edit was performed. Likewise, modifying the source code in Figure 1(c) requires a similarly laborious process. The “source document” design choice completely isolated authors from their writing. Finally, a third weakness with WEB was that traditional development tools such as debuggers and profilers were extremely difficult to deploy for WEB documents and their associated programs.

Today’s popular document generators do not easily allow elaboration about the internal workings of a program — the primary goal of LP. LP implementations developed after WEB somewhat addressed WEB’s weaknesses in language support and formatting. Some variants support additional programming languages: CWEB (for C), FWEB (Fortran, C, and C++), xmLP (XML), pyWeb, FunnelWeb, nuweb, and noweb (any language). Other tools provided simpler formatting syntax: nuweb uses LaTeX; noweb uses a simpler set of LP directives and also allows use of LaTeX; pyWeb uses restructured text. Pieterse’s survey reviews these and other LP approaches 12. More recently, documentation generated directly from code has become widespread. Documentation generators, such as Doxygen and JavaDoc, produce a document directly from formatted comments within source code, thus overcoming WEB’s second problem. These document generation tools are widely used by programmers, with thousands of programs employing these tools or variants. However, while tools like Doxygen, JavaDoc, and others, were inspired by LP principles 16, they are currently only used to document a program’s external interface (e.g., application programming interface (API)). Investigations into the use of documentation generators to support LP principles are needed.

Cognitive Load Theory

Knuth’s LP paradigm is consistent with cognitive load theory 17, which states that keeping related concepts close, temporally or spatially, can improve the ability of students to grasp difficult ideas. Sweller’s Cognitive Load Theory17, 18, 19 posits that new concepts (termed schema) must be learned in the limited capacity of short-term memory before they can be stored in and employed by long-term memory. The cognitive load of acquiring a new schema can be categorized into several distinct loads. When a schema consists of multiple, interconnected ideas, they must all fit in the limited capacity of short-term memory for learning to occur. The intrinsic cognitive load refers to the number of interconnected ideas which are required to learn a new schema. Poor instructional design can impose extraneous cognitive load, in which the instructional technique unnecessarily increases the number of interconnected ideas, making a schema harder to acquire. When intrinsic load is low, the additional extraneous loading has little effect on learning. However, high intrinsic loading combined with extraneous loading hinders learning. Therefore, instructors should strive to reduce extraneous load produced by their presentation of the material to improve student learning. LP supports instructor efforts by positioning student writing consisting of design decisions, formulas, and figures in close proximity to the source code they are writing.

Approach for Microprocessors

Our investigation made use of a cohort of 58 students enrolled in two sections of the microprocessors course during the Fall 2015 semester. Students self-selected their preferred section based on course scheduling (i.e., timing) preferences. One section, the control group of 27 students, employed no LP methods. The other section of 31 students alternated use of traditional with LP techniques. Specifically, instruction using traditional coding methods was employed for Test 1 material, by developing and discussing code snippets in assembly during in-class exercises. All code written during class was then uploaded to Github 24, a social coding website, providing convenient web access to the code for the students. Next, material for Test 2 was developed using LP techniques to produce a document during in-class exercises. Both the resulting assembly code and its rendering to a LP document were posted on Github and the course website 25, respectively. The same approach was taken for the second half of the course, which employed the C programming language. Test 3 material was covered using traditional techniques, while the material for the final was developed as a set of LP documents. For both cases, the C source and the resulting LP documents were available via the web. Figure 4 shows a comparison between traditional source code (in this case, assembly) used for Test 1 in-class exercises and its LP form used for Test 2 instruction. Likewise, Figure 2 compares traditional C code for Test 3 preparation with the LP variant for the final.

Data Collection for Microprocessors Case

To evaluate student performance, specific questions were selected on the tests and final which require writing snippets of code in C or assembly. In addition, students in the experimental section participated in an anonymous in-class survey given at the end of the course to provide feedback on the use of LP versus traditional code in the course. The questions were:

• How often did you refer to code discussed in class posted at https://github.com/bjones1/ece3724_inclass? Frequently, several times, a few times, once, or never?

• On a scale of 1 to 10, where 1 is the least helpful and 10 is the most helpful, how helpful was providing this code on the web?

• Comments – what would make the code posted more helpful?

• How often did you refer to the code discussed in class posted in LP form at http://courses.ece.msstate.edu/ece3724/in_class? Frequently, several times, a few times, once, or never?

• On a scale of 1 to 10, where 1 is the least helpful and 10 is the most helpful, how helpful was providing this code on the web?

• Comments – what would make the code posted in this form more helpful?

• Which format did you find more helpful? Traditional, literate programming, neither, or don’t care

• In terms of the code discussed, what would help you learn more during this course?

Results for Microprocessors Case

Table 1 shows the test questions used to examine student performance. Note that the control section only employs traditional source code, while the experimental section alternates between traditional and LP. Through the differences reported between the control and experimental sections, we attempt to control for the effects of confounding variables between the sections such as instructor style, student background, and student ability. The mean differences (∆_M ) show a statistically insignificant value of 3% between the two sections for Test 1, during which both sections employed traditional source code. This suggests that effects of confounding variables are small, particularly given the wide standard deviations. For Test 2, which compares traditional to LP instruction, a slightly larger difference of 8% was observed. Treating Test 1 as a baseline difference and Test 2 as the expected effect shows a vertical difference of 5%, suggesting a small, but statistically insignificant improvement using LP. The Test 3/Final comparison suggests confounding variables have a greater impact, with a 14% difference for traditional source code. Further, the effect of LP was a statistically insignificant decrease of 1%.

Students could provide multiple free-form answers to the eight survey questions listed in the previous section. Thus, the survey results were coded based on the categories provided in the questions. Table 2, Table 3, Table 4 show a summary of the frequencies reported for questions 1, 2, 4, 5, and 7. Based on these survey results, students refer to both traditional source code and its LP form with essentially equal frequency. The higher response of 61% for traditional versus 42% for LP in the category of “a few times” represents a difference for students who made little use of either system; the usage frequencies for “frequently” and “several times” are almost identical. Likewise, noting the wide standard deviation, students considered both traditional and LP forms equally helpful. Students reported preferring the LP format over the traditional format; however, given a large group (27%) of “don’t care” responses, this is a mild preference. Given that this is the first exposure students have to the LP format, and noting that all their previous instruction and programming assignments employ the traditional form, students seem willing to embrace the LP approach. This preference is also notable in that several students reported they were unaware of the existence of the LP webpages.

Analyzing the comments provided in response to questions 3, 6, and 8 produced the following four themes:

• Students liked the code organized by topic, rather than class date. This suggestion has been implemented for the Spring 2016 semester.

• Several students stated that they were not aware that the literate programming pages were available on the web, thinking they were only provided for in-class exercises. This response was provided even though students were sent several e-mails giving the address of the LP pages. For the Spring 2016 semester, a change was implemented that required students to access the LP website for every in-class assignment.

• Students requested live updates as the instructors add examples and notes to the LP document in class. Previously, the LP document wasn’t posted to the web until after each class period. This suggestion was implemented for future course offerings.

• Students requested more explanation and more videos. The class is “flipped,” which causes some students to feel cheated that lectures focus on in-class exercises, rather than delivering facts. In addition, some of the older videos need to be updated.

Discussion of Microprocessors Case

The results of our examination of LP in a microprocessors course indicate that literate programming examples may provide some benefit to students. In particular, a student survey shows a mild preference for the LP form when compared to the traditional form. In addition, an analysis of student performance on examinations shows a small, statistically insignificant improvement in student performance when the instructor employs LP for in-class examples. Larger datasets are needed to verify that this effect holds more generally. We note that in this implementation, students were not writing their own code using LP principles; instead, LP was used during the presentation of in-class exercises. A future examination of learning outcomes when students are writing their own literate programs in microprocessors is warranted and may result in expected learning gains.

Approach for Digital System Design

Our investigation made use of a cohort of 36 students enrolled one section of the course during a recent fall semester. The cohort was composed of ten electrical engineering majors who took the class as an elective and twenty-six computer engineering majors for whom the course is required. The course was composed of lecture periods (twice a week; one hour each) and a weekly lab session (once a week; two hours per week). The staffing included a graduate teaching assistant to assist students. Lecture periods were sprinkled with collaborative active exercises. Students were encouraged to work in teams on the lecture active exercises and on laboratory tasks. The instructor often had to expressly direct the collaborative effort as most students would choose to work alone if given the choice. The complexity of VHDL and the overhead of tool chain processes largely prohibited the use of the synthesis tools in-class for the active exercises.

In-class active exercises were mostly focused on the design approach for a particular problem, with students writing “pseudo-code” VHDL. Homework and lab activities consisted of students writing VHDL descriptions annotated with explanations in the CodeChat tool. In previous semesters, students wrote formal lab reports to describe their design approach and results. The addition of LP annotations to their VHDL was the only significant change to VHDL coding assignments compared to previous semesters. The CodeChat tool allowed students to include hyperlinks, figures, tables, equations, FSM diagrams, timing waveforms, etc. directly in their HDL descriptions. Since the code itself was descriptive, and each design’s test bench could be annotated with results obtained from the VHDL tools, students were not required to submit a formal lab report. Each design task had a deliverable of VHDL files that were to be liberally annotated with descriptions, explanations, figures, timing diagrams, and observations.

Data Collection for Digital System Design Case

To ascertain whether LP had an impact on the students’ ability to successfully learn VHDL and capture digital systems behavior, at the conclusion of the course students were asked to respond to a survey. Survey questions are provided in Table 5 . Students responded to the survey questions using five-point Likert-type scales. For questions 1 and 2, students could choose a response from the following list: poor, fair, satisfactory, very good, or excellent. The responses were mapped to the values zero (poor) through four (excellent). For questions 3-6, students were given the choice of strongly disagree, disagree, neutral, agree, and strongly agree. These responses were mapped to values -2 (strongly disagree) to +2 (strongly agree), which allows the sign of the response to indicate whether the response is negative or positive.

Degree program information (i.e., major) was collected to examine how differences in background affect a student’s view of LP as it was used in the digital systems design course. While computer engineering and electrical engineering are very similar, the computer science and programming skills and experience of the computer engineering majors usually is much more substantial than those of electrical engineering students.

Results for Digital System Design Case

Table 6 provides the statistics of the students’ self-assessed abilities by major for Q1 (knowledge at start of course) and Q2 (knowledge at end of the course). A small number of computer engineers (four out of 16) reported they had some knowledge of the subject before the course started, while electrical engineering majors all reported no prior knowledge of the course topics. At the end of the course, most students of both majors reported their knowledge as being “satisfactory” or better. The difference between a student’s response (Q2-Q1) would indicate the course impact on their knowledge. Computer engineering and electrical engineering majors reported a mean difference between Q1 and Q2 of 1.82 and 1.86, respectively.

Table 7 provides a summary of student preferences by major for Q3 (writing helped me learn), Q4 (prefer traditional HW style), Q5 (prefer traditional lab report style), and Q6 (writing helps me improve my design). The results from Q3 indicate that computer engineers were slightly more negative than neutral about the LP approach. Electrical engineers were exactly neutral as a group. Of course, individual students reported “strongly agree” and “agree”, but there were roughly equal numbers of students reporting “strongly disagree” and “disagree”. In Q4, computer engineers indicated a slight preference for homework based on LP-infused VHDL over a set of traditional homework problems. Electrical engineers indicated an even stronger preference for LP and VHDL, although neither group was overwhelming in their preference. Not surprisingly, Q5 responses indicate that both majors are more emphatic about preferring the LP approach to VHDL compared to writing a formal lab report. As with Q4, the preference among the electrical engineers was stronger than the computer engineers. Q6 was worded to try to elicit the student response about LP without ascribing it to the course and lab experience in support of Q3. Both majors were effectively neutral in their views, with electrical engineering majors being slightly positive on average, and computer engineers as a group were nearly neutral. Note that response in Q6 are a bit more positive than the very similar question Q3. Students appear to recognize that writing and natural language may help them express their designs, but the LP approach or negative feelings due to CodeChat installation drove their opinions a bit more negative when the writing involves commenting their HDL.

A Mann-Whitney U test was chosen as a non-parametric test to compare the two populations 26. No significant differences between the two populations were observed on questions Q1-Q6.

Discussion of Digital System Design Case

Students were somewhat ambivalent about the effectiveness of LP to improve their HDL in the digital design course with computer engineering students somewhat more negative. One theory is the computer engineers are more comfortable with the topics covered in the course and with programming languages and documentation than the electrical engineers. The design tasks in the course were not overly complex, so computer engineers may have felt the LP paradigm was simply too much overhead and work for the reasonably simple technical HDL that followed. Electrical engineers are less comfortable in the course described here, and the LP approach may have allowed them to better express themselves. If this is a valid view, one might expect the student enthusiasm to increase as assignments and designs get more complicated. Such a result would also agree with the observation that computer programmers tend to comment more heavily in complicated code, or code that they do not initially understand 27, 28. LP would provide the developer with a more suitable mechanism for human-digestible descriptions.

Students, especially the computer engineering majors who have more experience with computing languages, may have also suffered from the mistaken impression that any computer activity time spent not creating “working code” is a waste of time. With more experience, they may realize that documentation of computer code or VHDL descriptions leads to a much higher maintainability and, ultimately, shorter overall development cycles and lower costs 27, 28.

Anecdotal observations by the instructor and other faculty are that student comprehension and performance seemed largely unchanged compared to previous semesters. The next step would be to form a controlled experiment wherein one group will use traditional editors or IDEs to develop largely undocumented HDL descriptions. A treatment group would use the tools and LP approach described here. Incorporation of formal product metrics such as defect production and development time would also give more objective measures to the quality of student output.

Finally, student performance on other course measures or later academic outcomes might indicate a difference. Alternatively, an ideal, but often difficult to deploy, approach to testing the efficacy of LP in the digital systems design course would be a formal experiment with control and treatment groups of students taught under similar educational conditions during the same academic period. This was not possible due to existing scheduling and faculty workloads. Another approach could create control and treatment groups within the same lecture and lab population. This approach is problematic as students may view different assignments with different levels of difficultly or otherwise detrimental to their grade. Control and treatment groups could be formed by student self-selection, but groups of comparable sizes, abilities, and motivation are very suspect with this scheme. Given the circumstances, a decision was made to apply the treatment to the entire cohort.

Similar to prior LP implementations, concerns with the tool may have overshadowed the benefits of LP. The CodeChat tool was created by one of the authors. The tool is not as polished as commercial or mature open-source software applications. The tool itself works as advertised and is quite stable. Students never reported dissatisfaction with tool functionality. However, installation of the CodeChat tool as problematic. To aid in CodeChat development, the CodeChat tool uses a wide variety of open-source libraries and supporting frameworks. Several of these software packages had varying compatibility issues with student laptops. Students were quite vocal in expressing their frustration with the installation process at the beginning of the semester. The students’ unhappiness was compounded as installation on one student’s laptop would be smooth and flawless, and another student with an identical computer would experience installation problems. Ultimately, several installation problems were a result of students not following the detailed installation instructions provided by the instructor. Eventually, the instructor was able to assist every student and get the tool operational. However, this initial negative experience likely colored the students’ impression of the CodeChat tool and the LP paradigm.