Benchmarking Course Completion Rates: A Method with an Example from the British Columbia Open University

Louis Giguère

VOL. 22, No. 1, 73-86

Abstract

We report findings on the methodological phase of a research project designed to assess the progress of the British Columbia Open University (BCOU) toward a 1997 goal of increasing distance education course completion rates to British Columbia system levels by adapting existing 'off-line' courses for online delivery (a virtualization strategy). The method consists of identifying benchmarking factors statistically through the regression of 15 institutional factors on course completion rate data, and using these factors to establish 'off-line' course completion rate benchmarks. Off-line courses are print-based independent study courses providing instructional support through mail, e-mail, and telephone. The second phase of the project will benchmark BCOU online courses against their off-line counterparts. The 'off-line' completion rates data comprise 137 courses and 23,709 course enrolments and represent a 3-year period. Stepwise linear regression includes 8 factors and accounts for a significant amount of total variation (46.8%). The regression organizes benchmarks effectively along three Course Level and Subject Matter classification factors. We set BCOU benchmarks accordingly and compare them to the BC system. We highlight some unexpected results, including off-line completion rate benchmarks increasing over time. This suggests that BCOU's off-line courses are making their own contribution toward BCOU's goal of increasing distance education course completion rates to British Columbia system levels.

Resumé

Cet article rapporte les résultats de la phase méthodologique d'un projet de recherche conu pour évaluer les progrs de la British Columbia Open University (BCOU) vers son objectif d'augmenter les taux de complétion des cours à distance en regard des niveaux du systme universitaire de la Colombie Britannique, en adaptant les cours hors ligne pour en faire des cours en ligne (stratégie de virtualisation). La méthode consiste à identifier les facteurs d'étalonnage de faon statistique grâce à une régression sur quinze facteurs institutionnels à partir des données de complétion des cours, et en utilisant ces facteurs pour établir un étalonnage des taux de complétion de cours hors ligne. Les cours hors ligne sont sur support imprimé et de type auto-apprentissage avec encadrement par la poste, par courriel et par téléphone. La deuxime phase du projet étalonnera les cours en ligne de la BCOU en opposition à leurs contreparties hors ligne. Les données de taux de complétion hors ligne proviennent de 137 cours et de 23 709 inscriptions sur une période de trois ans. La régression linéaire comprend huit facteurs et rend compte d'un pourcentage significatif de la variation totale (46,8%). La régression organise l'étalonnage selon trois facteurs « Niveau de cours » et trois facteurs « Matière. » Nous avons établi l'étalonnage en conséquence et l'avons comparé à celui du systme universitaire de la Colombie Britannique. Nous mettons en exergue des résultats inattendus, comme l'augmentation dans le temps de l'étalonnage des taux de complétion des cours hors ligne. Ceci suggre que les cours hors ligne de la BCOU apportent leur propre contribution au but d'atteindre les niveaux du systme universitaire de la Colombie Britannique pour les taux de complétion des cours.

Introduction

Historically, distance education courses have exhibited low course completion rates relative to 'traditional' face-to-face education (Beatty-Guenter 2001, Carr 2000, Holmberg 1995, Howell et al., 2004, Keegan 1986, Munro 1991, Paul 1986, 1990, Zajkowski 1997). Ten years ago the British Columbia Open University (BCOU) resolved to close the completion rates 'gap' by upgrading its distance education platform and adapting its off-line courses for online delivery. BCOU off-line courses are print-based independent study courses that provide instructional support through mail, e-mail, and telephone. Research in the late 1990s suggested high completion rates for on-line courses, including claims of 'equivalency' with face-to-face course delivery (Card and Horton 1998, Eliesen and Black 1997, 1998a, 1998b, Fredda 2000a, 2000b, 2000c, McDonald and Gibson 1998, Rosberg 1997, Sener 1996). In 1997 the Open Learning Agency (OLA), the BCOU's parent institution, resolved to put together a comprehensive on-line distance education platform to increase course completion rates to provincial standards, with the proviso that it would do so without additional resources or without sacrificing access and flexibility. The OLA and BCOU stated goal was to increase completion rates "by an amount sufficient to ensure that within two years (i.e., by March 1999) they meet or exceed the minimum as established by the other institutions in the BC College, Institute and Agency system (controlling for program area and using system-wide key performance indicators.)" Open Learning Agency, p. 3, (1997).

This paper reports on the first phase of a research project designed to assess BCOU's progress toward that stated goal. Here we develop a benchmarking method by regressing 15 institutional factors on successful course completion rates and we benchmark course completion rates for BCOU's off-line distance education courses accordingly. We compare these benchmarks to BC system statistics. The second phase of the project will benchmark BCOU's online distance education against their off-line counterparts, and will examine the impact of online course delivery on course completion rates. Benchmarking is a necessary step to course completion rate comparisons since intra-institutional course completion rates are highly variable (Howell et al., 2004.)

The British Columbia Open University

In its final days, the BCOU was a comparable institution to TéléUniversité (Québec) and Athabasca University (Alberta). It was a single mode distance education institution offering flexible learning opportunities through credit banking and distance education. BCOU's course enrolment was slightly less than TéléUniversité and significantly less than Athabasca University (by a factor of about two and a half).

The British Columbia Open University Off-line Independent Study Courses

BCOU off-line independent study courses rely extensively on print, audio, and video instructional materials as well as textbooks. Students are given the choice of contacting a tutor via mail, telephone, or e-mail. Telephone calls are received primarily during scheduled twice-weekly tutor office hours. Tutors are required to respond to student e-mails within two working days as well as to return assignments submitted via e-mail within three working days. Though some students still participate in BCOU courses in the traditional fashion (i.e., without e-mail support), e-mail communication has become more and more prevalent in student-tutor and student-staff interactions. Administrative support is available by phone during business hours (usually from 8:30 AM till 4:30 PM from Monday to Friday).

Methods

Data

Defining Completion Rates

BCOU's successful completion rates (SCR) were therefore calculated as follows:

The total number of students who obtained a passing grade on an ABCD scale, divided by the total number of students who started (i.e., the number of students who obtained a passing grade on the ABCD scale plus those who received an Incomplete or a Fail grade) times 100.

This course completion rate statistic excludes "W" grades as defined by 'open and distance education' institutions like the BCOU, i.e., a grade given to students who did not complete any graded academic work.

Analysis

Stepwise linear regression was used to analyze the data and only those factors associated with SCRs at a 0.05 level of statistical significance are reported here.

The Institutional Factors

The institutional factors consist of course factors and tutor factors. The course factors are: Level, Subject Matter, Number of Credits, Year of Delivery as well as a unique number for each course (Course Number). They are coded as follows:

* ABE = Adult Basic Education

The Subject Matter is coded as follows: Applied Academic areas (Business Administration and Business computer courses) are coded as 1. Liberal Arts subject matters are coded from 2 to 4 according to perceived level of quantitative difficulty.

The tutor factors are: Course Load (the number of courses supported by a tutor for that year), Student Load (the number of students supported by a tutor for that year), Tutor Age (in 2002), Years of Service at OLA, Gender, Place of Residence, Academic Qualifications, as well as a unique number for each tutor (Tutor Number). They are coded as follows:

Results

Identifying Benchmarking Factors from Among Institutional Factors

The regression is summarized in Table 1. Eight steps account for 46.8% of the variation. This is a relatively high percentage considering that the data range widely (minimum 6.7% and maximum 100%). Previous attempts to regress factors on SCRs have accounted for 16.2% (Ostman and Wagner, 1987) and 32% of the variation (Sweet, 1986; for Open Learning Institute, the precursor institution to the OLA and the BCOU).

Table 1: Linear Regression of Institutional Factors on SCRs

Since Course Level and Subject Matter factors account for 40.6% of the variation before tutor specific or course specific factors are loaded, they represent good prospects for benchmarking.

Step 3 of the linear regression can be used to predict SCR benchmarks according to the general formula for regression:

Y = a + (b . X1) + (c . X2) + (d . X3) Equation 1

where X1 represents Course Level: Subfactor 2 codes, X2 the Subject Matter codes, and X3 Course Level: Subfactor 1 codes. The regression coefficients are shown in Table 2.

Table 2: Linear Model for Predicting BCOU SCR Benchmarks Based on Course Level and Subject Matter factors (from Step 3 of regression)

This model generates the following equation and the following SCR benchmarks (shown in Table 3 against actuals):

SCR = 61.93 + (4.81 . X1) + (-3.73 . X2) + (11.73 . X3) Equation 2

Table 3: BCOU SCR Benchmarks*: Benchmarks over Actuals (sample size in parenthesis). (Benchmarks based on linear model described in Equation 2.)

*Off-line independent study academic courses with mail, e-mail, and telephone support

The fit between the linear regression model and actuals is within a few percentage points for each pair. Two instances exceed 5 percentage points: first year Social Sciences actuals are 6.8 percentage points higher than prediction, and second year Science and Mathematics courses actuals are 9.2 percentage points below prediction. This suggests a possible course design concern for BCOU's second year Science and Mathematics courses.

Note that actuals shown in Table 3 are based on the same data used to generate the linear model itself. Thus they do not represent an independent test of the linear model. The fact that the two sets of data show remarkable linear congruence across five course level and four subject matter categories does however strongly support a benchmarking framework based on Course Level and Subject Matter classification. Note also that we could have successfully tested the 2002/03 data against independent 2003/04 or 2004/05 data.