VOL. 33, No. 1 2018
Are current evaluation scheme practices really inclusive of differing teaching and learning preferences and cultural sensitivities? Are students and faculty satisfied with the assignments they have now? Do assignments accurately reflect a student’s learning and skill acquisition? How can students be given assignment options to engage them more fully without increasing workload for faculty? This paper will examine how the flexible weighting option responds to the learning needs of students by promoting their success, building on their strengths, and giving them a sense of ownership and choice. Results of this pilot project have shown that flexible weighting can encourage student engagement and reduce their stress. What does this mean for faculty? Any course with a variety of assignments can implement flexible weighting. Flexible weighting can be successfully applied in courses regardless of the method of delivery and can be adapted for courses in a variety of disciplines.
Les pratiques actuelles des schémas d'évaluation tiennent-elles vraiment compte des différences d'enseignement, des préférences d'apprentissage et des sensibilités culturelles? Les étudiants et les professeurs sont-ils satisfaits des épreuves qu'ils ont maintenant? Les travaux ou examens reflètent-ils fidèlement l'apprentissage et l'acquisition de compétences d'un élève? Comment les étudiants peuvent-ils se voir proposer certains choix permettant de les engager davantage sans augmenter la charge de travail pour les professeurs? Cet article examinera comment l'option de pondération flexible répond aux besoins d'apprentissage des étudiants en favorisant leur réussite, en s'appuyant sur leurs points forts et en leur donnant un sentiment d'appartenance et de choix. Les résultats de ce projet pilote ont montré qu'une pondération flexible peut encourager l'engagement des élèves et réduire leur stress. Qu'est-ce que cela signifie pour le corps enseignant ? Tout cours avec une variété de travaux ou d'examens peut implémenter une pondération flexible. La pondération flexible peut être appliquée avec succès dans les cours, quelle que soit son mode d'administration, et peut être adaptée pour des cours de diverses disciplines.
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
This paper reports on two research studies undertaken to explore the validity and fairness of assignments in online/distance education courses, and the extent to which they allow students to use the perception of their own learning preferences to adopt a flexible weighting assessment method. Based on the changes that have taken place in the mode of delivery (online, blended, etc.) and the nature of the content (online, multimedia, interactive) it was determined that a review of assignments in online, distance education courses was necessary. The two research studies were completed in the spring and the fall of 2015 respectively. The aim of the initial study in the spring of 2015 was to gather feedback from both faculty teaching online and students taking online courses on whether current evaluation scheme practices in online courses at Laurentian University were really inclusive of differing teaching and learning preferences and cultural sensitivities. As well, the study asked whether the participants felt that the assignments accurately reflected their learning. The initial study identified some dissatisfaction on the part of students and faculty with the assignments being presented in online distance courses at that time.
The follow-up study in the fall of 2015 was conducted to determine the potential benefits of a flexible weighting assessment scheme designed to improve a student’s sense of accomplishment in successfully completing an online course. This second research study implemented a pilot project in response to the initial research and was conducted in the 2015F term with the offering of the online distance education course ISWK 2006 EL 12 Indigenous Social Welfare Issues which is a required course in the Honours Bachelor of Social Work, Indigenous Social Work delivered by Laurentian University, Sudbury, Ontario, Canada. This second-year course follows the first-year general Indigenous social work curriculum and provides students with an understanding of the history of oppression and colonization of Indigenous peoples and how this history continues to contribute to the social problems experienced in Indigenous communities to this day.
Flexible weighting is an evaluation method created to try to support students’ sense of ownership and level of investment and engagement with various assessment types in an online course. With the pilot study, the researchers hoped that the choices provided to students would alleviate high levels of anxiety when completing assignments, as students were able to choose a weighting scheme based on their perceptions of their own learning preferences. Initial findings suggest that those students who chose a flexible weighting option not only experienced less stress and anxiety, but improved their outcomes as the majority of students who chose the flexible weighting option also achieved a better grade (a range of 0.2% - 6.5% higher) than they would have if they had followed the default weighting scheme.
This paper contributes to the literature on assessment in online courses by exploring the potential benefits of a proposed flexible weighting assessment scheme on students’ sense of accomplishment in completing the various assessments of an online course in the field of indigenous social work. This discipline has not yet been discussed in the research literature dealing with flexible weighting as courses in this field tend to have smaller enrolments with more varied assessments, such as discussion forum postings and written assignments requiring qualitative evaluations.
Following a brief literature review, this paper will provide background information from the initial research study and then discuss the method and results of the subsequent flexible weighting follow-up study. The paper concludes with a discussion of the findings and suggestions for further research.
In very broad terms, assessment is “a process by which educators use students’ responses to specially created or naturally occurring stimuli in order to make inference about student knowledge, skills or affective status” (Popham & Popham, 2005; Waters & Anderson-Lain, 2014). More specifically, assessments should be varied according to subject matter and discipline, whether the course is taught face-to-face in class or online (Smith, Torres-Ayala, & Heindel, 2008). The discussions around the use of assessments in online courses in the current research literature has mainly centered on identifying how best to structure assessment tools and schemes to ensure that students learn effectively and retain the necessary knowledge from the material presented in the online format. (Cheng, Jordan & Shallert, 2013; Dennen, 2007; Hewson, 2012; Mao & Peck, 2013; Thiede, 2012). Little research has been conducted around tailoring assessment tools to student learning preferences in fully online courses with asynchronous delivery. This paper attempts to address this void by presenting original research into student assessment preferences when choosing flexible weighting options in relation to their perceived learning preferences in a qualitative discipline such as Indigenous Social Work.
Cook (2001) first used the term “flexible assessment” in her analysis of a flexible assessment regime for a course with large enrolments in the School of Economics at the University of Queensland. She concluded that while the flexible assessment approach gave “students some power of choice and thus a feeling of being more in control of their own learning approach” (p. 548), it did not increase the teaching load of faculty members (p. 548) and did reduce the stress levels of some students since they were given some autonomy over how the weighting for their assessments was being calculated (p. 549).
The research presented in this paper builds on the research begun by Cook (2001), and continued by Pacharn, Bay, and Felton (2013), by applying flexible weighting evaluation schemes to a fully online course with a smaller enrolment and in a different discipline. Pacharn et al., (2013) specifically measured both the grades on various course components, along with the final grade in the course and students’ attitudes along with the timing of when the student made the choice to participate in a flexible weighting system in multiple sections of an Intermediate Accounting Course. Students were able to participate either at the beginning or the end of term and also had the choice to not complete assignments that they disliked (p. 152). With the current emphasis on student-centered and self-directed learning (Mao & Peck, 2013; Pacharn et al., 2013; Wanner & Palmer, 2015), the choice presented to students by offering flexible weighting options for various assessment tools both supports and engages students in their learning.
Laurentian University’s Centre for Continuing Education (amalgamated in mid-2015 with the Centre for Academic Excellence) conducted an initial research study of all students completing online courses, and faculty members supervising online courses, in early 2015 to determine whether current evaluation scheme practices were really inclusive of differing teaching and learning preferences and cultural sensitivities. The study was also undertaken to determine whether students and faculty were satisfied with the assignments they were completing and grading. Currently, at Laurentian University, the Curriculum Development Team working with the content expert for various online courses creates different types of assignments such as the creative project assignment in ISWK 2006 that will be discussed further in this paper. More varied and creative options are continually being encouraged, as well as options for students to submit their work for assessment in different multi-media formats.
The initial research survey asked both students who had completed online distance education courses, and faculty members who had supervised online distance education courses, to reflect on the evaluation scheme of their respective courses (formative and summative assessments) and to provide their thoughts and opinions on the following questions:
The survey was sent to students and faculty after final grades were posted – once in February 2015 for courses ending in December 2014 and again in May 2015 for courses ending in April 2015 — using Laurentian University’s in-house survey tool REDCap. Students in fully online courses in the 2014F, 2015W and 2015FW academic sessions, along with the faculty members supervising the online courses in the 2014F, 2015W and 2015FW academic sessions, were asked to participate in the survey. Responses were received from 26 students and 14 faculty members representing a 1.5% response rate for students and a 37% response rate for faculty. Even though a very low number of students responded to the survey, the results of this initial study indicated that students all have different ideas of a fair and effective evaluation scheme—some want several, smaller assignments to keep them engaged, some want more substantial assignments that require them to immerse themselves in research. Furthermore, some students wanted to utilize digital media and social networking, while others wanted to write in traditional forms.
The faculty respondents indicated that there is more work in an online course because of the time needed to monitor the course website (i.e., discussion forums and email) on a regular basis. Many faculty also felt that grading discussion forum postings can take up disproportionate amounts of time. One faculty member also stated that assessment with a “blend of developmental, narrative and testing” was fair and effective because it allowed different types of students to express themselves. One of the faculty members remarked that “Ultimately it is important to use team approach and update as new knowledge and experience is acquired.”
The researchers concluded that in order to meet the needs of all students there should be an element of choice in how students are evaluated in order to provide some ownership and flexibility. The challenge of providing choice is that there is difficulty in grading diverse formats and multi-media submissions at different times. Another challenge is ensuring that consistency in the fairness of the evaluation for the student is maintained while providing options.
Flexible Weighting was designed to address the issues identified in the initial research — to allow students to have more emphasis attached to the assessment elements that they felt most accurately reflected their learning, to allow for different ways of expressing that learning, and to provide options for students without overburdening faculty already feeling the stress of constant online monitoring.
For flexible weighting to be effective in addressing differing learning preferences of students, it is necessary to have two or three different types of assignments all of which differ in methods and focus. These assignments should be designed to appeal to a wide range of students with different learning preferences. The assignments must require a similar amount of time studying and researching and reflect achievement of the learning outcomes. Faculty members should choose the default weighting based on how they think the majority of students would prefer to demonstrate their learning. The weighting must vary by approximately 5% and any decisions about essential elements must be made before locking in the options available to students. For example, keeping the exam at a set percentage for all students.
The Director of the School of Indigenous Relations was approached in early summer of 2015 to see if ISWK 2006 EL 12 “Indigenous Social Welfare Issues” could be involved in a pilot project to find out how flexible weighting might improve student success in their assignments. This course was chosen because of the varied nature of the assignments already in place and the willingness of the faculty member teaching the course to implement flexible weighting. Approval was received and the pilot project was scheduled for the Fall term of 2015.
The original evaluation scheme for ISWK 2006 EL 12 consisted of discussion forum postings worth 20%, two major assignments worth 20% and 30%, respectively, and a final exam worth 30%. The first assignment was designed to allow students to express themselves in a creative way. Students were asked to submit a creative project, which could be in the form of a piece of art (painting, sculpture, song, music, collage, etc.), a display, a video, storytelling (book), etc., that captured their feelings and reactions to some of the social issues and social policies explored in the course, specifically, how an issue has impacted the student in some way and has meaning to that student. This type of assignment was felt to be in line with Indigenous traditions of art and storytelling. Not surprisingly, in previous offerings, there was a wide range of responses from students. Some students used their skills in arts and crafts to create physical expressions of the concepts under discussion. Others, who were more comfortable with the written word created poetry. This assignment also required a written description of the submission to ensure the academic rigour of this type of assignment.
The second assignment for ISWK 2006 EL 12 was a term paper based on the content of the course and asked students to clearly demonstrate their knowledge of the history of Indigenous people within Canada and how this has affected the way social services are delivered in Indigenous communities.
Another element of the evaluation scheme was a number of discussion postings in response to questions in the course material, including debate points. The final element was the final exam. In theory, the proposed flexible weighting assessment scheme would look as follows: assuming four assignments, the faculty member supervising the course would decide how many could be flexed (two, three, or four). Prior to the submission of the first assignment, students had the option to change the weighting of their assignments. The course supervisor would grade all assignments in the same order using the same grading scheme. The grade achieved would then be given the weighting requested by the student. Grades could be exported to a predesigned Excel spreadsheet and, at the same time, any flexible weighting requests could be entered in order to automatically calculate grades.
For students, the proposed flexible weighting assessment scheme would provide choice about how they would be evaluated, which could alleviate stress over difficult elements; encourage them to learn about their individual learning preferences; and enable them to use them to their advantage. Furthermore, it could help to remove any stigma attached to accommodation for learning disabilities.
For faculty teaching the course, the only addition to their workload would be in the initial creation of different types of assessments and ensuring that the grades were entered into the appropriate Excel spreadsheet, prepared by the Online Course Technician and Instructional Designer. Although students are provided with the flexible weighting options, the faculty continue to grade the assignments as they normally would. This would ensure consistency in evaluation.
Based on the values of the existing assignments the faculty created the default marking scheme ensuring a small differential in the weightings. The online course technician created the Excel spreadsheet, which would convert student grades based on their selected option. Detailed instructions were created to ensure that students were able to make an informed decision. It was decided that should students’ grades be lower because of their participation in the pilot then they would be allowed to request reverting back to the default scheme, i.e., the original method of evaluation.
Students were advised that the weighting of their assignments is the percentage value each assignment was worth. For example, the student might write an essay, which the faculty scored out of 50 points. However, it might be worth only 30% of the student’s final grade, so 30% is the weighting. The weighting of the final exam was set and could not be changed. Students were expected to complete all assignments in the same order, with the same due dates, and they would be scored out of the same total number of points, regardless of the weighting (percentage) the student chose.
Students had the option to change the weighting of their assignments by choosing among five weighting options (see Table 1), depending on which assignment they thought would represent their best performance. For example, if students considered themselves to be very creative or artistic and felt that they could express themselves better through their art than their writing, they might choose to give the Creative Project a weighting of 30%. If students considered themselves better at writing, especially in interactive situations, and did not feel that they were very artistic, they might choose to give the Discussion Forum postings the 30% weighting. Students had to use 15%, 20%, and 30%. They could not change the number but they could move the number to ensure that they did well in the course.
Students were asked to indicate their flexible weighting scheme choice by the end of the second week of the course and were also advised that Flexible Weighting was optional. If they decided not to make any changes, they were not required to do anything because they would automatically be given the default weighting scheme. At the end of the course (or within one week of receiving the final mark for the essay), students who felt that the flexible weighting scheme they chose had a negative effect on their grade were asked to contact the faculty supervising the course and ask to revert to the default weighting scheme.
Table 1 outlines both the default scheme and the five options of flexible weighting given to students.
Table 1: Default and Flexible Weightings Schemes |
||||||
Assessment Type |
Default |
Option 1 |
Option 2 |
Option 3 |
Option 4 |
Option 5 |
Discussion Forum Postings |
15% |
15% |
30% |
20% |
20% |
30% |
Assignment 1 |
20% |
30% |
15% |
30% |
15% |
20% |
Assignment 2 |
30% |
20% |
20% |
15% |
30% |
15% |
Final Exam |
35% |
35% |
35% |
35% |
35% |
35% |
Once the course was completed, a link to an online questionnaire was sent out to all students enrolled in the course, whether or not they had chosen a flexible weighting scheme option, requesting feedback and asking them to reflect on the effects of the flexible weighting scheme option. The questionnaire consisted of 10 questions that asked students to comment on the clarity of the instructions, their impressions of how having a flexible weighting scheme option benefitted them with their coursework, and if they had any suggestions for improvements in further offerings of the flexible weighting scheme options. Students were also asked if they were interested in being contacted by phone to review their answers to the survey and expand on their comments. The questionnaire consisted of ten questions and was conducted using Laurentian University’s in-house survey tool REDCap.
The pilot project ran very smoothly with 28 of the 61 students, or 46% of the enrolled students, choosing a flexible weighting scheme option. Table 2 summarizes the options selected by the 28 students.
Table 2: Weighting Scheme Options and Student Distribution |
||||||
Weighting |
Default |
Option 1 |
Option 2 |
Option 3 |
Option 4 |
Option 5 |
Number of |
31 |
12 |
1 |
10 |
3 |
2 |
Discussion |
15% |
15% |
30% |
20% |
20% |
30% |
Assignment 1 |
20% |
30% |
15% |
30% |
15% |
20% |
Assignment 2 |
30% |
20% |
20% |
15% |
30% |
15% |
Of the 28 students who chose one of the flexible weighting scheme options, 26 students, or 93%, improved their marks with a range of 0.2% to 6.5%. The default mark was given to the two remaining students as it turned out to be higher than the option they chose.
Of the 61 students 16, or 26% of the students enrolled, completed the online questionnaire. Of those respondents, 10 had chosen one of the five flexible weighting scheme options while the remaining six had not. Of the responses received, 15 of 16 of respondents would consider a flexible weighting scheme option again if given the opportunity, regardless of whether they had chosen it this time and whether or not they benefitted from it.
Question 3 of the questionnaire asked students about what motivated them to choose flexible weighting and Figure 1 presents the results indicating that the majority of students made their choice by evaluating their perceived learning strengths.
Figure 1: Graph indicating the reasons students gave for their decisions to choose a flexible weighting scheme option.
The most significant question asked was Question 5: How do you think choosing the flexible weighting scheme affected your work? e.g., How did the flexible weighting scheme make you feel about the assignments and the course? Figure 2 shows the distribution of student responses indicating that the majority of students felt less stressed (seven responses) and more involved (five responses).
Figure 2: Distribution of student responses indicating students’ perception of how choice
of a flexible weighting scheme option affected their work.
In addition, students who chose flexible weighting said: “…I was able to put the most energy into the assignment I most enjoyed without being afraid of losing points for doing so” (P13), “Never again” (P2), “I thought this was a great way to allow students an opportunity to use their strengths to demonstrate course knowledge” (P10), and “Strength based! Made me feel, like, wow my learning style finally matters!” (P6).
Students who did not choose flexible weighting said: “I did not use flexible weighting as I agreed with the pre-set weighting options. Had I known I had a creative side I would have used the flexible weighting option.” (P9), “I thought it was great that this was an offered option.” and “I was empowered to take the course based on my strengths as a student, which I think is key in learning” (P15).
All sixteen respondents to the questionnaire felt the instructions for flexible weighting were clear, allowing them to make an informed decision. Two students were subsequently interviewed by phone and expanded on their survey answers by stating that they both appreciated the choices provided. One student also spoke at length about wanting the exam to be included because of the stress involved in taking a final exam. The choice of giving this option would depend upon the faculty supervising the online course. One student commented that relieving the stress allowed them to relax and do better on all the assignments.
Both the quantitative and qualitative feedback received from the students support the assumption that providing options, in this case differing weighting schemes for course assessments, would lead to a reduction in stress and a strengthened sense of ownership by students. By giving five options for a flexible weighting scheme, students were able to choose a scheme that fit their perceived individual learning preferences, creating a more inclusive environment for learning in an online course. The options gave students a more significant amount of choice when compared with the four flexible assessment options described by Cook (2001), who also stated that students did not have to choose an option as their scores were automatically calculated based on the four options presented and “…the maximum of the four options was used as the mark for the semester” (p. 544). While Cook (2001) discussed a face-to-face course using online components, this paper discusses a fully online course, showing that flexible weighting can be successfully applied in courses regardless of method of delivery and can be adapted for courses in a variety of fields.
Pacharn et al.’s work (2013) focused on student autonomy and inclusion in the decision-making process in accounting courses. In contrast, our research study looked at how having options and making choices could provide the benefits of reducing anxiety over assignments that students find particularly difficult. Unlike the Pacharn et al. (2013) study, we did not allow students to omit an activity – instead, the pilot project was designed to encourage students to be as successful as possible on every component of the assessment scheme. We also took the very practical approach that online courses can mean a high volume of grading, so an element of choice for students which does not lead to an increased workload for faculty was accomplished by setting up the grading tool in the LMS used at Laurentian University in such a way that final grades were calculated automatically, regardless of the flexible weighting option chosen by the student.
The following questions have been raised in discussion with faculty members and could be the basis for further research: Should students be allowed to choose any time during the term? Should students just receive the best mark anyway (i.e., just use an algorithm without actual options given to students)? Should the final exam be included in the flexible weighting?
Cheng, A-C., Jordan, M., & Shallert, D. (2013). Reconsidering assessment in online/hybrid courses: Knowing versus learning. Computers & Education, 68, 51-59.
Cook, A. (2001). Assessing the use of flexible assessment. Assessment & Evaluation in Higher Education, 26(6), 539-549.
Dennen, V. (2007). Looking for evidence of learning: Assessment and analysis methods of online discourse. Computers in Human Behavior, 24, 205-219.
Hewson, C. (2012). Can online course-based assessment methods be fair and equitable? Relationships between students’ preferences and performance within online and offline assessments. Journal of Computer Assisted Learning, 28, 488-498.
Mao, J., & Peck, K. (2013). Assessment strategies, self-regulated learning skills, and perceptions of assessment in online learning. The Quarterly Review of Distance Education, 14(2), 75-95.
Pacharn, P., Bay, D., & Felton, S. (2013). The impact of a flexible assessment system on students’ motivation, performance and attitude. Accounting Education: An International Journal, 22(2), 147-167.
Popham, W. J., & Popham, J. W. (2005). Classroom assessment: What teachers need to know. Boston: Pearson/Allyn and Bacon.
Smith, G. G., Torres-Ayala, A., & Heindel, A. (2008). Disciplinary differences in e-elearning instructional design: The case of mathematics. Journal of Distance Education, 22(3), 63-88.
Thiede, R. (2012). Best practices with online courses. US-China Education Review, A(2), 135-141.
Wanner, T., & Palmer, E. (2015). Personalising learning: Exploring student and teacher perceptions about flexible learning and assessment in a flipped university course. Computers & Education, 88, 354-369.
Waters, S., & Anderson-Lain, K. (2014). Assessing the student, faculty and community partner in academic service learning: A categorization of surveys posted online at campus compact member institutions. Journal of Higher Education Outreach and Engagement, 18(1), 89-122.
Authors
Bettina Brockerhoff-Macdonald is the current Director of the Centre for Academic Excellence at Laurentian University and oversees both the development and delivery of online programs and courses, as well as supporting faculty with teaching with technology both in the classroom and online. She has taught as a sessional instructor in the Department of Modern Languages and Literatures and holds an Honours B.A. and M.A. from McMaster University and a PhD in Human Studies from Laurentian University. E-mail: bbrockerhoff@laurentian.ca
Moira Morrison trained as a teacher in the UK. She moved into the field of Distance Education when she joined the Centre for the Continuing Education (now amalgamated with the Centre for Academic Excellence) at Laurentian University in 1994. In 2010, Moira completed a Masters in Online and Distance Education through the Open University in the UK consolidating her knowledge in the field and providing the invaluable opportunity of being an online student herself. E-mail: mmorrison@laurentian.ca
Susan Manitowabi is the Director of the School of Indigenous Relations at Laurentian University in Sudbury, Ontario. Her doctoral work focuses on the engagement of the “Raising the Spirit” Mental Wellness Team First Nations with First Nations mental health programs. Her research interests include: Aboriginal mental health; traditional Aboriginal healing practices; Aboriginal child welfare; Indigenous research; and Aboriginal mental health policy development and community development. E-mail: smanitowabi@laurentian.ca