10779/rcsi.10786031.v1
Eamon Costello
Eamon
Costello
Jane C. Holland
Jane C.
Holland
Colette Kirwan
Colette
Kirwan
The future of online testing and assessment: question quality in MOOCs
Royal College of Surgeons in Ireland
2019
MOOCs
Educational futures
Tests
Quality
Multiple choice questions
Assessment
Teacher Education and Professional Development of Educators
2019-11-22 16:37:18
Journal contribution
https://repository.rcsi.com/articles/journal_contribution/The_future_of_online_testing_and_assessment_question_quality_in_MOOCs/10786031
<p>If MOOCs are to play a role in the future of higher education it is imperative that we critically examine how they are currently functioning. In particular, questions persist about the role MOOCs will play in the future of formal accredited learning. As the focus turns from informal and free to formal, accredited and paid, greater scrutiny will be brought to bear on the quality of the courses themselves. Although there have been some empirical studies into the quality of MOOCs, a notable gap exists in that such research has not examined Multiple Choice Questions (MCQs) which are a key component of much MOOC assessment and testing. Previous research suggests that flawed MCQ items may compromise the reliability and validity of these assessments, potentially leading to inconsistent outcomes for students. This study was hence designed to examine MCQ quality in MOOCs. 204 MCQs were analysed, from a selection of 18 MOOCs, sampling the domains of computing, social science and health sciences. Over 50% of MCQs (112) contained at least one item flaw; 57 MCQs contained multiple flaws. A large proportion of MOOC MCQs violated item-writing guidelines, which is comparable with previous studies examining the prevalence of flaws in assessments in more traditional educational contexts. The problem of low quality MCQs can be ameliorated by appropriate faculty training and pre- and post-test quality checks. These activities are essential if MOOCs are to become a force that can enable enhanced and improved pedagogies in the future of higher education, instead of simply proceeding to replicate existing poor practices at scale.</p>