AI Summary & Key Takeaways
AI InsightErrors should be acknowledged readily by all, and system owners should take a scientific approach to understanding and learning from them.
📌 3 Key Takeaways
Abstract
Errors in examination papers and other assessment instruments can compromise fairness. For example, a history question containing an incorrect historical date could be impossible for students to answer. Incorrect instructions at the start of an examination could lead students to answer the wrong number of questions. As there is little research on this issue within the educational assessment community, we reviewed the literature on minimizing errors in other industries and domains, including aviation, energy, and medicine. We identified generalizable principles and applied them to our context of educational assessment. We argue that since assessment instrument construction is a complex system comprising numerous interacting components, a holistic approach to system improvement is required. Assessment instrument errors stem primarily from human failure. When human failure occurs, it is not good enough to suggest that ‘to err is simply human’. Instead it is necessary to look deeper, evaluating the latent working conditions that underpin the efficacy of procedures, making the human failure more or less likely. Drawing from the aviation industry’s ergonomic SHELLO model, we articulate and explore three of the most critical working conditions that relate to our context: (i) time pressure, (ii) workload and stress, and (iii) wider organizational culture, including good error data collection. We conclude with recommendations for best practice in minimizing errors in assessment instruments. A ‘good’ error culture should be promoted, which avoids blaming individuals. Errors should be acknowledged readily by all, and system owners should take a scientific approach to understanding and learning from them.
Özet
Errors in examination papers and other assessment instruments can compromise fairness. For example, a history question containing an incorrect historical date could be impossible for students to answer. Incorrect instructions at the start of an examination could lead students to answer the wrong number of questions. As there is little research on this issue within the educational assessment community, we reviewed the literature on minimizing errors in other industries and domains, including aviation, energy, and medicine. We identified generalizable principles and applied them to our context of educational assessment. We argue that since assessment instrument construction is a complex system comprising numerous interacting components, a holistic approach to system improvement is required. Assessment instrument errors stem primarily from human failure. When human failure occurs, it is not good enough to suggest that ‘to err is simply human’. Instead it is necessary to look deeper, evaluating the latent working conditions that underpin the efficacy of procedures, making the human failure more or less likely. Drawing from the aviation industry’s ergonomic SHELLO model, we articulate and explore three of the most critical working conditions that relate to our context: (i) time pressure, (ii) workload and stress, and (iii) wider organizational culture, including good error data collection. We conclude with recommendations for best practice in minimizing errors in assessment instruments. A ‘good’ error culture should be promoted, which avoids blaming individuals. Errors should be acknowledged readily by all, and system owners should take a scientific approach to understanding and learning from them.