Royal College of Surgeons in Ireland
Why do residents fail simulation-based assessments of operative competence A qualitative analysis.pdf (502.74 kB)

Why do residents fail simulation-based assessments of operative competence? A qualitative analysis

Download (502.74 kB)
journal contribution
posted on 2024-05-07, 15:45 authored by Conor Toale, Marie MorrisMarie Morris, Dara KavanaghDara Kavanagh

This qualitative study explored factors influencing the scoring decisions of raters in a simulation-based assessment of operative competence, by analysing feedback provided to trainees receiving ‘borderline’ or ‘not competent’ global scores across stations in an Objective Structured Clinical Examination (OSCE). Directed qualitative content analysis was conducted on feedback provided to trainees during a novel simulation-based assessment. Initial codes were derived from the domains of the Objective Structured Assessment of Technical Skills (OSATS) tool. Further quantitative analysis was conducted to compare the frequency of codes across feedback provided to ‘junior’ and ‘senior’ general surgery trainees. Thirteen trainees undertook the eight-station OSCE and were observed by ten trained assessors. Seven of these were ‘senior’ trainees in their last 4 years of surgical training, while 6 were ‘junior’ trainees in their first 4 years. A total of 130 individual observations were recorded. Written feedback was available for 44 of the 51 observations scored as ‘borderline’ or ‘not competent’. On content analysis, ‘knowledge of the specific procedure’ was the most commonly cited reason for failure, while ‘judgement’ and ‘the model as a confounder’ were two newly generated categories found to contribute to scoring decisions. The OSATS tool can capture a majority of reasons cited for ‘borderline’ or ‘not competent’ performance. Deficiencies in ‘Judgement’ may require simultaneous non-technical skill assessments to adequately capture. It is imperative that assessors and candidates are adequately familiarised with models prior to assessment, to limit the potential impact of model unfamiliarity as a confounder. 


Open Access funding provided by the IReL Consortium

Royal College of Surgeons in Ireland/Hermitage Medical Clinic Strategic Academic Recruitment (StAR MD) programme



The original article is available at

Published Citation

Toale C, Morris M, Kavanagh DO. Why do residents fail simulation-based assessments of operative competence? A qualitative analysis. Global Surg Educ. 2023;2(1)

Publication Date

29 August 2023


  • Surgical Affairs


Springer Nature


  • Published Version (Version of Record)