Assessing the utility of hybrid simulation with wearable technology and repeated peer assessment on medical student learning and performance in cardiology long case examinations – the assimilate excellence study.

Background: The long case examination in medicine is regarded as an authentic test of clinical competence; however, it has been shown to have low reliability and validity due to the variability in the patients used and subjective examiner grading. In our study, we hypothesised that expert tuition withhybrid simulation using a standardised patient wearing a novel auscultation vest, i.e., a hybrid patient,and repeated peer grading would improve student learning and performance in sequential cardiology long case examinations. Furthermore, we hypothesised that the format’s validity would improve through the use of less subjective quantitative scoring checklists.

Methods: Scripted histories and scoring checklists for three clinical scenarios in cardiology were co-created and refined through iterative consensus by the researcher and a panel of clinical experts; these were then paired with recordings of auscultatory findings from three patients with known valvular heart disease. A wearable vest with embedded pressure-sensitive panel speakerswas developed to transmit these recordingswhen examinedat the anatomically standard auscultation points. Students in the Graduate Entry Medicine degree programme at RCSI were invited to enrol in the study and undertake a series of three long case examinations in cardiology (LC1 –LC3) using hybrid simulation. Each participant’s performance was recorded and graded using the novel scoring checklist by two peer participants and two RCSI examiners. In addition, participants were randomised into two groups: Group 1 received individual and smallgroup teaching with a hybrid patient from an expert trainer between LC1 and LC2; those in Group 2 received the same intervention between LC2 and LC3. Participants completed a pre-and post-study questionnaire. Data are presented as number (%), mean ± standard deviation, or median (interquartile range). Group comparisons were made usingeithertheunpaired t-test or chi-squared (χ2)test. A p-value <.05 was considered statistically significant. Multivariate analysis was undertaken using general linear mixed modelling and multiple logistic regression. Inter-observer variability was assessed using the Intraclass Correlation Coefficient (ICC).

Results: Overall, 77 participants were enrolled in the study and completed the pre-study questionnaire:of these, 68 were included (age 27.6 ± 0.1 years; 74% female sex)and randomised into two groups; there were no significant differences in baseline characteristics between groups. Of those included:68 completed LC1, 59 completed LC2, and 41 completed LC3; of those whowithdrew from the study (n= 36), all did so for reasons related to the COVID-19 pandemic. Overall, the average median examiner score was 39.8% (35.8 –44.6%) in LC1 and increased to 63.3% (56.9 –66.4%) in LC3, i.e., a 59% increase from baseline. The mean total examiner score increased between cases: a greater increase was observed between LC1 and LC2 in Group 1, i.e., in those whoreceived the intervention with an expert trainer (p< .001), when compared to those in Group 2, i.e., those whoundertook peer grading only; however, a greater increase in mean total examiner score was observed in Group 2 between LC2 and LC3 when compared to Group 1 (p< .001). Multiple line arregression was statistically significant (R2=0.51,F(5,53)=11.05,p<.001) and found that the mean examiner total score in LC1(β=0.89,SE0.17,p<.001), malesex(β=-24.95,SE11.48,p=.034), and intervention group (β=-33.49,SE16.65,p=.049) significantly predicted the mean examiner total score in LC2. Across all cases, mean peer total scores were higher than mean examiner total scores. Using the novel checklist, ICC was excellent between examiners’ total scores in all cases: ICC.994–.997(p<.001); the correlation between peer and examiner scores improved in LC2 following participants grading their peers’ performances in LC1:ICC.857–.867(p<.001). Overall, 30 participants completed the post-study questionnaire, i.e., 73% of those whocompleted the study. Participants reported significant improvement in their perceived ability to clinically evaluate cardiac patients (p< .001) and improved confidence in future long case examination performances (p< .001).

Conclusions: In our study that assessed the impact of both hybrid simulation and repeated peer grading with a novel scoring checklist on learning and performance in cardiology long case examinations, theresults show that repeated peer grading improves future performance scores; furthermore, a combination of individualised expert teaching and peer grading improves medical student performance scores over peer grading alone. Thus, we propose that a quantitative scoring checklist for long case examinations has the potential to improve: (a) performance in formative assessments through learning by repeated peer grading; and (b) the validity of long case examinations by improving examiner agreement. Moreover, hybrid simulation in cardiology cases has the potential to (a) improve student confidence through repeated practice and formative assessment in a variety of presentations of valvular heart disease; and (b) improve standardisation of the long case format to allow its more rigorous use in high-stakes summative examinations.