Evaluation of the quality of information provided by ChatGPT on pelvic and acetabular surgery

Background: Pelvic and Acetabular fractures are complex injuries commonly caused by both high-energy mechanisms in young patients and low velocity trauma in older patients, which can result in significant morbidity and mortality for both. This study aimed to assess the information provided by ChatGPT on pelvic and acetabular surgery using standard scoring systems to assess the quality, reliability, and readability of the content. We hypothesized that while ChatGPT would generate information of high quality, its readability would be low.

Methods: An open AI model (ChatGPT) was used to answer 20 commonly asked questions from patients about Pelvic and Acetabular surgery. These answers were evaluated for medical accuracy, quality, and readability using the JAMA Benchmark criteria, DISCERN score, Flesch-Kincaid Reading Ease Score (FRES), and Grade Level (FKGL).

Results: The JAMA Benchmark criteria score was 0, the lowest score, indicating no reliable resources cited. The DISCERN score was 42.5, which scores information of fair quality. The area the open AI model scored lowest was the reliability portion of the DISCERN score due to a lack of resources. The FRES was 52.9, and the FKGL was at a 10th-12th grade reading level.

Conclusion: A 10th to 12th grade reading level was required to comprehend the information provided by ChatGPT with regards to Pelvic and acetabular surgery, and the evidence supplied was of fair quality. With no citations provided, it remains unclear where these answers originate and how scientifically valid they are with respect to the latest literature. In spite of this, ChatGPT did safety-net patients by encouraging the importance of further discussion with a surgeon.