PO:38:283 | Accuracy of chatgpt 4 omni in providing clinical recommendations in musculoskeletal care: a valuable support for clinicians and patients?

Dario Taborelli¹, Silvia Negro¹, Federico Padovani², Tiziano Innocenti^1|3, Stefano Salvioli¹ | ¹DINOGMI, Università di Genova, Savona; ²Dipartimento di Neuroscienze e Riabilitazione, Università di Ferrara; ³GIMBE Foundation, Bologna, Italy

Background. Generative artificial intelligence models, such as ChatGPT-4 omni, is gaining popularity in healthcare and could provide valuable support for the clinical management of widespread diseases in the population, such as osteoarthritis. However, its accuracy in providing up-to-date clinical recommendations remains underexplored, particularly in the context of osteoarthritis (OA), one of the most common conditions with a high socio-economic impact, where it is critical that recommendations are up-to-date, clear and consistent with scientific evidence. Objective: to evaluate the accuracy and consistency of the clinical recommendations generated by ChatGPT-4 omni in relation to the most recent international guidelines for osteoarthritis (NICE, EULAR, OARSI) and to compare them with the responses given by Italian physiotherapists.

Materials and methods: cross-sectional study based on 24 clinical questions derived from OARSI, NICE and EULAR guidelines. The questions were presented to ChatGPT-4o in the form of ‘classic’ and ‘medical’ prompts, in Italian and English, for a total of nine administrations each. A complex clinical case was also evaluated. Accuracy, internal reliability (SD, CV%) and concordance (Cohen's K) were calculated.

Results: ChatGPT-4o showed good internal consistency (CV < 25%) and moderate to high agreement between Italian and English (K up to 0.902). Accuracy ranged from 66.7% to 79.2%, higher in classical and English prompts. Agreement with Italian physiotherapist was low (K = 0.106). In the clinical case, the mean accuracy was 81.4%, but answers were only 38.2% complete.

Conclusion: ChatGPT-4o proved to be potentially useful to support clinical practice in the musculoskeletal field. However, its efficacy in daily practice is contingent on its integration with human clinical reasoning and critical practitioner supervision. Keywords: Artificial Intelligence (AI), Deep Learning (DL), Large Language Model (LLM), Large Multimodal Model (LMM), Clinical Practice Guidelines (CPG), Osteoarthritis (OA), knee, hip.