PO:38:283 | Accuracy of chatgpt 4 omni in providing clinical recommendations in musculoskeletal care: a valuable support for clinicians and patients?
Abstract
Background. Generative artificial intelligence models, such as ChatGPT-4 omni, is gaining popularity in healthcare and could provide valuable support for the clinical management of widespread diseases in the population, such as osteoarthritis. However, its accuracy in providing up-to-date clinical recommendations remains underexplored, particularly in the context of osteoarthritis (OA), one of the most common conditions with a high socio-economic impact, where it is critical that recommendations are up-to-date, clear and consistent with scientific evidence. Objective: to evaluate the accuracy and consistency of the clinical recommendations generated by ChatGPT-4 omni in relation to the most recent international guidelines for osteoarthritis (NICE, EULAR, OARSI) and to compare them with the responses given by Italian physiotherapists.
Materials and methods: cross-sectional study based on 24 clinical questions derived from OARSI, NICE and EULAR guidelines. The questions were presented to ChatGPT-4o in the form of ‘classic’ and ‘medical’ prompts, in Italian and English, for a total of nine administrations each. A complex clinical case was also evaluated. Accuracy, internal reliability (SD, CV%) and concordance (Cohen's K) were calculated.
Results: ChatGPT-4o showed good internal consistency (CV < 25%) and moderate to high agreement between Italian and English (K up to 0.902). Accuracy ranged from 66.7% to 79.2%, higher in classical and English prompts. Agreement with Italian physiotherapist was low (K = 0.106). In the clinical case, the mean accuracy was 81.4%, but answers were only 38.2% complete.
Conclusion: ChatGPT-4o proved to be potentially useful to support clinical practice in the musculoskeletal field. However, its efficacy in daily practice is contingent on its integration with human clinical reasoning and critical practitioner supervision. Keywords: Artificial Intelligence (AI), Deep Learning (DL), Large Language Model (LLM), Large Multimodal Model (LMM), Clinical Practice Guidelines (CPG), Osteoarthritis (OA), knee, hip.
Metrics
Citations and online attention from Dimensions, Altmetric and PlumX.
License
Copyright (c) 2025 The Author(s)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
PAGEPress has chosen to apply the Creative Commons Attribution NonCommercial 4.0 International License (CC BY-NC 4.0) to all manuscripts to be published.