AI Summary & Key Takeaways
AI InsightOverall accuracy, class-specific F1 scores, and Cohen’s kappa coefficients were calculated.
📌 3 Key Takeaways
Abstract
Aims: To evaluate the performance of ChatGPT, Gemini, and Grok in selecting among camouflage treatment, functional treatment, and orthognathic surgery for patients with skeletal class II malocclusion and to compare their classifications with expert orthodontist consensus. Methods: This retrospective comparative decision-accuracy study included a balanced sample of 45 patients, comprising 15 cases in each treatment category based on the unanimous agreement of three specialist orthodontists. Standardized scenarios containing anonymized demographic, clinical, and cephalometric data were submitted independently to the three large language models. Overall accuracy, class-specific F1 scores, and Cohen’s kappa coefficients were calculated. Differences in accuracy were evaluated using Cochran’s Q and exact McNemar tests with Bonferroni correction. The association between age and prediction accuracy was assessed using the Mann-Whitney U test. Results: Overall accuracy was 77.8% for Grok, 75.6% for ChatGPT, and 68.9% for Gemini. Grok and ChatGPT showed substantial agreement with expert consensus, with kappa values of 0.667 and 0.633, respectively, whereas Gemini showed moderate agreement (κ=0.533). All models achieved their highest F1 scores for functional treatment (0.929-1.000). Gemini frequently classified camouflage cases as requiring orthognathic surgery, whereas Grok tended to classify surgical cases as suitable for camouflage. Overall accuracy did not differ significantly among the models (Q=1.300; p=0.522). Incorrect ChatGPT predictions were associated with a higher age rank than correct predictions (p=0.006). Conclusion: The models demonstrated moderate to substantial agreement with expert orthodontists and performed particularly well in functional treatment cases. However, errors in distinguishing camouflage treatment from orthognathic surgery indicate that LLM-based classification should be interpreted cautiously and should not replace comprehensive clinical assessment.