AI Summary & Key Takeaways
AI InsightWhile previous studies have primarily focused on overall accuracy, the structural characteristics and distribution of errors remain insufficiently understood.
📌 3 Key Takeaways
Abstract
Aims: Large language models (LLMs) are increasingly used in scientific writing; however, concerns remain regarding the reliability of the bibliographic references they generate. While previous studies have primarily focused on overall accuracy, the structural characteristics and distribution of errors remain insufficiently understood. This study aimed to evaluate the patterns, co-occurrence, clustering, and predictors of bibliographic inaccuracies in LLM-generated references in endodontics. Methods: A comparative experimental study was conducted using 2 LLMs, ChatGPT and Claude. Ten predefined endodontic queries were combined with 3 prompt types, and each model generated 5 references per query-prompt combination, resulting in 300 references. All references were verified using PubMed, Google Scholar, and CrossRef. Errors were categorized as digital object identifier (DOI) inaccuracies, title errors, journal mismatches, author inconsistencies, and year errors. Error burden, co-occurrence patterns, DOI error clustering, and predictors of DOI error occurrence were analyzed using Chi-square tests, multivariable logistic regression, and a generalized linear mixed-effects model. Results: DOI errors were the most frequent inaccuracy, followed by title and journal errors. ChatGPT generated a higher proportion of multiple-error references than Claude (48.7% vs 6.0%, p<.001), whereas fully accurate references were more common with Claude (79.3% vs 26.0%). DOI errors were associated with title and journal inaccuracies (p<.001), and DOI error clustering was more common in ChatGPT than Claude (47.3% vs 6.0%, p<.001). ChatGPT was independently associated with higher odds of DOI errors compared with Claude (OR=16.60, 95% CI: 8.70-31.66, p<.001), whereas prompt design was not significant. Conclusion: LLM-generated references showed structured, clustered, and model-dependent error patterns rather than random inaccuracies. Model type appeared to be a stronger determinant of DOI error occurrence than prompt design. LLM-generated citations should not be used without external verification through reliable bibliographic databases.