Add like
Add dislike
Add to saved papers

Evaluation and Comparison of Ophthalmic Scientific Abstracts and References by Current Artificial Intelligence Chatbots.

JAMA Ophthalmology 2023 July 28
IMPORTANCE: Language-learning model-based artificial intelligence (AI) chatbots are growing in popularity and have significant implications for both patient education and academia. Drawbacks of using AI chatbots in generating scientific abstracts and reference lists, including inaccurate content coming from hallucinations (ie, AI-generated output that deviates from its training data), have not been fully explored.

OBJECTIVE: To evaluate and compare the quality of ophthalmic scientific abstracts and references generated by earlier and updated versions of a popular AI chatbot.

DESIGN, SETTING, AND PARTICIPANTS: This cross-sectional comparative study used 2 versions of an AI chatbot to generate scientific abstracts and 10 references for clinical research questions across 7 ophthalmology subspecialties. The abstracts were graded by 2 authors using modified DISCERN criteria and performance evaluation scores.

MAIN OUTCOME AND MEASURES: Scores for the chatbot-generated abstracts were compared using the t test. Abstracts were also evaluated by 2 AI output detectors. A hallucination rate for unverifiable references generated by the earlier and updated versions of the chatbot was calculated and compared.

RESULTS: The mean modified AI-DISCERN scores for the chatbot-generated abstracts were 35.9 and 38.1 (maximum of 50) for the earlier and updated versions, respectively (P = .30). Using the 2 AI output detectors, the mean fake scores (with a score of 100% meaning generated by AI) for the earlier and updated chatbot-generated abstracts were 65.4% and 10.8%, respectively (P = .01), for one detector and were 69.5% and 42.7% (P = .17) for the second detector. The mean hallucination rates for nonverifiable references generated by the earlier and updated versions were 33% and 29% (P = .74).

CONCLUSIONS AND RELEVANCE: Both versions of the chatbot generated average-quality abstracts. There was a high hallucination rate of generating fake references, and caution should be used when using these AI resources for health education or academic purposes.

Full text links

We have located links that may give you full text access.
Can't access the paper?
Try logging in through your university/institutional subscription. For a smoother one-click institutional access experience, please use our mobile app.

For the best experience, use the Read mobile app

Group 7SearchHeart failure treatmentPapersTopicsCollectionsEffects of Sodium-Glucose Cotransporter 2 Inhibitors for the Treatment of Patients With Heart Failure Importance: Only 1 class of glucose-lowering agents-sodium-glucose cotransporter 2 (SGLT2) inhibitors-has been reported to decrease the risk of cardiovascular events primarily by reducingSeptember 1, 2017: JAMA CardiologyAssociations of albuminuria in patients with chronic heart failure: findings in the ALiskiren Observation of heart Failure Treatment study.CONCLUSIONS: Increased UACR is common in patients with heart failure, including non-diabetics. Urinary albumin creatininineJul, 2011: European Journal of Heart FailureRandomized Controlled TrialEffects of Liraglutide on Clinical Stability Among Patients With Advanced Heart Failure and Reduced Ejection Fraction: A Randomized Clinical Trial.Review

Get seemless 1-tap access through your institution/university

For the best experience, use the Read mobile app

Read by QxMD is copyright © 2021 QxMD Software Inc. All rights reserved. By using this service, you agree to our terms of use and privacy policy.

You can now claim free CME credits for this literature searchClaim now

Get seemless 1-tap access through your institution/university

For the best experience, use the Read mobile app