Add like
Add dislike
Add to saved papers

Evaluating ChatGPT's Utility in Medicine Guidelines Through Web Search Analysis.

Permanente Journal 2024 April 27
INTRODUCTION: With the rise of machine learning applications in health care, shifts in medical fields that rely on precise prognostic models and pattern detection tools are anticipated in the near future. Chat Generative Pretrained Transformer (ChatGPT) is a recent machine learning innovation known for producing text that mimics human conversation. To gauge ChatGPT's capability in addressing patient inquiries, the authors set out to juxtapose it with Google Search, America's predominant search engine. Their comparison focused on: 1) the top questions related to clinical practice guidelines from the American Academy of Family Physicians by category and subject; 2) responses to these prevalent questions; and 3) the top questions that elicited a numerical reply.

METHODS: Utilizing a freshly installed Google Chrome browser (version 109.0.5414.119), the authors conducted a Google web search ( on March 4, 2023, ensuring minimal influence from personalized search algorithms. Search phrases were derived from the clinical guidelines of the American Academy of Family Physicians. The authors prompted ChatGPT with: "Search Google using the term '(refer to search terms)' and document the top four questions linked to the term." The same 25 search terms were employed. The authors cataloged the primary 4 questions and their answers for each term, resulting in 100 questions and answers.

RESULTS: Of the 100 questions, 42% (42 questions) were consistent across all search terms. ChatGPT predominantly sourced from academic (38% vs 15%, p = 0.0002) and government (50% vs 39%, p = 0.12) domains, whereas Google web searches leaned toward commercial sources (32% vs 11%, p = 0.0002). Thirty-nine percent (39 questions) of the questions yielded divergent answers between the 2 platforms. Notably, 16 of the 39 distinct answers from ChatGPT lacked a numerical reply, instead advising a consultation with a medical professional for health guidance.

CONCLUSION: Google Search and ChatGPT present varied questions and answers for both broad and specific queries. Both patients and doctors should exercise prudence when considering ChatGPT as a digital health adviser. It's essential for medical professionals to assist patients in accurately communicating their online discoveries and ensuing inquiries for a comprehensive discussion.

Full text links

We have located links that may give you full text access.
Can't access the paper?
Try logging in through your university/institutional subscription. For a smoother one-click institutional access experience, please use our mobile app.

Related Resources

For the best experience, use the Read mobile app

Mobile app image

Get seemless 1-tap access through your institution/university

For the best experience, use the Read mobile app

All material on this website is protected by copyright, Copyright © 1994-2024 by WebMD LLC.
This website also contains material copyrighted by 3rd parties.

By using this service, you agree to our terms of use and privacy policy.

Your Privacy Choices Toggle icon

You can now claim free CME credits for this literature searchClaim now

Get seemless 1-tap access through your institution/university

For the best experience, use the Read mobile app