Accuracy of a Chatbot in Answering Questions that Patients Should Ask Before Taking a New Medication.

BACKGROUND: The potential uses of artificial intelligence have extended into the fields of healthcare delivery and education. However, challenges are associated with introducing innovative technologies into healthcare, particularly with respect to information quality.

OBJECTIVE: To evaluate the accuracy of answers provided by a chatbot in response to questions that patients should ask before taking a new medication.

METHODS: Twelve questions obtained from the Agency for Healthcare Research and Quality were asked to a chatbot for the top 20 drugs. Two reviewers independently evaluated and rated each response on a 6-point scale for accuracy and a 3-point scale for completeness with a score of 2 considered adequate. Accuracy was determined using clinical expertise and a drug information database. After the independent reviews, answers were compared, and discrepancies were assigned a consensus score.

RESULTS: Out of 240 responses, 222 (92.5%) were assessed as completely accurate. Of the inaccurate responses, 10 (4.2%) were mostly accurate, 5 (2.1%) more accurate than inaccurate, 2 (0.8%) were equal parts accurate and inaccurate, and 1(0.4%) was more inaccurate than accurate. Of the 240 responses, 194 (80.8%) were comprehensively complete. There were 235 (97.9%) responses that scored 2 or higher. Five responses (2.1%) were considered incomplete.

CONCLUSION: Utilizing a chatbot to answer questions commonly asked by patients is mostly accurate but may include inaccurate information or lack valuable information for patients.