The AI Dilemma: Can ChatGPT Outperform Radiology Residents?

Over the course of the past few decades, the field of artificial intelligence (AI) has experienced exponential growth in various sectors, including healthcare. AI has proven beneficial in image interpretation and the identification of disease predictors. Particularly promising are the advancements in large language models, and specialized neural networks that continuously learn from vast amounts of data. This opens up possibilities for applications beyond medical diagnoses, such as generating management plans.

The Promise of Large Language Models in Healthcare

Many experts predict that these large language models will eventually surpass human performance across diverse domains, including healthcare. One notable example is ChatGPT, an AI chatbot based on OpenAI's generative pre-trained transformer (GPT) model, first introduced in 2018. ChatGPT Version 4, released earlier this year, made significant headlines and generated concerns within and beyond the medical field. It became evident that ChatGPT could successfully pass a non-image-related radiology board-style examination without specific radiology training.

It is highly likely that these AI models will further enhance their capabilities as reinforcement learning is incorporated, specifically using radiology-specific imaging and language data.

Bloom's Taxonomy: Understanding Knowledge and Cognitive Processes

Now, let's delve into Bloom's taxonomy and its subsequent updates, which serve as a framework in educational and cognitive psychology for classifying types of knowledge and the cognitive processes associated with that knowledge. This taxonomy can be summarized as follows:

Types of Knowledge

• Metacognitive knowledge: Strategic, contextual, self-awareness, and conditional knowledge.
• Procedural knowledge: Subject-specific skills, algorithms, techniques, methods, and criteria.
• Conceptual knowledge: Classification and categories, principles, theories, and models.
• Factual knowledge: Terminology, symbols, and details.

Cognitive Processes for Knowledge Processing

• Creating: Generating, planning, producing, and designing.
• Evaluating: Critiquing, defending, and checking.
• Analyzing: Differentiating, organizing, and attributing.
• Applying: Executing and implementing.
• Understanding: Interpreting, classifying, summarizing, explaining, and comparing.
• Remembering: Recognizing and recalling.

This taxonomy provides a comprehensive framework for understanding and categorizing different types of knowledge and the cognitive processes involved in processing that knowledge.

We discover an intriguing pattern as we delve deeper into Bloom's taxonomy. The positioning of a question on the taxonomy list determines the level of knowledge and cognitive processing required to answer it. Higher-order questions demand more abstract thinking and critical analysis, while lower-order questions involve more concrete thinking and rote memorization. In essence, large language models, like ChatGPT, excel at tackling low-hanging fruit, the questions on the lower end of the taxonomy spectrum. However, when it comes to higher-order taxonomy material, ChatGPT's performance falls short, potentially indicating its inability to achieve proficiency in that area.

Over the past few decades, scientific advancements and the exponential growth of medical content have intensified the learning load for residents within the same four-year timeframe. This intensified pressure has led trainees to allocate more time and effort to lower-order learning, which is more time-efficient, at the potential expense of higher-order learning. Consequently, the application, analysis, evaluation, and creation of knowledge, particularly higher-order knowledge, may suffer. Humans typically tackle higher taxonomy content by initially mastering the low-hanging fruit, establishing a sequential learning process that proves challenging to replicate in artificial intelligence.

Simultaneously, there has been a shift in the evaluation of residents, exemplified by the American Board of Radiology (ABR) Core and Certifying examinations. In 2015, the ABR transitioned from an oral certifying examination to a fully computer-based multiple-choice format, reducing the reliance on interactive conferences where trainees review cases, develop differential diagnoses, and communicate diagnoses and management plans. Instead, trainees focus on memorizing broad knowledge for board preparation using multiple-choice question banks.

ChatGPT's Performance in Radiology Board Examinations

"The use of large language models like ChatGPT is exploding and only going to increase," said lead author Rajesh Bhayana, M.D., FRCPC, an abdominal radiologist and technology lead at University Medical Imaging Toronto, Toronto General Hospital in Toronto, Canada. "Our research provides insight into ChatGPT's performance in a radiology context, highlighting the incredible potential of large language models, along with the current limitations that make it unreliable."

Dr. Bhayana also noted that ChatGPT was recently named the fastest-growing consumer application in history, and similar chatbots are being incorporated into popular search engines like Google and Bing, which physicians and patients use to search for medical information. To assess its performance on radiology board exam questions and explore its strengths and limitations, Dr. Bhayana and colleagues conducted a series of tests using ChatGPT based on GPT-3.5, currently the most commonly used version. They used 150 multiple-choice questions designed to match the style, content, and difficulty of the Canadian Royal College and American Board of Radiology exams. The questions did not include images and were grouped by question type to gain insight into performance in both lower-order (knowledge recall, basic understanding) and higher-order (apply, analyze, synthesize) thinking.

The higher-order thinking questions were further subclassified by type, including the description of imaging findings, clinical management, calculation and classification, and disease associations. The researchers evaluated the overall performance of ChatGPT as well as its performance by question type and topic. They also assessed the confidence of language in the model's responses.

The findings revealed that ChatGPT based on GPT-3.5 answered 69% of the questions correctly (104 out of 150), which is near the passing grade of 70% used by the Royal College in Canada. The model performed relatively well on questions requiring lower-order thinking, correctly answering 84% of them (51 out of 61). However, it struggled with questions involving higher-order thinking, with an accuracy of 60% (53 out of 89). Specifically, it faced difficulties with higher-order questions related to the description of imaging findings (61% accuracy, 28 out of 46), calculation and classification (25% accuracy, 2 out of 8), and application of concepts (30% accuracy, 3 out of 10). These challenges in higher-order thinking were expected, considering the model's lack of radiology-specific pre-training.

In March 2023, GPT-4 was released in a limited form to paid users, claiming to have improved advanced reasoning capabilities compared to GPT-3.5. In a follow-up study, GPT-4 correctly answered 81% of the same questions (121 out of 150), outperforming GPT-3.5 and surpassing the passing threshold of 70%. GPT-4 demonstrated much better performance on higher-order thinking questions (81%), particularly in the areas of describing imaging findings (85% accuracy) and applying concepts (90% accuracy).

The findings suggest that GPT-4's claimed improved advanced reasoning capabilities lead to enhanced performance in a radiology context. They also indicate an improved contextual understanding of radiology-specific terminology, including imaging descriptions, which is crucial for future downstream applications. "Our study demonstrates an impressive improvement in ChatGPT's performance in radiology over a short time period, highlighting the growth potential of large language models in this context," said Dr. Bhayana. However, GPT-4 did not show improvement on lower-order thinking questions compared to GPT-3.5 (80% accuracy vs. 84%). Additionally, GPT-4 incorrectly answered 12 questions that GPT-3.5 had answered correctly, raising concerns about

The performance of ChatGPT in examinations raises intriguing considerations regarding teaching to the test. Despite its lack of specific training, ChatGPT performs reasonably well on non-image-based radiology board-style questions and has demonstrated the ability to pass law school and Master of Business Administration (MBA) course examinations, as well as perform close to the passing threshold in the U.S. Medical Licensing Examination. Given ChatGPT's performance, important questions arise. Is ChatGPT truly prepared for clinical practice if it can successfully pass a board examination? What is the value of having residents undergo an examination that ChatGPT can conquer or even surpass in performance? We must exercise caution, as AI has the potential to outperform residents not only in examinations but also in other aspects if we solely focus on teaching to the test.

Residents possess a wealth of knowledge that surpasses that of AI, both currently and in the future, although this superiority may not always be evident in multiple-choice examinations. An intriguing finding from current studies referenced below is that ChatGPT sometimes provided incorrect information in its answer explanations, even when it supplied the correct answer. This underscores the risk of using ChatGPT as a study aid or clinical tool, as it could paradoxically hinder resident education when AI is "correct" for the wrong reasons. We should be cautious about relying solely on AI and neglecting the importance of human intelligence.

Regarding the study's findings, ChatGPT performed better on lower-order thinking questions than on higher-order ones. It struggled particularly with questions involving the description of imaging findings, calculation, and classification, application of concepts, and physics. Radiology educators must refocus their teaching and evaluation methods on higher-order cognition and knowledge that AI cannot easily replicate. Emphasizing patient interactions and communication is essential to ensure radiologists continue to add value as the field rapidly evolves. It is alarming to note that the AI chatbot presented results with high confidence and human-like language, even when the answers may be incorrect or misleading. Novice residents who lack the knowledge to recognize ChatGPT's fallibility could inadvertently compromise patient care. Radiologists must not overlook the importance of timely communication of imaging results with providers and patients, even as AI becomes more integrated into image interpretation.

 
Teaching to the Test: Implications for AI and Radiology Education

To address these challenges, evaluations should prioritize higher-order knowledge and engage higher-order cognition. Incorporating various formats of examinations, such as oral examinations and clinical simulations, will allow residents to apply their knowledge, make judgments, solve problems, critique cases, weigh pros and cons, and effectively communicate their thought processes. By doing so, we can train future radiologists who add value to patient care in ways surpassing the's capabilities of ChatGPT alone. The recent announcement by the ABR regarding plans to reintroduce oral board examinations is a positive step in this direction.

While it is undeniable that AI models will eventually outperform residents on multiple-choice board-style examinations as radiology-specific data are incorporated into supervised and reinforcement learning, we must adapt radiology education and evaluation accordingly. We must ensure that AI algorithms assist in educating future radiologists about patient care without sacrificing critical thinking and communication skills. Ultimately, AI holds the promise of advancing the field of radiology, but we must remember that not everything necessary can be measured through tests, especially multiple-choice exams, nor can it be replicated by AI.




Rayscape was designed to help radiologists be more, to be better. Our suite of products dedicated to the health of the lungs is seamlessly integrated into the doctor’s and hospital’s workflow, helping them to analyze chest X-rays and lung CTs faster and more efficiently. Learn more HERE

References:

• Muller, J. (2018). The Tyranny of Metrics. Princeton University Press, 29–31.
• Choi, JH., Hickman, KE., Monahan, A., Schwarcz, D. (2023). ChatGPT Goes to Law School. Minnesota Legal Studies Research Paper No. 23–03. [Link](https://ssrn.com/abstract=4335905) .
• Terwiesch, C. (2023). Would Chat GPT Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course. Mack Institute for Innovation Management. [Link](https://mackinstitute.wharton.upenn.edu/2023/would-chat-gpt3-get-a-wharton-mba-new-white-paper-by-christian-terwiesch/) .
• Kung, TH., Cheatham, M., Medenilla, A., et al. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health, 2(2), e0000198.
• Gilson, A., Safranek, CW., Huang, T., et al. (2023). How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ, 9, e45312.
• Loh, E. (2018). Medicine and the rise of the robots: a qualitative review of recent advances of artificial intelligence in health. BMJ Leader, 2, 59–63. [Link](https://doi.org/10.1136/leader-2018-000071)
• Shen, Y., Heacock, L., Elias, J., et al. (2023). ChatGPT and Other Large Language Models Are Double-edged Swords. Radiology, 307(2), e230163.
• Bhayana, R., Krishna, S., Bleakney, RR. (2023). Performance of ChatGPT on a radiology board–style examination: insights into current strengths and limitations. Radiology, 307(5), e230582.
• Sawin, EI. (1957). Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook 1. Committee of College and University Examiners, Benjamin S. Bloom. Elem Sch J, 57(6), 343–344.
• Anderson, LW., Krathwohl, DR. (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Pearson, 28–37.