Medical AI Outperforms Doctors in Diagnosis and Prescribing, Raising Need for ‘Evaluation Science’

Studies have increasingly found that medical artificial intelligence (AI) can perform as well as or better than doctors, from diagnosis to prescribing and care management. But they also raise the need for “evaluation science” to objectively validate that performance.

The National Biotechnology Policy Research Center reported these findings in a report published on the 10th, citing a July 30 (local time) report by Nature News. The report discussed two papers on the performance of MIRA, an agentic AI that directly performs clinical tasks, and AMIE, an AI that develops treatment plans based on patients’ progress.

According to a study published in Nature in June, MIRA (Medical Intelligence for Reasoning and Action) is an AI that uses electronic health records to perform real clinical tasks, including ordering tests and imaging and prescribing medications. In a simulation involving more than 500 actual patient records, it outperformed doctors overall in diagnostic accuracy, procedure recommendations and adherence to guidelines.

MIRA recorded higher diagnostic accuracy than doctors for most conditions, including appendicitis, cholecystitis, diverticulitis, pulmonary embolism, pancreatic cancer, pancreatitis, pneumonia and urinary tract infections. Its prescribing process also showed no errors involving prescriptions for drugs with interaction risks or dose adjustments based on kidney function, indicating safety.

MIRA has 11 medical-specific functions, including requests for tests, imaging and prescriptions, as well as decisions about hospitalization. It goes beyond simple conversational responses to autonomously perform actions in electronic health record systems. This is a “scaffolding” approach that adapts general-purpose AI for medical use by having it consult clinical guidelines and drug information and adding functions for ordering tests and executing prescriptions.

AMIE (Articulate Medical Intelligence Explorer) is an AI that extends beyond diagnosis to treatment planning. In specialist evaluations, it made judgments that were as precise as or more precise than those of primary care physicians. It showed a large gap over doctors on “treatment precision,” a measure comparing the specificity of treatment and testing plans based on patients’ progress.

AMIE was designed to search a total of 627 clinical guidance documents, including guidelines from the U.K.’s NICE, and use them as evidence. It generally covered a broader range than doctors across six care-management areas: appropriateness of care plans, incorporation of patient preferences, monitoring and plan adjustments, reasoning about medication prescriptions, selection of clinical guidelines, and treatment precision.

However, both studies have fundamental limitations in their performance and evaluation methods. MIRA and AMIE are built by adding medical-specific functions to general-purpose large language models (LLMs), making it difficult to determine whether their performance comes from the AI’s own capabilities or from the added knowledge and tools.

In the AMIE study, performance gains from adding scaffolding to an older AI were also reproduced by a newer general-purpose AI alone, and adding the same scaffolding to that newer AI did not further improve performance. The MIRA study did not conduct a comparative experiment to assess the effect of scaffolding, making it difficult to compare its performance with that of newer AI systems.

The report therefore raised the question of whether specialized features such as scaffolding will continue to be needed in medicine, or whether general-purpose AI will surpass specialized designs as AI advances. This is based on the concept of the “bitter lesson”: that general methods for improving a model’s own performance ultimately work better than specialized features designed by people.

The report also pointed to limitations in the evaluation methods. Both studies measured performance by “how closely it matches doctors’ judgments,” meaning results could vary depending on which doctors were selected as evaluators. The report said this reflects a structural problem similar to the reproducibility crisis in science and medicine, in which repeating the same experiment in the past did not produce the same results.

The report proposed validating medical AI against clear criteria based on treatment outcomes for actual patients, rather than whether it agrees with evaluators selected in advance by researchers. It said rigorous investigations are needed into preplanned clinical trials and human-AI interactions in real-world clinical settings, and that some such studies have already begun.

The National Biotechnology Policy Research Center identified the need to continue refining rigorous, sustainable evidence and evaluation methods that can guide decisions about which medical AI tools to trust and when.