Doctors are like detectives. They scrutinize symptoms, piece together medical histories, and narrow down possible diagnoses. Simple cases may be solved on the spot. But trickier ones require tests to gradually zero in on the culprit.
The stakes are high. A misdiagnosis can delay treatment, allowing diseases to fester and cause lasting harm or prolonged suffering. Costs also stack up. Even with health insurance, multiple tests, some unnecessary in hindsight, can leave patients with staggering bills.
Medical debt is already a major burden in the US. According to one estimate, more than one-third of households carried some type of it in 2024. Some patients even opted out of subsequent care because of costs, which are often opaque.
A faster, more efficient path to the right diagnosis could vastly improve medical care. To that end, AI could be a useful tool. This month, in a paper published in Science, Microsoft researchers described MAI-DxO, a sophisticated system that acts like a crack medical team.
When challenged with extraordinarily complex cases, MAI-DxO was more than twice as accurate as physicians armed with textbooks and the internet—and nearly four times as accurate as those relying solely on their own knowledge. The system also surpassed doctors on typical emergency room cases, achieving 80 percent accuracy versus 47 percent for physicians.
“I’ve been working on principles and applications of AI in medicine since the 1980s,” study author Eric Horvitz told Science. “My hope is that these advances will help bring timely and effective care to more people, and that’s what makes this progress so exciting to me.”
The ER Pitt
AI already outperforms doctors at spotting breast cancer on mammograms and detecting skin cancer in curated datasets. Systems can also detect the telltale signs of cancer spread and heart disease. But while modern models routinely ace standard medical licensing exams and tackle hypothetical clinical scenarios, these tests gauge a physician’s knowledge, not the messy realities of diagnosis. All clues are given up front, and the AI simply picks an answer.
In practice, diagnostics is a winding journey. Doctors continuously gather symptoms, order tests, and assess whether new results support their initial hunch or torpedo it, requiring them to start anew.
Decades ago, physicians proposed using benchmarks to judge how AI doctors navigate real-world cases. One of these is the New England Journal of Medicine (NEJM) clinicopathological series, which is often used to challenge novice doctors with complex medical mysteries. And earlier this year, a team at Harvard Medical School and colleagues developed a benchmark to compare the performance of AI to physicians. Pitting OpenAI’s o1-preview reasoning model against two expert physicians on emergency room cases, the AI performed better across clinical tasks, from triage and chart reviews to deciding whether to admit or discharge.
High Standards
The Microsoft team took benchmarking a step further. In the new study, they selected 304 complex cases from NEJM and a similar number of typical emergency room encounters from a database. Together, “they span a broad spectrum of clinical presentations and documentation styles, with final diagnoses ranging from common conditions to rare disorders,” wrote the authors.
The team transformed each case into a standardized, step-by-step interactive challenge for both AI and physicians. As you might expect in a clinical setting, the cases started with limited information—say, a 30-year-old man presenting with chest pain. Participants then had free rein to ask about medical history, order tests, and update hunches before committing to a diagnosis.
The benchmark measured two main outcomes, said study author Harsha Nori. “Did you get the right diagnosis? And how expensive was the path you took to get there?”
The team then built MAI-DxO. Powered by GPT-5, the AI acts like a physician panel. One specialist proposes possible diagnoses. Another suggests questions and tests and iteratively challenges the emerging diagnosis. The last considers costs. Meanwhile, an “orchestrator”—think head physician—coordinates the debate, weighs new clues from test results, and decides whether another test, and which one, is worth the cost.
Pitted against 40 doctors, each with roughly a dozen years of clinical experience, MAI-DxO excelled. On complex NEJM cases, it reached an accurate diagnosis 83 percent of the time. In comparison, physicians were 22 percent accurate when relying on their own knowledge. Giving them access to the internet and other resources only boosted their performance by a smidge. For typical emergency room cases, the AI was nearly twice as accurate as physicians.
MAI-DxO was proficient even under tight budgets. When diagnosing NEJM cases, a cost-conscious configuration maintained nearly 80 percent accuracy while spending an estimated $1,300 dollars per case, based on 2023 hospital pricing data from a national healthcare system. Doctors, in contrast, racked up a bill of over $4,700.
Although GPT-5 supplies the brainpower, MAI-DxO performed better than when the team used the model alone. In a tricky case of scurvy, caused by low vitamin C levels, the system connected the patient’s dietary history to bleeding gums and tooth loss. After ordering a targeted vitamin C blood test, it reached the correct diagnosis after spending just over $1,100. The standalone version of GPT-5, meanwhile, asked for an extensive neurological workup, resulting in a bill that was nearly 7.6 times more expensive.
MAI-DxO also outmatched a range of general language models, including Anthropic’s Claude, Google’s Gemini, and xAI’s Grok. “If you run it in this configuration, the AI systems get to the right diagnosis far more often and at far less cost than out-of-the-box runs,” said Nori. And the system isn’t tied to GPT-5. Swapping in the newer GPT-5.5 further boosted accuracy.
These findings, while promising, don’t mean we’ll be seeing unsupervised AI diagnosticians in hospitals anytime soon. The experiments took place in controlled, text-based simulations. Real-world clinicians navigate inaccurate and incomplete patient histories and analyze medical images and other clinical data—often under time pressure. Chronic or evolving diseases, such as cancer, could also stump the system.
Then there’s the human factor. Patients have the final say over whether to undergo a test. A diagnostic plan may look perfect on paper, but if someone refuses a procedure, the AI must be able to pivot. “Critically, we have to consider each patient’s understandings and their own goals and preferences,” said Horvitz.
The team has partnered with Beth Israel Deaconess Medical Center in Boston to evaluate MAI-DxO in different hospital settings. One of their goals is to figure out how best to fit the system into everyday clinical practice. Another is to see how patients directly interact with it and how its performance stacks up against assessments from doctors.
“Working alongside clinical teams will show where the technology succeeds, where it falls short and how it can fit into busy workflows,” the team said in a blog post.
If all goes well, unexpected emergency room visits might come with a little less sticker shock.
