Microsoft says its new health AI beat doctors in accurate diagnoses by a mile

In a blog post published on Monday, the tech giant said its AI system, the Microsoft AI Diagnostic Orchestrator, diagnosed cases four times as accurately as a group of experienced physicians in a test.

The experiment involved 304 case studies sourced from the New England Journal of Medicine. Both the AI and physicians had to solve these cases step by step, just like they would in a real clinic: ordering tests, asking questions, and narrowing down possibilities.

The AI system was paired with large language models from tech companies like OpenAI, Meta, Anthropic, and Google. When coupled with OpenAI’s o3, the AI diagnostic system correctly solved 85.5% of the cases, Microsoft said.

By contrast, 21 practicing physicians from the US and UK — each with five to 20 years of experience — averaged 20% accuracy across the completed cases, the companyadded. In the study, the doctors did not have access to resources they might typically tap for diagnostics, including coworkers, books, and AI.

The AI system also solved cases “more cost-effectively” than its human counterparts, Microsoft said.

“Our findings also suggest that AI reduce unnecessary healthcare costs. US health spending is nearing 20% of US GDP, with up to 25% of that estimated to be wasted,” it added.

“We’re taking a big step towards medical superintelligence,” said Mustafa Suleyman, the CEO of Microsoft’s AI division, in a post on X.

He added that the cases used in the study are “some of the toughest and most diagnostically complex” a physician can face.

Microsoft did not respond to a request for comment from Business Insider.

Will AI replace doctors?

Microsoft said in the blog post that AI “represents a complement to doctors and other health professionals.”

“While this technology is advancing rapidly, their clinical roles are much broader than simply making a diagnosis. They need to navigate ambiguity and build trust with patients and their families in a way that AI isn’t set up to do,” Microsoft said.

“Clinical roles will, we believe, evolve with AI,” it added.

“AI will come in and provide medical IQ, and there won’t be a shortage,” he said on an episode of the “People by WTF” podcast published in April.

But doctors have toldBI that AI can’t and shouldn’t replace clinicians just yet.

Chatbots and AI tools can handle the first mile of care, but they must escalate to qualified professionals when needed, he told BI last month.