AI in Healthcare: Why Doctors Override 99% of It

First of all, it it has to be set up in a way that the human is in the loop. Um because otherwise we did a consumer survey and uh study as well and we did a study of clinicians as well. What consumers are telling us is not that they'll reject the AI. They're saying they need the explanability and they need a human in the loop. then they're willing to accept the benefits of an AIdriven system and from an um from a clinician perspective for this to be compliant with you know just not just regulations with privacy etc which has to be very highly secure safety data sharing is utmost uh concern and all of that you know the infrastructure exists to do that but then more than that even when the human is in the loop for the clinicians to trust the recommendation from and

the AI there are three things that matter. This is what the study showed, right? one of our um you know PhD data scientists along with some um health system collaborators did this study and the study was simply looking at clinicians using AI and what drives overrides right so you know low confidence predictions get overridden 99% of the time and when you show the explainability so transparently how the AI came to that conclusion in a very simple way and show why the result is to be a high confidence one the overridden went down to overrides went down to 1.7%. 99% to 1.7%. So, so completely different picture on overrides, right? Yeah. It's published study. And so what it was saying is that doctors and clinicians weren't rejecting AI, they were rejecting uncertainty. And so the way to solve this problem is first put clinicians always in

the loop, human in the loop. So AI is not autonomously deciding, it's recommending for the clinicians and making the administrative burden less for the clinicians. Then you have to work on um really explainability which is the I started with that right in anything from research to anywhere if it's totally a black box nobody's going to trust it no matter how good it is not going to be trust so better systems that shows the rationale and the reasoning for the decisioning and what's the evidence base against that decisioning and rendering that in a very simple human experience so they don't have to go digging and doing their own research. search to find the explanability very prominently explainability and the third is how do you get the confidence factor of the recommendation very high by showing that multiple models converging to the same recommendation and that it's

a groundability that it's repeatedly asking with different prompts the same question leads to the same answers when you show those three things the clinician's override go down the compliance is very high and and you know they can do their job so I think that's the key in making making Sure. We're creating very trusted systems that can be effective in, you know, doing what it does.