KLAS Research recently released a report on the role of Suki’s AI-powered clinical intelligence platform in delivering measurable return on investment across three health systems – FMOL Health, McLeod Health, and Rush University System for Health. Healthcare Innovation recently sat down with Bryon Frost, M.D., chief medical information officer at Florence, S.C.-based McLeod Health, to talk about his experience choosing and implementing an ambient AI solution.
The KLAS report noted that “all three organizations experienced reduced documentation burden, greater time savings, and improved E/M coding, which have led to improved efficiency and clear financial gain. Additional benefits include improved provider satisfaction, enhanced patient care, and improved patient satisfaction.”
Healthcare Innovation: When McLeod started considering an ambient scribe solution for documentation, did you look at several different solutions and what did you like about Suki?
Frost: I spent probably a year looking at the various vendors, and one of my biggest concerns was group think. The last thing I wanted to do was purchase Abridge just because everyone else was doing it. So I set up a really rigorous experiment where I narrowed down the field from about 20 down to four vendors.
I invited these four vendors to McLeod Health for a real event. I wrote out 15 very detailed patient scripts — five each for three types of doctors — a primary care doctor, a cardiologist and a surgeon, and I had actors come in and play patients. These scripts were fantastic. They challenged the AI. I had patient interruptions, and I had family members contradicting the patient. One script was all about the surgeon. He was incredibly rude and dismissive of the patient — just to challenge the artificial intelligence and how well it would generate a note from that.
We took all these notes, blinded it to the vendor, and ran them through three groups of people to grade the notes. We had physicians, revenue cycle people, and non-clinical patients who would eventually see their notes in our patient portal.
In phase two of the experiment, we brought the two surviving vendors to present their solution. How would it be embedded within Epic, within the workflow? And finally, I was going to do a bake-off between the two vendors. But 90% of the physicians chose Suki over the competitor, and my CEO said that since 90% of docs preferred one, let’s go with that.
HCI: You have seven hospitals in your system, as well as outpatient clinics and primary care clinics, right? How did you roll this out?
Frost: I picked 30 docs who I thought would be good users of it. This proved another fallacy — that you cannot predict who will be a champion of ambient. We started with a small pilot, and we made a little bit of a mistake in the pilot study. The initial concern I had was financial. How are we going to pay for this thing? We did not go into this experiment with any financial expectations. The problem that we’re trying to solve is cognitive burnout for physicians. If we lose money on this, so be it. That’s what I said publicly. In my head, I was thinking that I don’t want to have to justify to the CFO in three months why we chose the solution.
So I made the decision, along with our organizational leaders, that we’re only going to give the product to people who are above the 75th percentile of efficiency. I feared paying $250 a month for a license that nobody used. But that was a dumb decision, because the doc who really needs it is not at the 75th percentile; it’s the guy who’s at the 30th. So through negotiations with the vendor, we moved to utilization-based pricing. That was the game-changer. We could not unlock return on investment without that decision. We essentially pay a very small fee per encounter. Now I don’t have to be in the business of license swapping. If you only use it 10 times a month, I don’t have to worry about justifying the situation. And we get white glove treatment from Suki because if we don’t scale the product, they don’t get paid, so they’re very invested, whereas with the average subscription-based license, you don’t get that kind of service.
HCI: So you started with those 30 docs. How long did it take before the next step to expand usage more broadly?
Frost: Oh, we had so much demand from other people who wanted in. Probably after three months, we just acquiesced and gave it to a whole bunch of other folks — all ambulatory and the emergency department. We are rolling it out on the inpatient side now.
HCI: I’ve talked to a few people about the use of these tools in the emergency department, and they have said it is more challenging in that environment.
Frost: It was challenging when we started. It it was so challenging that I turned off the medical decision-making section, because it was horrible. It actually added to note bloat. It was sound and fury signifying nothing. But then Suki did more work on it. Their machine learning engineers did more work at the specialty level. They did some really cool stuff, and now the output of the LLM is spot on.
HCI: I interviewed Suki CEO Punit Soni about the creation of a nursing consortium….
Frost: We are in that consortium. Epic is the rate-limiting factor on that consortium. There are some features that Epic has to release, and I think they have no motivation to have any vendor make gains in this space, so they haven’t been able to really make a difference in documenting and flow sheets the way we want to.
HCI: Can you talk about measuring the ROI impact of this deployment?
Frost: We had rigorous analytics teams around this project to make sure that we were accurate in our data. Two big things stuck out with KPIs — one was on the financial side. Initially we were a little over $1,000 per provider per month net. And now, after just re-running the numbers a month or so ago, we were at almost $2,600 per provider per month net, after subscription costs.
And then the patient satisfaction scores was the other one that was not anticipated. That wasn’t even one of my KPIs. The patient satisfaction team at McLeod came to me and said you have got to look at these numbers. We had like 6% increases in their patient satisfaction score. So that was probably the coolest thing we got out of this.
HCI: What about looking at the clinicians’ own reports of after-hours work or cognitive load? Were you looking at the Epic Signal data?
Frost: Yes, we looked at measuring pajama time. We also looked at hours worked on unscheduled days. The anecdotal results were awesome, but the metrics were a lot more difficult to measure. We’d hear someone say, ‘I’m getting 30 and 40 hours back in my life per week,’ and I would say I don’t know how you’re getting that, but I’m not going to argue with you getting more time back in your day. But it was a challenge to look at the Signal data to try to narrow down and say quantitatively we return this amount of time to you. That’s probably the hardest one for us to measure.
HCI: Do you think there’s something that separates the ambient AI tools that scale well across a health system from ones that might stall?
Frost: The friction involved in onboarding is definitely a problem with a lot of these systems. I want to see native, natural adoption. I want to see people wanting to adopt it because it’s fantastic. I think a lot of people report adoption rates that are over-inflated. They consider “adoption” when someone used it to to create one or two notes. In our pilot, we had a 74% adoption rate, and we didn’t consider you to have adopted it unless you were still using it two months later for at least 80% of your encounters. We were very rigorous on our definition of adoption.
HCI: Can you describe the AI governance you have set up there at McLeod?
Frost: We’re not a big academic center, so we’re nimble, and we can really do some practical things. One of the things I’m working on is a tiering system. We have a four-tiered system that rates the various AI projects. Tier one is something that only impacts you all the way to tier four, which is a true agentic workflow where a human is not in the loop. I never plan on approving a tier-four at McLeod Health, but we’ll have a few tier threes, which do impact clinical workflows. We have to figure out how we are going to monitor them.
HCI: Anything else you want to mention?
Frost: One of the biggest problems I’m dealing with right now is downtime. I have this theory that healthcare does not concentrate enough on the fragility of the system that gets amplified by artificial intelligence. I have a presentation I’m working on about how we mitigate that risk. If we ever did have a complete outage of our network, with cognitive offloading and the de-skilling that occurs with automation, it’s a significant risk that everybody’s not talking about enough.
