AI governance and monitoring platforms are a key new solution category for health system chief AI officers to consider. Healthcare Innovation recently spoke with Jon McManus, Northern Virginia-based Inova Health’s chief data and AI officer, about the health system’s needs in this area and its decision to deploy a solution from Toronto-based Signal 1. Joining the conversation was Tomi Poutanen, Signal 1’s CEO.
Healthcare Innovation: Jon, you came to Inova from a similar position at Sharp HealthCare in San Diego. Are the two health systems working on similar things in regard to AI governance?
McManus: One of the reasons I came to Inova is they were interested in maturing their approach to AI governance and the capabilities to make that set of services for both data and AI a beacon of excellence. I would say we were a bit more mature in California. It has been wonderful partnering with Matt Kull, who left his post as the chief information officer at Cleveland Clinic to come to Inova as well. Dr. Jones [Inova CEO J. Stephen Jones, M.D.] is forming a bit of a star-studded lineup at Inova.
HCI: Did Sharp either build something or have a partnership with a company like Signal 1 to do something similar?
McManus: We did not, and I don’t think anybody did. Establishing the mechanics and what would be the requirements of these programs has evolved over the past couple of years. One of the things that Sharp certainly is recognizing today — and what I think most health systems are coming up against — is you can have good processes and use Excel spreadsheets and have good methods for governance that work when you’re dealing with 30, 40, or 50 things. But when you’re dealing in AI governance with feature sets numbering in the multiple hundreds, you really have to think about scaling from a platform standpoint. And that’s where I think our partnership with Signal 1is important. We believe that they are a vehicle to help us scale.
HCI: Tomi, please tell us a little about your background and Signal 1’s founding.
Poutanen: I am a repeat AI company founder, having worked in both Silicon Valley and the banking industry before. Immediately before starting Signal 1, I was the chief AI officer of TD Bank. A lot of the practices that we bring into healthcare are ones that we have learned in other industries. Healthcare is a little bit behind other industries in its adoption of AI. Other industries think about AI adoption and scaling across an enterprise as a shared service, as an enterprise capability, and that means that AI governance, AI investments, etc., are arbitrated at the center and managed from the center, but then implemented at the edges.
A lot of health systems are hiring people like Jon to oversee their data and AI practices, and now they’re arming them with tools to manage AI at scale across a very complex enterprise. Historically, these AI solutions have been managed via e-mail, in-person committee meetings, Microsoft Excel, and that just doesn’t scale. It works at the early phases when you’re experimenting with AI, but it no longer works at enterprise scale, with hundreds of AI applications running through an enterprise. And the solution that we provide offers the tooling for the person overseeing the AI program, that individual’s team, and also the broader implementers and the champions throughout the organization.
HCI: Is there a fair amount of customization that needs to happen at each health system? Or do the tools look much the same in each health system setting?
Poutanen: The tooling is the same. The overall tool we call the AI Management System, or AIMS for short. The product is the same for everyone. Where the customization comes in is in the evaluation of each and every AI application, right? You’re looking at measuring how it’s being used, the impact it is having and what the proper guardrails are. Those are very specific to a health system, so that’s where we lean in and help our partners put up the proper guardrails and evaluations in place.
HCI: Is Inova the first major U.S. health system that you guys are partnering with? Or do you have other ones that you’ve already worked with?
Poutanen: We have one other — a very large East Coast academic medical center that we’re working with as our second U.S. client.
HCI: Jon, from your perspective, what are some of the challenges that this platform can help with, as far as monitoring algorithms or generative AI solution performance? What kind of metrics do you need to see and how does Signal 1’s platform help with that?
McManus: I think Signal 1 comes in with the mature core competency of monitoring functions like predictive AI. That could be traditional data science predictive models. What do you monitor in those type of things? Positive and negative predictive value, Brier score, how often it is firing. There’s a variety of things to pay attention to: model drift and performance and success. What I think has been special about Signal 1 is seeing them take that same core competency and add the flexibility and the evolution to support generative AI. Now the unit of measure in many AI products is not about predictive AI. Within the structure of Signal 1 they’re giving us the support to make those design decisions for a feature so it’s tailored for that feature.
I can give you a very real example. With our partners at Epic, we, like many health systems across this country, implement a generative AI draft assistant for patient messages through their portal that go to our primary care physicians to help them respond to common and low-risk patient messages. When you think about the things you need to measure for that, we want to be able to know first off, how many messages is it drafting? How frequently are providers using it? We also want to know how often are they changing the words and by what degree. The Signal 1 team lets us introduce that component as part of the measurement. So instead of the place you normally find positive predictive value, we replace that with this metric that’s important for that particular feature. What we are looking for is a unified pane of glass for monitoring these advanced intelligence assets, whether they’re AI or traditional data science.
It’s also allowing us to think about the future of our informatics function. We have wonderful nursing- and provider-led informatics teams here at Inova, We want to empower those licensed physician informaticians with the ability to monitor these capabilities within their own field of practice. What better than a primary care physician being able to keep tabs on the performance of the Epic automated draft reply tool with this type of capability? So it’s really giving us a chance to centralize how we do monitoring at scale for this portfolio. I also want to highlight that is different than the inventory that we’re trying to manage for AI. Not every AI item needs monitoring at this scale, but we want to have a unified approach to the cohort that does.
HCI: I was interested when you mentioned that example of drafting the responses from clinical inboxes, because I was just listening to several CMIOs up in the Boston area talking about how the percentage of the drafts being used in their health systems so far was very low — like 5 to 10% — and they were weighing the ROI of that. They weren’t getting a lot of usage yet, and they have to think about what they’re going to do about that.
McManus: That’s the other nice thing about the AIMS concept that Tomi mentioned — it’s not just about the safety and the performance measures. There’s also the opportunity to standardize how we approach value.
So let me go right back to that same model. Most organizations that deployed at a large enough scale of primary care are probably running that Epic AI draft tool on about 60,000 messages a year. The organizations that tend to implement that well usually can get up to about 30% utilization for the primary care physicians. We typically see somewhere around 16 seconds of time savings when those messages are used. And there have been multiple papers published on this that you could correlate that to, so how would you measure value? Well, what’s 60,000 messages a year divided by 12, and what’s 30% of that? Multiply that by 16 seconds per message, convert that to hours, and what’s the average hourly rate of a primary care physician? You start to come up with a value, and then you correlate that with how much Epic charges for that model to run over the same time period. Then you can get a certain X return.
We are seeing a lot of consistency that there tends to be about a 4X return on cost related to this particular feature of multiple health systems. But the problem is that’s a soft number, because you don’t know where those 16 seconds of savings go. Do they go to productive time? Do they not? But I think it’s important to have the ability to communicate that feature by feature, and what we’re looking at at Inova is doing that with rigor and at scale from a platform. So when my leadership team asks me, what is the overall expense to the production-enabled AI portfolio, what is the overall return for that investment, I am able to offer that type of answer, and then I’m also able to say, here’s the safety scorecard and here’s the performance scorecard of that same portfolio. We were able to do this by hand before and with manual survey work. Signal 1 gives us an opportunity to really be more quantitative and platform-oriented in that approach.
HCI: I read that Inova was the first health system to commit to the Joint Commission’s responsible use of health date data criteria. Are there elements of using this platform that align with the things that are on their checklist, such as oversight structure or algorithm validation?
McManus: I think it’s all about standards. It gives us a chance to do that methodically and at scale consistently. We’re also a HIMSS Stage 7 EMRAM organization. We’ve worked hard at Inova to ensure we have the highest credentials for our data and AI program. We were honored to be the first in getting that designation with the Joint Commission. A lot of what that certification is about is: are you able to demonstrate through Joint Commission’s guidelines that you are responsible in your use of data at scale? Are you organized? What are your controls? What are your standards? How are you ensuring that there are feedback loops that still focus on a culture of safety?
Something that’s on our Q1 and Q2 roadmap is working with our partners at Press Ganey to do the work on enabling an official AI safety reporting mechanism. We have an informal function now, but we will be really changing what that front door looks like, so that AI-related safety events are able to be reported with the same rigor as other type of safety events going forward. Signal 1 gives us an important tool as part of our response plan if those type of events are to occur.
HCI: Jon, are there other platforms that you looked at? I’ve seen a couple of startups announced in the same space. One was Vega Health, which is a spin-out from Duke Health.
McManus: Dr. Mark Sendak of Vega Health and I know each other relatively well. He came by and we had a good update on Vega. I think a lot of the problem that his team is solving is how to deal with the noise of the AI vendor space more consistently. It’s a little bit less about monitoring your existing production deployments.
I’ve also had a chance to speak with Dennis Chornenky, CEO of Domelabs AI, and they’re doing a very interesting product that’s a little bit more on the governance side, not as much on the monitoring side.
When we had a chance to speak with Tomi and his team, there was really an opportunity to do both. We felt that we needed a platform to help manage the scale of governance that was required, but we also needed a technological platform to do universal monitoring. Epic, for example, has invested quite a bit in its trust and assurance suite, but it’s still very much good for monitoring things in Epic. It’s not available to serve the dozens of features that we have.
