For startups and investors aiming to bring health technology to underserved populations, evidence of impact should be everything: It’s what shows employers and payers a new offering is worth paying for, and what can help convince patients to give it a try.
But while the market is crowded with companies claiming their products meaningfully improve health — especially for underserved groups — there is still no industry-wide standard slate of metrics to fully evaluate them. Some are reporting how many new appointments they’ve created among lower income or underrepresented populations; others are coming up with new ways to approximate how their app or wearable specifically helped rural and urban patients with chronic conditions, for instance.
“It’s a really big problem,” said Alex Lennox-Miller, a health care analyst with venture research firm CB Insights. “So much of the future of health systems is to be in care that’s happening outside of appointments — as we’re talking about patient engagement, as we’re talking about remote monitoring, as we’re talking about home care, those are the places we have the worst ways to measure and the least visibility into their impact.”
advertisement
Measurement challenges extend beyond issues of access and equity; venture fund Rock Health recently developed its own “clinical robustness” score after concluding that only a small portion of health tech companies even make any claims about their clinical impact on patients.
And while there’s growing interest among investors in turning from traditional financial metrics like market size and growth rate to numbers reflecting meaningful clinical impact — lowered A1c levels, for instance — it’s “still relatively early, so these metrics are not yet fully standardized,” said Sunny Kumar, a partner at GSR Ventures, which invests in early-stage health tech companies.
advertisement
Still, there’s debate about what exactly companies should measure to show that they’re meaningfully reaching patients who might not have strong established relationships with traditional health care. “What is a clinical touch? Does a text message count, is it nursing visits, is it only when they come into the clinic?” said Zen Chu, who leads MIT’s Hacking Medicine Initiative.
Given that companies are still experimenting with new measures, it’s typically up to health systems or employers buying these services to set their own benchmarks for equity — whether it’s reaching specific populations or getting them to stay active on the app or wearable device, Lenox-Miller said. “It’s almost more of an issue for investors, because ideally if you’re a buyer or health system looking at these solutions, you’re going to have done the legwork ahead of time to figure out the issues you’re trying to address.”
Investors, on the other hand, should “be aware that these aren’t questions with hard answers yet and start paying attention to what companies, what founders, are acknowledging this and saying, ‘We’re figuring this out, we don’t have a solid answer,’ [and] a willingness to even acknowledge that this is a question,” Lenox-Miller added.
If they don’t get it right, he said, companies making unsubstantiated claims about equity risk losing trust from providers and payers aiming to reach those underserved groups. “What they’re doing is trying to fake it till they make it and hoping that they stick many answers out before people start figuring out that they’re really not making a difference. That destroys the trust that providers, health care organizations, and patients have,” he said.
The lack of standards also presents an opportunity for companies to get creative, he added. Ventures like Alphabet-spinout Cityblock, which focuses largely on urban Medicaid populations, “have the privilege to almost be able to start to figure out what some of these metrics should be and what those data points should look like,” he said.
Cityblock CEO Toyin Ajayi told STAT that while there are some widely used measures for clinical improvement, like the ones developed by the National Committee for Quality Assurance, metrics like engagement might require different benchmarks for different services or populations. But she argued that any company or provider serious about reaching underserved populations likely has some way to measure how often patients interact with the service and how satisfied they are with it, as well as how often they meet clinicians.
Instead of searching for an industry standard for these factors, she said, health tech ventures may develop their own measures.
“If you’re seeking to address serious mental illness in populations of folks who struggle with mental health challenges and social needs, then we can all agree that a text message is not enough…if you’re seeking to move the needle on getting someone who has insurance, who has a primary care provider, to see their primary care provider for a preventive visit, and you know they’re digitally engaged, maybe a text message is enough,” she said. “I agree there isn’t a standard measure. But I would argue that that may not be the right thing to push for.”
Cityblock, for instance, has varying benchmarks for engagement depending on a patient’s health risk levels. “We certainly have built out those types of metrics that are very bespoke to our understanding of our population, and a recognition that that one size fits all is not going to be the right answer,” she said.
Some industry groups are whipping up their own tools for better quantifying health tech companies’ reach into underserved populations. The Digital Medicine Society helped build a “Market Opportunity Model” that CEO Jennifer Goldsack said companies could use to slice their market and revenue size opportunities by variables like age, race and ethnicity, income, education, and disability status.
But she said health tech companies and investors are often preoccupied with measuring factors like whether new patients download or use apps instead of focusing more on whether their health improves. “As an industry we’re drilling down on the wrong thing. We’re spending time with great consternation about new sorts of process measures, engagement, and retention. How about whether it makes people better?” she asked.
Health tech companies facing increasing economic uncertainty and belt-tightening from customers — including employers and payers — could find that clearer metrics on patient impact will make or break their businesses, said Rock Health president Tom Cassels. “What I’d expect to see is a reversion to the mean with respect to pitches, which is, ‘we can’t go in without some evidence of life or patient outcomes, because our competitors have them.’”
This story has been updated with additional reporting on Cityblock Health.
This story, part of a series on health tech for underserved populations, was supported by USC Annenberg Center for Health Journalism’s national fellowship.
