CA Notify, the Silicon Valley-originated exposure notification app, launched in December with all the trappings of success. An Oxford-backed study concluded that if just 15 percent of the state signed up, cases could fall by 8 percent — a possibility that seemed within reach, given Gov. Gavin Newsom and the state’s heavy push for residents to register.
And register we did: Roughly a quarter of Californians signed up. And yet, with more than 9.9 million state residents at least nominally using the app, scientists are scratching their heads as to whether CA Notify made any difference at all.
There’s an app for that, as the saying goes — but instead of demonstrating the utility of quick technological solutions to a hard problem such as Covid contact tracing, CA Notify ended up more a case study of the shortfalls of allowing Silicon Valley to take the lead on public health, according to researchers.
“I haven’t seen evidence that the app has made a public health difference,” said Dr. Greg Marcus, a professor of medicine at UCSF. “A very technically savvy, well-financed individual or company can build an app and enroll a huge number of people, but that is far from sufficient to make good use of those data.”
The app protects user privacy to the point that it has been incredibly hard to ascertain its impact. Moreover, the app is designed in such a way that its usefulness to the communities hardest hit by the pandemic is likely low. And by getting ahead of other technology in the rush to partner with the government, Silicon Valley may have closed the door on other, more fruitful applications.
“We made it so private that any meaningful analysis is impossible,” said Doug Schultz, a spokesperson for the Minnesota Department of Health, one of 20 states employing apps using the same technology as CA Notify.
CA Notify, built on Apple and Google’s Exposure Notification software, sends out exposure notifications based on Bluetooth proximity and does not collect people’s names, contact information or location. The app only sends out alerts to those who are believed to have been within six feet of a Covid-positive individual for at least 15 minutes.
The design was marketed as a feature, not a bug: “This was intentional since the main worry was that people wouldn’t use it if they thought that their location or demographic data would be used,” Schultz added.
As a result, there’s not much data to work off of to evaluate whether the app was effective. California’s public health department was unable to provide numbers on how many people reported positives through the app or how many potential exposure alerts were sent out, which would illuminate whether people are actively using the app.
The state’s public health department, instead, noted that there have been more than 108,000 visits to a webpage that provides self-quarantine information and resources linked in the text messages sent out to those identified as potentially exposed.
“Not only does this save valuable time in getting contacts into quarantine before they can be reached with manual contact tracing calls, but some of these contacts may be unknown to the infected person and so would not be identified in a subsequent contact tracing interview,” the state’s public health department wrote in an email.
Perhaps, but this is simply a measure of how many people visited a site. It remains unknown how many of these visitors were previously unaware of being exposed or even followed the advice they found.
The state public health department also declined to disclose what zip codes were represented in the sign-ups. This deficit precludes evaluation of whether the hardest hit zip-codes — often the most hesitant to adopt technology like this — used the app. (The state has conducted outreach in English and Spanish across multiple channels, including on social media and through local health partners, to try and reach these groups, according to a state health department email.)
And there’s an upper limit to how effective a contact tracing app can be in neighborhoods hardest hit by the pandemic, according to Dr. Bernard Lo, a UCSF professor emeritus who co-wrote an analysis on ethically implementing effective contact tracing.
“The Mission, Bayview Hunters Point, the Tenderloin — do those people use the app, and would the app be helpful?” Lo asked. “If someone is infected, but they live in a crowded apartment with a lot of other people, you can’t help but be part of the chain of transmission. Certainly in homeless shelters, if you’re exposed, there’s very little you can do.”
Perhaps a better-designed or better-planned app could have helped these populations, but meticulous responses fall outside the Valley’s modus operandi, “move fast and break things.”
“The Silicon Valley stuff is slick, it goes fast, but they may be more looking at the trees rather than the forest,” said Dr. George Rutherford, a UCSF epidemiologist and infectious disease expert. “There’s no way my mother-in-law’s going to be able to use this. There’s no way somebody who is sharing phones with four other farmworkers is going to be able to use this.”
One could make the argument that the heavy promotion and widespread adoption of the CA Notify app was, at worst, harmless — and if just one death was averted by an exposure notification, wasn’t this all worth it?
Perhaps in a vacuum, yes — but in the real world, there’s limited political capital for pandemic interventions and a limited number of apps any one person is willing to download and actively use. Publicizing the app also came with a price tag: In response to a query about how much public money was invested in the project, the state’s public health department wrote that it has entered a $2 million inter-agency agreement with UC San Diego to support the project through May.
Some UCSF researchers said they suffered the tradeoffs involved in public health departments prioritizing CA Notify first-hand.
UCSF scientists helped develop Covidseeker and Citizen Science. These are two lesser-publicized studies that also collect user phone data — but with the intent to analyze and make public the results. When the researchers approached public health departments in the state to partner, they said they were turned down in favor of CA Notify, sometimes referred to as Covid Notify.
“The answer [from health departments] was, well, ‘There’s already Covid Notify,’” said Marcus, who co-leads Citizen Science. “Covid Notify was everywhere we went. ‘We’re going to work with Covid Notify, so we don’t need to work with you.’”
The state’s department of public health contests this, writing that it did not turn down a collaboration based on an existing partnership with Google and Apple.
“The state talked to many research organizations and vendors in the process of exploring Exposure Notification (EN) technology for California,” the department wrote. “We had early conversations with UCSF and UCSD regarding their exploration of this technology, and Covid Watch, which was demonstrating their solution in Arizona.”
Both UC studies have, regardless, managed to move forward in the past months. Covidseeker is starting to analyze location data contributed by the public to improve contact tracing and better understand the spread of Covid. Citizen Science has enrolled about 60,000 participants that it surveys and collects location data from to further analyze risk factors for Covid, and is now publishing studies based on the data.
But the more people involved in the studies, the better —something state and county health department partnerships would have facilitated.
“The utility is directly related to how dense the population is that’s participating. We still would love to catch fire, go viral and enroll a million or more,” Marcus said. “The more data, the more confident we can be of the results.”
And this data would’ve been a public asset.
“With Google and Apple, they’ve never shared that information,” said UCSF’s Dr. Yulin Hswen, who leads Covidseeker. “With us, when working with public health forums, we were very much willing to share data and collaborate.”
If there’s a lesson that should be taken away from CA Notify for future pandemics, Hswen said, it’s that public health interventions should be designed with metrics that can be evaluated.
“It’s important to do evaluation research before taking something on to understand if something is effective,” she said, “instead of just selection based on potential.”
About a year ago, the computer security expert Bruce Schneier predicted this:
“My problem with contact tracing apps is that they have absolutely no value,” Bruce Schneier, a privacy expert and fellow at the Berkman Klein Center for Internet & Society at Harvard University, told BuzzFeed News. “I’m not even talking about the privacy concerns, I mean the efficacy. Does anybody think this will do something useful? … This is just something governments want to do for the hell of it. To me, it’s just techies doing techie things because they don’t know what else to do.”
https://www.schneier.com/blog/archives/2020/05/me_on_covad-19_.html
What confuses me is that my phone might have a better record than my memory of where I’ve been and who I’ve been near. Maybe the issue is that phones have a hard time reliably detecting how long two people were near each other, or they can’t tell whether two people were in the same room vs. in the same location but on different floors?