SFPD system meant to prevent misconduct is eviscerated as ‘highly inaccurate’ by outside researchers

The San Francisco Police Department’s in-house system for preventing misconduct by officers is “highly inaccurate,” flagging officers who do not need interventions and missing officers who do. 

That’s according to an analysis conducted by the University of Chicago’s Center for Data Science and Public Policy and recently obtained by Mission Local. The 32-page study was undertaken with the police department’s cooperation. 

It critiqued the SFPD’s “Early Intervention System,” which keeps track of police officers’ documented use of force incidents, citizen complaints, lawsuits, and other “indicators” that may point to future misconduct. If the system “flags” officers, higher-ups can intervene before their behavior becomes problematic. 

The SFPD has, nominally, utilized an early intervention system for more than a decade. But it has not been upgraded to reflect today’s technological capabilities — and researchers found the department’s current system to be archaic.

They compared the SFPD’s current system to a system the university has developed, and the report asserts the University of Chicago’s prototype system outperforms the SFPD’s. Using the same raw SFPD data, the university’s system identified a greater number of SFPD officers who went on to engage in problematic behavior than the SFPD’s own system, according to the study. 

Crunching data from the third quarter of 2016, for example, the university’s system identified 33 officers in need of intervention, while the SFPD system found only 19. 

Moreover, the SFPD system flagged too many officers who were not problematic. A mere 10 percent of the officers the SFPD flagged went on to engage in some kind of misconduct. That’s because the SFPD system relies too heavily on signs relating to officers’ out-of-context activity, such as use-of-force incidents in a three-month time period, the study says — as opposed to its “data-driven” approach that considers an officer’s history and many more factors such as arrests and training. 

Consequently, based on interviews with SFPD staff, researchers “found that [the early intervention system] lacks legitimacy among many officers and supervisors.” 

“Some officers even take pride in receiving alerts because they feel alerts indicate hard work,” the study continues, noting that officers and supervisors alike feel that the “paperwork required to address each alert feels like a burden and creates resentment.” 

Furthermore, the study reveals, supervising officers rarely act when the system alerts them to a potentially problematic officer, “which means officers likely do not receive the support intended by the program.” 

Review process too subjective 

Any time an officer receives an alert, one sergeant has sole discretion over whether the department should intervene with that officer. This setup has been repeatedly questioned by San Francisco Police Commissioners. 

“My issue with the [early intervention] system is the subjectivity, especially when the decision lies with one person, and that one person will decide whether someone needs intervention or doesn’t,” said Commissioner Cindy Elias at Wednesday night’s Police Commission meeting. 

Indeed, in the second quarter of 2019, the system flagged officers 175 times — and the sergeant, weighing input from supervisors, determined that none of them required an intervention. 

Commission President Bob Hirsh also raised questions at an October meeting. “Is that just subjective? Is there some protocol you follow? How do folks make that determination?” he asked Sgt. Wesley Villaruel, who is currently the sergeant in charge of reviewing alerts. 

Villaruel indicated that the department has no codified method. Villaruel, rather, uses his individual judgment to decide whether the officer is showing a “risky” pattern of behavior. He will review an officer’s incident report, body camera footage, and the circumstances of an alert, such as whether an officer used force unnecessarily. “It’s on a case-by-case basis for me,” Villaruel said. 

In all 175 cases, he opted to not intervene. 

The University of Chicago study also critiqued this review process, noting that a sergeant’s subjective review does not necessarily result in higher accuracy. “About the same number of flags are correct before and after the review process, which leads us to believe that there is a potential in improving this Sergeant review process.” 

The researchers added that the reviewing sergeant is often overwhelmed with a high volume of alerts generated by officers pointing their guns at people, which was around “150 alerts every two months” in 2016. “This high volume of alerts has strained the … sergeant’s ability to review each alert.” 

A new system? 

The study said the university’s own “data-driven” system — which utilized “machine learning”  – compiles factors beyond use-of-force, complaints and lawsuits, such as traffic stops, arrests, compliments, dispatches, internal investigations, and training. Over time, it evolves and determines which factors best identity officers most at risk. 

Rather than simply flagging officers, it ranks all of the officers from highest to lowest risk. “The risk score for an officer is highly contextual and changes over time since it depends on the officer’s behavior and situations they have been exposed to recently,” the study says. 

It’s unclear whether the SFPD has any interest in adopting the University of Chicago system. “SFPD is currently exploring options with regards to vendors who may offer beneficial technologies,” was all SFPD spokesman David Stevenson would offer. 

Commenting generally on the report, Stevenson acknowledged that the study highlighted challenges with the current system. It was created “at a time when technology was nowhere near where it is today,” he said. (The current system, according to the report, began in early 2016.)

“Any program will see a decay over time if the technology running that program is not updated as technology progresses,” Stevenson said. “The report highlighted that the current EIS system is inaccurate due to the fact that EIS generates alerts on officers that do not go on to have adverse incidents.” 

Stevenson also commented on the low number of formal “interventions” the SFPD has initiated — a mere six over the last five years on hundreds of flagged officers. He said that informal interventions happen frequently.  

“While these meetings between supervisor and officer do not constitute an intervention nor are they discipline related, they do trigger a dialogue between the supervisor and the officer that may not have occurred without receiving the EIS alert,” he said. 

Stevenson would not offer a timeframe for if or when the department will replace the outdated system. But the report was clear: The “SFPD should develop a data-driven” early intervention system.