SFPD officer
Police officers walked alongside demonstrators in a 2014 photo.

The San Francisco Police Department’s in-house system for preventing misconduct by officers is “highly inaccurate,” flagging officers who do not need interventions and missing officers who do. 

That’s according to an analysis conducted by the University of Chicago’s Center for Data Science and Public Policy and recently obtained by Mission Local. The 32-page study was undertaken with the police department’s cooperation. 

It critiqued the SFPD’s “Early Intervention System,” which keeps track of police officers’ documented use of force incidents, citizen complaints, lawsuits, and other “indicators” that may point to future misconduct. If the system “flags” officers, higher-ups can intervene before their behavior becomes problematic. 

The SFPD has, nominally, utilized an early intervention system for more than a decade. But it has not been upgraded to reflect today’s technological capabilities — and researchers found the department’s current system to be archaic.

They compared the SFPD’s current system to a system the university has developed, and the report asserts the University of Chicago’s prototype system outperforms the SFPD’s. Using the same raw SFPD data, the university’s system identified a greater number of SFPD officers who went on to engage in problematic behavior than the SFPD’s own system, according to the study. 

Crunching data from the third quarter of 2016, for example, the university’s system identified 33 officers in need of intervention, while the SFPD system found only 19. 

Moreover, the SFPD system flagged too many officers who were not problematic. A mere 10 percent of the officers the SFPD flagged went on to engage in some kind of misconduct. That’s because the SFPD system relies too heavily on signs relating to officers’ out-of-context activity, such as use-of-force incidents in a three-month time period, the study says — as opposed to its “data-driven” approach that considers an officer’s history and many more factors such as arrests and training. 

Consequently, based on interviews with SFPD staff, researchers “found that [the early intervention system] lacks legitimacy among many officers and supervisors.” 

“Some officers even take pride in receiving alerts because they feel alerts indicate hard work,” the study continues, noting that officers and supervisors alike feel that the “paperwork required to address each alert feels like a burden and creates resentment.” 

Furthermore, the study reveals, supervising officers rarely act when the system alerts them to a potentially problematic officer, “which means officers likely do not receive the support intended by the program.” 

Review process too subjective 

Any time an officer receives an alert, one sergeant has sole discretion over whether the department should intervene with that officer. This setup has been repeatedly questioned by San Francisco Police Commissioners. 

“My issue with the [early intervention] system is the subjectivity, especially when the decision lies with one person, and that one person will decide whether someone needs intervention or doesn’t,” said Commissioner Cindy Elias at Wednesday night’s Police Commission meeting. 

Indeed, in the second quarter of 2019, the system flagged officers 175 times — and the sergeant, weighing input from supervisors, determined that none of them required an intervention. 

Commission President Bob Hirsh also raised questions at an October meeting. “Is that just subjective? Is there some protocol you follow? How do folks make that determination?” he asked Sgt. Wesley Villaruel, who is currently the sergeant in charge of reviewing alerts. 

Villaruel indicated that the department has no codified method. Villaruel, rather, uses his individual judgment to decide whether the officer is showing a “risky” pattern of behavior. He will review an officer’s incident report, body camera footage, and the circumstances of an alert, such as whether an officer used force unnecessarily. “It’s on a case-by-case basis for me,” Villaruel said. 

In all 175 cases, he opted to not intervene. 

The University of Chicago study also critiqued this review process, noting that a sergeant’s subjective review does not necessarily result in higher accuracy. “About the same number of flags are correct before and after the review process, which leads us to believe that there is a potential in improving this Sergeant review process.” 

The researchers added that the reviewing sergeant is often overwhelmed with a high volume of alerts generated by officers pointing their guns at people, which was around “150 alerts every two months” in 2016. “This high volume of alerts has strained the … sergeant’s ability to review each alert.” 

A new system? 

The study said the university’s own “data-driven” system — which utilized “machine learning”  – compiles factors beyond use-of-force, complaints and lawsuits, such as traffic stops, arrests, compliments, dispatches, internal investigations, and training. Over time, it evolves and determines which factors best identity officers most at risk. 

Rather than simply flagging officers, it ranks all of the officers from highest to lowest risk. “The risk score for an officer is highly contextual and changes over time since it depends on the officer’s behavior and situations they have been exposed to recently,” the study says. 

It’s unclear whether the SFPD has any interest in adopting the University of Chicago system. “SFPD is currently exploring options with regards to vendors who may offer beneficial technologies,” was all SFPD spokesman David Stevenson would offer. 

Commenting generally on the report, Stevenson acknowledged that the study highlighted challenges with the current system. It was created “at a time when technology was nowhere near where it is today,” he said. (The current system, according to the report, began in early 2016.)

“Any program will see a decay over time if the technology running that program is not updated as technology progresses,” Stevenson said. “The report highlighted that the current EIS system is inaccurate due to the fact that EIS generates alerts on officers that do not go on to have adverse incidents.” 

Stevenson also commented on the low number of formal “interventions” the SFPD has initiated — a mere six over the last five years on hundreds of flagged officers. He said that informal interventions happen frequently.  

“While these meetings between supervisor and officer do not constitute an intervention nor are they discipline related, they do trigger a dialogue between the supervisor and the officer that may not have occurred without receiving the EIS alert,” he said. 

Stevenson would not offer a timeframe for if or when the department will replace the outdated system. But the report was clear: The “SFPD should develop a data-driven” early intervention system. 

Follow Us

Julian grew up in the East Bay and moved to San Francisco in 2014. Before joining Mission Local, he wrote for the East Bay Express, the SF Bay Guardian, and the San Francisco Business Times.

Join the Conversation


Please keep your comments short and civil. Do not leave multiple comments under multiple names on one article. We will zap comments that fail to adhere to these short and very easy-to-follow rules.

Your email address will not be published. Required fields are marked *

  1. Julian,

    Thank you for the discussion.

    I’ve said for years that we should have an exchange program
    with Sister Cities which allows us to send small contingents of
    our Emergency System Personnel on 2 year tours with their

    London, Paris, Rome, Tokyo …

    Places that have earthquakes and huge fires and floods.

    They should have their own shows on SFGTV.

    It’s the basis of a truly International Emergency Mutual Aid System.

    Probably should end up being coordinated thru UN.

    Locally, I believe the main problem with the cops is that their
    leadership is pulled in all different directions because they
    don’t have an elected Chief immune (one would hope) from
    pressure from the Union or the Mayor’s Office.

    That last idea there on electing our Top Cop is not mine.

    Former Sheriff for 32 years, Michael Hennessey.

    His idea.

    Yes, it was.

    Best in the country.

    Go Giants!


  2. My takeaway from this is that profiling is both inaccurate and harmful whether it’s applied to stereotyped racial categories or stereotyped police officers. The tragedy is that profiling is widely regarded as “common sense” so it keeps getting reinvented as a shortcut solution to predict a person’s future behavior. But it never actually works when applied to a large number of random people.

  3. I think we need a new Police Commission Commission to oversee the Police Commission and make sure the Police Commission is properly overseeing the Police Department. Because there is a lot of low hanging fruit to be harvested. Not only an effective EIS system, but also implementing all those Obama-era Justice Department recommendations.

    1. How is the Chicago system better? It’s not explained in this article. How could any system proclaim “better” without investigating the nuances of each case?
      Sergeants are not the end of the system, they are the beginning. The Unit or station Captain has the final say. It’s their responsibility to make sure the sergeants assessment is correct. If not, the case should be reviewed again with concerns cited by the captain.