Summary: Chapter 5

Chapter 5: Civilian Casualties: Justice in the Age of Big Data

This brief chapter surveys the ways in which Big Data models are applied to criminal justice. O’Neil begins by describing the rise of predictive policing using maps of crime “hotspots” within a city. She argues that predictive policing models have a dangerous feedback loop in that they send officers to areas where many minor “nuisance” crimes, such as underage drinking, are witnessed. These then contribute to the model’s designation of the neighborhood as “high-crime,” diverting police resources from the prevention of homicide, robbery, and other serious offenses. In the worst case, such models lend an air of credibility to practices like stop-and-frisk or zero-tolerance policing that alienate communities and erode trust. Meanwhile, O’Neil points out, white-collar crimes seldom get this kind of intensive attention from law enforcement.

O’Neil draws her next example from the courtroom. In many US states, she notes, judges apply mathematical models of recidivism risk in sentencing offenders. Often, judges use these models to support greater sentences for likely recidivists, but O’Neil finds this misguided: longer prison sentences heighten recidivism risk by making it harder for an inmate to rejoin society. Police departments have used similar “digital dragnets” to try to identify individuals likely to commit crimes before they do so. This, O’Neil says, leads to the surveillance and intimidation of innocent people who happen to live in impoverished, high-crime areas.

Analysis: Chapter 5

Criminal justice represents a perfect storm of the factors that create and exacerbate the dangerous models O’Neil calls WMDs (“Weapons of Math Destruction”). The feedback loop, or the tendency of the model to create the conditions it aims to predict, is pronounced in every example that O’Neil cites. Predictive policing confirms the status of poor neighborhoods as “high-crime” neighborhoods, and recidivism models become tools for dealing out sentences that fail to prevent (and may even promote) recidivism.

The models described in Chapter 5 thus fit several of O’Neil’s proposed characteristics of WMDs. Not only are they prone to feedback loops, but they do significant harm to those they affect, and they operate (collectively) at a large scale. Yet in principle, these models have greater oversight than the largely opaque and proprietary ones used by financial companies. This potential for transparency and accountability might seem to negate or at least diminish their status as WMDs, whose opacity and unaccountability O’Neil considers to be defining features.

However, as O’Neil points out, oversight is only useful if it is exercised consistently and conscientiously. When police departments fail to distinguish between violent crimes and “nuisance” crimes, they are discarding potentially useful data. In the case of recidivism modeling, the courts face a conflict of interest: publicly detailing the model’s inputs would potentially allow people to game the system and obtain a lighter sentence. The models are unaccountable because accountability, though possible, is not exercised, and they are opaque because transparency is deemed too risky.