How we did it: Methods used to test the relationship between Taser model and effectiveness

APM Reports began investigating Taser effectiveness in February 2018 by filing records requests to police departments, including those in the 20 largest U.S. cities. We wanted to know three things: how often those departments used Tasers, how often the departments categorized those uses as effective and which type of Taser the departments used.

The answers would help us get closer to what became the central question of the investigation: How had changes in design between different models affected effectiveness? That meant — along with data that showed effectiveness — we also needed the model of Taser used in each incident and whether the definition of effectiveness was consistent across the data.

We weren't sure how many departments tracked that information — or how hard it would be to acquire — which is why we asked more than 20 departments, hoping that perhaps 10 would have the data. Indeed, not every department could produce data that met these criteria. Some departments had records of Taser uses, but did not include an evaluation of effectiveness. Others did not track the model used in an incident.

There were other complications.

The Fort Worth Police Department, for example, had data that allowed analyzing effectiveness and model, but added a "limited" category of effectiveness around the same time it started switching to newer-model devices. The simultaneous changes created a comparison problem: Any change in effectiveness could be attributed to the different device, or it could be because the outcome was now designated "limited."

Twelve departments provided data that allowed the calculation of overall effectiveness rates. Of those departments, four — Charlotte-Mecklenburg, Houston, Los Angeles and New York City — had data that allowed comparisons between the number of effective uses of different models. In those cities, the percent of effective uses for the newer models, X26P or X2 devices, was lower than that of the X26 devices, the older models.

Could this relationship be due to chance? We used statistical methods to try to find out.

APM Reports used a statistical test called Pearson's chi-squared test on the table of effective and ineffective uses by model. For Houston, Los Angeles and New York City, the test showed a value that led APM Reports to conclude that there was a statistically significant relationship between model and effectiveness.

Even though the test revealed evidence that effectiveness and device model were related, there was still the possibility that the relationship was due to another factor related to both device type and effectiveness.

APM Reports then used a statistical method called logistic regression, which finds the relationship between an outcome variable that can have one of two values, and multiple predictor variables. (Logistic regression is a common analytical method in social science and data journalism.)

With the Taser data, the outcome variable is whether the use was effective. Predictor variables could be the age of the person on whom police used the Taser, or the rank of the officer who used the Taser.

For each city where the chi-squared test showed a significant relationship between model and effectiveness, APM Reports analyzed the variables in the data to see which seemed to correspond to differences in effectiveness. APM Reports processed some variables so that they would meet the requirements for use in logistic regression analysis.

For the variables that seemed to be related to effectiveness, APM Reports built one single-variable logistic regression model with effectiveness as the dependent variable, and another with Taser type as the dependent variable. If the variable being screened was a statistically significant predictor of both effectiveness and Taser type, APM Reports further analyzed the effect of including the variable. Variables that changed the predicted effect of Taser type in the simple model were included in a final logistic regression model.

Logistic regression assumes that each observation — in this case Taser use — is independent of others. However, that's not always true with the Taser data. Depending on how the department tracks effectiveness, there could be multiple records related to Taser use from one incident.

Los Angeles, for example, tracks each time an officer pulls the Taser's trigger. So, if the officer fires probes at someone and follows that up with a drive stun, there will be two records in the data. Clearly, those uses are related. To account for that, APM Reports used a statistical technique called a generalized estimating equation.

In the end, our models found that there was a statistically significant relationship between Taser model and effectiveness, even when accounting for other variables and the relationship between uses in the same incident.

In the three cities we analyzed, the statistical models predicted that the use of a newer-model device, either an X26P or X2, would have lower odds of being effective than when using an X26 device.

Although it's possible that the relationship between model and effectiveness is the result of some other factor not reflected in the data, APM Reports' analysis strengthened the evidence that newer-model Tasers are less effective than the X26.

Counting people shot and killed after a Taser

APM Reports started with the database of people shot and killed by police that the Washington Post has compiled since 2015 and reviewed shootings from 2015 to 2017 to find incidents where it appeared that the police shot someone after they used a Taser on that person. In addition to the shootings flagged in the Post's data, APM Reports identified other shootings where a Taser had been used.

APM Reports filed public records requests with the police departments, prosecutors and investigating agencies for investigatory reports or other records providing officer and witness accounts of the incidents, purchase records or data downloads from the devices used in the incident and video footage from the incidents. When records were unavailable through public records requests, APM Reports reviewed media reports of the incident to better understand the role that officers' Taser use played leading up to a fatal shooting.