On the heels of the recent controversy over major statistical revisions of unemployment data by the Labor Department, the Bureau of Justice Statistics (BJS) has just released a report containing highly questionable state violent crime rates. The report’s unreliable numbers are not the result of errors in calculation or any deliberate attempt to skew the results. Rather, they are the product of applying a methodology that is incapable of producing accurate results for the statistical categories in question. Indeed, they provide further evidence of the folly of parroting the mantra, “Trust the experts.”
I’m intimately familiar with the BJS, having served as its director during the first Trump administration. The bureau generally focuses on providing national, rather than state or local, statistics. The report in question, however, provides crime rates for each of the 22 largest states, comparing those rates for the three-year period spanning 2017 to 2019 and the three-year period spanning 2020 to 2022. The findings suggest that the BJS should have stuck to producing national statistics.
For example, the BJS report claims that the number of victims of violent crime in Arizona dropped nearly in half during those three-year periods, to just 20.0 per 1,000 people from 36.8 per 1,000. But the state’s largest law-enforcement agency, the Phoenix Police Department, reports that over those periods the city’s homicides rose 42 percent and its violent crime rate rose by 9 percent. It’s awfully hard, therefore, to believe the BJS numbers that claim violent crime fell precipitously statewide when statistics from the state’s dominant city say otherwise.
The report’s comparisons across other states are similarly suspect. The FBI’s Uniform Crime Reporting figures collected from police show that over the years in question Georgia had a higher rate of violent crime than did the state of Washington, by about 10 percent. The BJS report, however, maintains not only that it was Washington’s violent crime rate that was higher than Georgia’s, but higher by triple. Likewise, the FBI figures show that North Carolina had a 25 percent higher violent crime rate than Indiana from 2020–22, while the BJS report again skews the opposite direction, showing Indiana’s rate was more than double North Carolina’s.
While the FBI’s figures have their own problems—as I noted last year in City Journal—it strains credulity to believe that one state had two-to-three times the rate of violent crime as another state, when reporting from the police in the states in question found something closer to the opposite.
Nor are these discrepancies explainable by the fact that the FBI doesn’t include simple assault in their data (a charge that is pretty much the lowest rung on the violent-crime ladder), but BJS does; because BJS also publishes figures excluding simple assault (see Appendix Table 3), and these reflect similar discrepancies.
Two things are going on here. First, BJS is relying on the National Crime Victimization Survey (NCVS), which dates to the Nixon administration and is the largest crime survey in the country and one of the largest federal surveys on any topic. The NCVS is an indispensable resource because it collects crime data from actual victims, rather than relying merely upon police reports. But the NCVS was principally designed to capture national statistics, not state-by-state ones. Its sample size and methodology are much better suited to the former than the latter. If one doesn’t (or doesn’t adequately) survey a key, high-crime neighborhood in Atlanta, for instance, the results won’t have much of an effect on the national numbers, but it could have a big effect on the numbers from Georgia.
The Trump administration has not yet installed a political appointee to run BJS, as is statutorily authorized by Congress, so the bureau is currently being led by a career employee in an acting director position. When I was appointed and served as the BJS head from 2017 to 2021, our senior statistician—who by then had served under six presidents, starting with Ronald Reagan—was extremely skeptical that the NCVS could accurately summarize statewide crime statistics, and he thought that recently initiated efforts to use it to do so should therefore be abandoned. I agreed with him. This latest BJS report strongly suggests that our skepticism was warranted.
The other thing going on is that the 2020 and 2021 versions of the NCVS were rendered essentially useless by a decision made during COVID by the Census Bureau (which administers the NCVS), to suspend the in-person interviews upon which the survey relies. The NCVS is extremely valuable but its data is also rather fragile. Changes in how it is administered can change its results, often dramatically. Thus, while the FBI reported a nearly 30 percent murder spike in 2020—an all-time record—the NCVS, which was prevented from being conducted in its usual manner for many months, indicated that violent crime actually dropped that year, a statistic contradicted both by Americans’ shared observed experience and by police forces’ own numbers.
As a result, the current BJS report compares an era of reliable NCVS figures (2017–19) with an era (2020–22) in which only one of the three years—2022—is reliable. The results, therefore, are essentially meaningless. Even if the NCVS could produce accurate state-level estimates—which it has shown no ability to do—the only meaningful comparisons would be between 2019 or earlier and 2022 or later, when COVID-era alterations couldn’t noticeably skew the numbers.
In short, statistical agencies’ unwillingness to question their own methodologies or execution can lead to problematic results. When groupthink takes over, such agencies often fail to examine whether the figures produced by a given methodology or process are believable.
We recently saw this at the Labor Department, where revising job-growth figures downward by 87 percent for May and 90 percent for June—in what economists at Goldman Sachs called “the largest two-month downward revision to the jobs numbers since 1968”—was summarily dismissed with just one uninformative, parenthetical sentence in a press release. When only 13 percent and 10 percent of the prior estimated job-growth remains after revisions, it should raise major red flags about the accuracy of the original estimates.
Yet the Bureau of Labor Statistics didn’t seem curious or concerned. Nor did it feel the need to provide any substantial explanation for these historic revisions. The American public was left with nothing but a plain assertion that it should accept that there was nothing unusual or interesting about something that hadn’t previously happened since before Nixon took office and we first landed on the moon.
Federal statistics deserve neither universal trust nor universal distrust. To paraphrase President Reagan, the appropriate way to view such statistics is to trust, but verify. When the methodology or execution clearly isn’t producing accurate statistics, as in the case of the latest BJS report, the “experts” need to get back to work and fix the flawed processes that led to the flawed results.
Jeffrey H. Anderson is president of the American Main Street Initiative, a think tank for everyday Americans, and served as director of the Bureau of Justice Statistics at the U.S. Department of Justice from 2017 to 2021.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.