As far back as 2015, agency insiders criticized delay data as “suspect,” “incomplete or unreliable,” but agency chiefs still trumpeted misleading stats
MTA’s new “Major Incidents” metric similarly misleads by failing to account for incidents caused by planned work on subway tracks or signals
Stringer blasts MTA’s “Culture of Concealment,” urges more transparency in current data reporting
Relying on internal MTA documents never released publicly, New York City Comptroller Scott M. Stringer today released a damning investigation that reveals how the MTA has misrepresented its performance and delay data for years and shifted the focus from the subways’ real operational problems. Shockingly, these practices endured despite repeated warnings from agency insiders that much of the delay data released to the public was all but meaningless. The investigation, based on internal documents primarily prepared by the MTA’s Performance Analysis Unit, as well as interviews with key MTA officials, makes clear that a history of distorted metrics has prevented transparency into what is going on underground in our subways. More specifically, the investigation found that the MTA failed to provide clear guidance on how to identify the causes of millions of delays, relied on distorted metrics to report system progress, and hid delay data where causes were unknown.
“It is an insult to anyone who has ever been late to work or stranded at the station that MTA leadership passed along bogus delay data just to make the agency look good, even as its own staff were raising red flags,” said Comptroller Stringer. “While the new leadership deserves credit for trying to clean things up, the MTA still has a long way to go to ensure accuracy and reliability and to regain the public’s trust. For too long the MTA has failed to transparently report what is actually going on underground, a problem which can only mean more delays and more frustration for the working New Yorkers who rely on the subway every day.”
The Comptroller’s investigation found:
Incomplete and Unreliable Delay Data
From mid-2015 forward, numerous in-house MTA analyses concluded that the Authority’s internal systems could not accurately identify the causes of delays and, in particular, chronically misattributed delays to “Overcrowding.” A July 2015 internal MTA memo described the methodological breakdowns in stark terms: “No policy or guidance exists on how dispatchers should properly identify the cause of a particular delay or on how delays should be assigned to incidents… Dispatchers rely on train crews to report the cause of delays, and these explanations are suspect.”
Meanwhile, in January 2016, another memo noted that “much of the delay data is incomplete or unreliable, particularly the classification/categorization of delays and the assignment of delays to particular incidents.” Despite multiple warnings, MTA officials continued for years to publicly promote misleading information around delays, resulting in the agency being portrayed in a more positive light.
Meaningless and Disappearing “Wait Assessment” Results
Throughout 2016, MTA officials repeatedly asserted that subway service was improving based on apparent increases in “Wait Assessment” scores, a metric intended to measure the reliability and consistency of train service and touted by the MTA as its most important indicator of customer service quality.
However, internal analyses obtained by the Comptroller’s Office showed MTA analysts warning that alleged improvements in “Wait Assessment” were statistically insignificant and likely the result of “sample error.” After technological advancements in data collection finally made clear that “Wait Assessment” scores had actually gotten worse, not better, the MTA quietly stopped touting the metric without ever acknowledging that the prior reporting had been incorrect.
Hidden “Unknown” Delay Causes
For nearly a decade, the MTA further distorted its disclosures by effectively hiding delays it had chalked up internally to “Unknown” causes. Instead of publicly listing these delays simply as “Unknown” in a separate category, MTA officials folded them into other publicly identified delay causes – burying the “Unknowns” without any explanation.
In this way, in Monthly Operations Reports provided to the public and the MTA Board from 2013 through mid-2018, the MTA hid 525,710 delays internally grouped under “Unknown.”
Flaws in New Metrics
Similar to the MTA’s misattribution of “Unknown” delays, the MTA’s current reporting of “Major Incidents” – defined as any event that delays 50 or more trains – conceals critical information. Specifically, “Major Incidents” exclude a large category of service disruptions tracked internally by the MTA – “Planned Work,” which includes track and signal maintenance that regularly bogs down whole subway lines. To date, the MTA has yet to acknowledge publicly that its “Major Incidents” metric excludes such work and, as such, fails to capture what is actually going on underground.
Under New York City Transit President Andy Byford, the MTA has recently acknowledged some of these issues, introduced reforms, and begun to reduce delays. Yet the report reveals systemic deficiencies, some of which remain embedded in the MTA’s performance reporting and continue to obscure the true causes of delays.
Comptroller Stringer is calling on the MTA to:
- Structure public reporting of delay data to maximize transparency, reliability and accountability.
- Publicly define all delay categories, specifically indicating what each one includes and, as necessary, omits.
- Ensure that all protocols relevant to performance reporting are formally codified in official policies and procedures, including establishing written definitions and instructions for all key terms, data categories, and work protocols.
- Adequately train all relevant personnel on protocols relevant to performance reporting.
- In the context of public reports of “Major Incidents,” provide the public with information about all categories of service disruptions tracked as incidents within its Subway Incident Reporting System (SIRS) that cause 50 or more delays, including specifically “Planned Work.”
- Transparently disclose in each Monthly Operations Report and on the MTA’s subway performance dashboard, the methodologies used to calculate performance metrics, including all exceptions and revisions to those methodologies, and including methodological weaknesses.
- Make available on the MTA’s website or through an Open Data portal each month all data in the SIRS database and any other databases relied on for public reporting.
To read the full findings of Comptroller Stringer’s investigation, click here.
