The Independent View: NYC Blindsided by Bug that Crashed Wireless Network

Despite Warnings, New York City’s Technology Managers were Blindsided by the so-called “GPS Rollover”, Failing to Install simple but necessary Upgrades. After a Wireless Network used by City Agencies Crashed in April, the Police Department relied on Mounted License Plate Readers.

For at least a year, Federal Officials and Technology Companies had been Warning of the Rollover, a once-in-20-year Event that had the Potential to Wreak Havoc on Computer Networks around the World.

Yet somehow, New York City’s Technology managers were caught completely Off Guard, and did Nothing to prepare for the Calendar Reset of the Centralized Global Positioning System. As a result, a Wireless Network used by City Agencies Crashed, Crippling many Services that relied on it, including some Police Department License Plate Readers and a system to Remotely Control Traffic Lights. It took 10 days to get the Network Running again.

Officials at several City Agencies, including the Police Department and the Office of Emergency Management, knew about the Rollover, according to a Report released by the City on Friday.

But Officials at the Department of Information Technology and Telecommunications (DoiTT), which was Responsible for Operating the Wireless Network, claimed “that they were not aware” of the Rollover before it occurred, the Report said.

After the Network went down, Confusion, Poor Communication. and a Lack of Coordination hampered Attempts to get it Working again, according to the Report, which was Compiled by the Consulting Firm Gartner at a Cost of $300,000.

A week before the Report’s Release, the DoiTT Commissioner, Samir Saini, Resigned. Mayor Bill de Blasio said that Saini wanted to Return to the Private Sector, and Disputed the notion that his Departure was Connected to the Failure.

The Report did Not name any of the People who were Responsible for the missteps, and the City has Not publicly Disciplined anyone in relation to the Incident. Nonetheless, the Report, with its Revelations of Poor Preparation and the Chaotic Response, could be Embarrassing for de Blasio, who is Running for President and has argued that his Experience Managing the Nation’s Largest City makes him more Qualified than other Candidates.

On the day the Wireless Network Crashed, de Blasio was in Nevada, an Early Primary State, as he considered whether to Declare his Candidacy for President. City Hall initially tried to Hide the Shutdown. It made No Public Acknowledgment of the Problem and, in response to Questions from The New York Times, Officials initially Characterized it as a Routine Maintenance Issue.

The Report does Not indicate when de Blasio was told of the Problem or whether he was Informed of the Confusion surrounding the Attempts to get it working again. In a Statement accompanying the Report’s Release, Laura Anglin, the Deputy Mayor for Operations, who also Oversees the Information Technology Department, asserted that “there were no interruptions to city services during the NYCWiN outage,” but Acknowledged that “it is critical we learn from this event.”

Ms. Anglin’s Statement, however, is directly Contradicted by the Report, which Details several Service Interruptions. About Half of the City operated Signs showing Arrival Times at Bus Stops were Disabled, as were about 200 Cameras that Provide Online Images of Traffic Conditions. Many other Tasks handled by the Network were Knocked Offline, requiring City Workers to be Reassigned to Perform the Tasks Manually.

The Report made it clear the Episode could easily have been Avoided. The Wireless Network, like many other Computerized Systems, uses GPS Data to keep Track of Time. The GPS Rollover was widely known, and Government & Industry Notices encouraged Technology Managers to Upgrade Systems to Avoid possible Interruptions.

The Report’s Authors interviewed Eight Top Officials at the information technology Department, including Saini. But the Report said that No one at the Agency Admitted being Aware of the approaching Rollover. It does Not say whether it considered those Denials to be Credible, given the Amount of Publicity related to the Rollover in the Technology Industry.

Saini was Hired just a year and a half ago by the Mayor and was involved, among other Key Initiatives, in the Modernization of the 911 system.

According to the Report, the System could easily have been Upgraded by Replacing what is known as the Firmware in the Dozens of Nodes, or Antennas, that make up the Network.

Northrop Grumman, the Contractor that Maintains and Operates the Network at a Cost to Taxpayers of $37 Million a year, also did Not Alert City Officials to the Need for an Upgrade, the Report said. “Northrop Grumman worked expeditiously” with City Officials “to address the GPS rollover event,” said a Company Spokesman, Tim Paynter, in an emailed Statement.

Many Passages in the 35-page Report were Blacked-Out, which City Officials said was done for Security reasons.

The Wireless Network was Built for about $500 Million and has been in use since 2008. But today it is Used by Only about 10 City Agencies. The City Plans to Shut it Down in the coming years and Shift its Wireless needs to Commercial Carriers, which it says will Save Money.

But the City has been Slow to carry out Plans for such a Transition. The Report recommended that New York Review its Technology Infrastructure and Warned that the City “may be exposed to more risk than necessary regarding technology-related incidents.”