Delta computer risk ‘went undetected,’ CEO says

A failure to ensure backup power for some of its computer servers led to the meltdown at Delta Air Lines this week that forced more than 2,000 flight cancellations through Wednesday afternoon, the airline’s top executive said.

Delta CEO Ed Bastian, in an interview with The Atlanta Journal-Constitution, said he takes “personal responsibility” for the episode and added new detail to its cause.

The problem, Bastian said, was that about 300 of the Atlanta-based airline’s 7,000 servers were not wired to backup power. Early Monday, when a power control module at Delta’s technology command center failed and caught fire, it caused a surge to a Georgia Power transformer and a loss of power.

It was initially assumed a Georgia Power outage started the chain of events, but Bastian said that’s not the case.

“Delta’s responsible for this,” said Bastian, adding, “The buck stops with me. I’m the CEO.”

Bastian said the backup power vulnerability “went undetected and should not have.”

The problem brought Delta’s operations to a temporary halt and caused a cascade of cancellations and delays that disrupted travel for hundreds of thousands of people around the world.

By late Wednesday the effects were receding but Delta still canceled more than 300 flights for the day.

Bastian, a longtime Delta executive who became CEO last spring, said only the government-ordered grounding after 9/11 caused a shutdown as complete as Monday morning’s after the computer systems failed.

Some flights resumed after a few hours but by then the airline had to deal with an avalanche of displaced customers and flight crews and out-of-position aircraft, and systems that came back up were operating slowly.

“When we first were there analyzing it in the early hours, all you saw was that there was a loss of power,” Bastian said. “We’re not here pointing fingers at Georgia Power.”

The servers that lacked backup power are a relatively small percentage of the total, but they are “meaningful in their impact,” Bastian said.

When other servers that were connected to dual power sources came back on, he said, “they did not get responses from the 300 servers… [which] caused the entire system to crash.”

Many reports questioned whether Delta and other airlines have let their technology grow old and vulnerable. Southwest Airlines recently suffered a similar snarl, and technology outages have hit other carriers as well.

Bastian acknowledged that “our infrastructure is dated, no question,” with “legacy mainframes.” But, he said, “I don’t think that was the problem.”

He said Delta spends about $1 billion a year on technology and this year hired a new chief information officer from insurance giant AIG and a new head of technology infrastructure from Marriott, who were “pulling together the next level of investment.”

“This is an area that we know was in need of investment. We have been investing in it,” Bastian said. “It’s not an area that we’ve been dismissive of.”

He also acknowledged: “I knew there was more to be done to make certain we didn’t have vulnerabilities such as this, and we’re in the process of continuing to monitor and upgrade.”

“The fact that this is something we were unaware of is unacceptable,” Bastian said.

Delta on Wednesday said some flights were still being delayed or canceled due to crews stuck in the wrong place or exceeding federal caps on on-duty hours.

Delta said it was focusing on moving flights through its Atlanta hub, the largest in its operation, and also used its Delta Private Jets subsidiary to get 40 customers from Atlanta to their destinations.

Many exhausted passengers were still suffering the effects of flight cancellations.

Mariah Paden spent close to 11 hours waiting in the Seattle airport, trying to get to Atlanta, after her flight was canceled.

“It was just really chaotic,” Paden said Wednesday. “There were a lot of people that were really upset…. I’m going to try to plan my flights differently just so I can get away from errors like this.”

Reader Comments ...

Next Up in Business

After mauling, Delta tightens emotional support animal restrictions
After mauling, Delta tightens emotional support animal restrictions

In the wake of a horrific mauling of a Delta passenger by another traveler’s emotional support dog last year, Delta is tightening restrictions on emotional support animals in flight. To travel with an emotional support animal, starting March 1 Delta Air Lines will require a “confirmation of animal training” form signed by the...
Metro Atlanta, Georgia officials to present united front to nab Amazon
Metro Atlanta, Georgia officials to present united front to nab Amazon

Georgia’s full-fledged competition for Amazon’s second headquarters has entered a new and unpredictable phase now that it’s on the tech giant’s short list. But the state’s hunt for another major headquarters three years ago could provide valuable clues into its strategy for winning the project. Georgia’s long...
Kempner: Amazon, thanks for sort of picking Atlanta, but …
Kempner: Amazon, thanks for sort of picking Atlanta, but …

Amazon, it’s not that we’re ungrateful for Atlanta being one of your Top 20 choices for your second headquarters and 50,000 delicious jobs. We’ll hold off celebrating or feeling like we’ve gotten some kind of affirmation until we figure out if you just like us a good bit or really love us. Yes, we do think we are all that...
New Chick-fil-A opens in Vine City, part of revitalization efforts
New Chick-fil-A opens in Vine City, part of revitalization efforts

Chick-fil-A has opened a new restaurant in Vine City, as part of an ongoing effort to revitalize the Westside area of Atlanta. The new restaurant located on MLK Drive, has provided 90 jobs, seventy percent of which went to Westside residents and also includes a conference room for people to hold community meetings. “This restaurant serves as...
Atlanta survives the first cut in Amazon’s HQ2 sweepstakes
Atlanta survives the first cut in Amazon’s HQ2 sweepstakes

Amazon narrowed its list of potential communities for its massive second headquarters on Thursday to a final grouping that includes Atlanta and 19 others in the U.S. and Canada, setting off a battle royale for a project promising prestige and 50,000 high-paying jobs. Georgia and metro Atlanta leaders hailed the news, while noting much work remains...
More Stories