Lessons learned from the computer glitch that crashed Nissan

A critical Nissan Group data center in Denver crashed Aug. 17 apparently because of a power outage. The results were widespread and frustrating for the automaker and its retailers last week, sending customers away empty-handed and interrupting factory production — as Nissan Group is doggedly trying to shore up falling U.S. retail sales.

But the crisis also demonstrates a larger vulnerability for today's auto industry, which depends on complex digital vehicle distribution systems that link data and commerce among consumers, retailers, distribution networks, manufacturing plants and finance companies.

All of that shut down for Nissan Group last week, affecting the operations of Nissan and Infiniti's approximately 1,300 U.S. dealers as well as an undetermined number of retailers in Canada and Mexico.

A Nissan Canada spokesman wasn't specific in saying how the outage affected Canadian operations, just that it did.

It was unclear at week's end how many new-vehicle sales Nissan Group lost from the glitch or what the ultimate cost will be to dealers, sales personnel and service shops.

The system, referred to internally as NNANet, is the retailer's tool for ordering cars and parts, obtaining product rebate information to know how to structure a sale, checking on vehicle recalls, filing warranty claims to enable service work to be performed and priced, and seeking factory financing information.

"Everything we do with Nissan goes through NNANet," said Tim Hill, owner of Hill Nissan in Winter Haven, Fla. "That is our lifeblood."

The system remained down for four days, grounding operations at many retailers. Service was restored by 6 p.m. Central Daylight Time on Wednesday, Aug. 21.

The data crash also brought down production at Nissan's factories in Smyrna, Tenn., and Canton, Miss., according to the company.

Nissan spokesman Chris Keeffe did not estimate lost sales or production. Nissan also declined to elaborate on how a power outage at a data center could take down such a mission-critical network and disable the company for several days.

Dealers were surprised the shutdown was so widespread and lengthy. Some told Automotive News they suspected the automaker's linchpin information technology system had been the target of a cyberattack. Keeffe said it was not a cyberattack.

"Our data center in Denver experienced a power failure, disabling our backup systems and prolonging the systems outage," Keeffe said.

He did not say why the power outage also took down the backup systems nor provide details about Nissan's business recovery plan.

Extreme heat in Colorado led to the outages in the Denver area.

Nissan's lengthy recovery period is instructive for automakers.

Businesses should have backups of their data, applications and systems in different places across the country, said Armistead Whitney, CEO of Apptega, an Atlanta company that helps companies manage IT risks.

"That way, when one facility, area or region experiences a major disruption, the chances are extremely low that another geography is impacted, resulting in the disruption only lasting minutes or a couple of hours instead of days," Whitney said.

The consultant advises against co-locating communication systems such as phones and email with core business applications.

"When a major business system goes down, it also brings down the communications system, which compounds the outrage of the outage by not being able to provide timely updates to colleagues and customers."

Nissan's disruption should have been anticipated, said Ray Brandt, CEO of Ray Brandt Automotive Group in suburban New Orleans.

"We were surprised that it happened without a real backup plan to make sure that if it did happen, we could minimize the pain," Brandt said. "I'm surprised it took four days to get it resolved."

During the NNANet shutdown, Nissan and Infiniti stores were unable to order new vehicles from Nissan, report vehicles that have been sold, place online part orders or process warranty claims.

Customers complained on social media about being unable to get recall information, having service appointments rescheduled and, for one customer, being unable to have an ignition key reprogrammed.

The service disruption occurred at a particularly bad time: Dealers typically book most of their sales toward the end of the month.

"This is the crunch time of our month," Brandt said. "We've got to make big sales. We have big objectives and money on the table. We were penalized. There's no doubt about it."

Brandt anticipates the outage cost his dealership about 10 percent of its sales for the month as customers had to be turned away because the store could not access financial information from Nissan and its finance arm. "It's a very impatient world," Brandt said.

Nissan said it expects the service outage to dampen August sales.

The company told retailers late last week it will trim August dealer sales goals 10 percent "to reflect the business disruption caused by the Nissan computer network system outage over the past few days."

One New Jersey dealer said the service disruption affected "literally every aspect of the business."

During the outage, the dealership focused on selling used cars and got parts from other dealers.

The disruption also affected Nissan's financial arm, Nissan Motor Acceptance Corp. That meant dealers could not check customer payoff information and customers could not make monthly car payments.

The New Jersey dealer said, "I had a gentleman come in and we couldn't get his payoffs, so we had to turn him away."