The Night the Lights Went Out

Caroline McDonald

|

April 1, 2015

super bowl risk management

During Super Bowl XLVII in February 2013, the Baltimore Ravens were leading the San Francisco 49ers 28-6 with 13:22 left in the third quarter when half of the lights in New Orleans’ 73,000-seat Superdome went out. The biggest game of the year was left partially shrouded in darkness.

While auxiliary power kept the playing field from going totally dark, escalators stopped, credit card machines shut down, and the concourses were only illuminated by small banks of emergency lights.

Randy Philipson, regional director of operations and New Orleans director for international building management company SMG, remembers that night all too well. It started when their utility company, Entergy, decided to replace the lines to its nearby electrical vault mere weeks before the Super Bowl. They requested that the Superdome also replace its two lines leading to the vault. “So we replaced our feeder cables, an A feed and a B feed—one feeds the east side of the building and one feeds the west side,” he explained.

Once that was done, Philipson decided that he wanted to know exactly what was on each feed, so he scheduled a shutdown test in early January. “We killed the A feed to the west side and we documented everything that was off,” he said. “We checked the emergency lighting and which elevators were affected. We brought that back up and then did the same thing with the B feed to the east side of the stadium.”

After rigorously testing both feeds, the conclusion was that “we were nowhere near being at risk of failure,” Philipson said. Nevertheless, they wrote a protocol detailing what would be done if power was lost during the Super Bowl, with scenarios for losing either feed. “We actually created the scenario that ended up happening, which helped us to respond accordingly,” he said.

Philipson then went to Entergy and asked for extra staff in the vault during the Super Bowl, just in case something went wrong. “They staffed people in the vault and had a copy of my protocol, and we communicated with them throughout the night,” he said.

Because halftime performer Beyoncé wanted to shut off all the stadium lights as part of her performance and all of the backlit ad panels had to be controlled individually, 15 to 20 extra electricians were positioned throughout the building to throw the switch for each sign. “We also decided to staff every transformer room in the building to respond in case of a power failure—another 15 to 20 electricians,” he said. “So there were more than 40 in the building.” Every electrician in every position was listed in the protocol, which also noted who would call whom in the event of an emergency.
Check out "In the Event of an Emergency" for more about how big events require hospitality professionals to become risk managers.

It was all running smoothly when Beyoncé left the field. “Everything went back on and we were feeling good,” Philipson said. “I was just walking off the field, into the tunnel closest to our vault, and the lights went out. I heard it and saw it. I took off for our vault to talk to the guys stationed there and find out what happened.”

They discovered that the failure only impacted the A-feed, which powered the entire west side of the building. The east side was fine. Entergy investigated the cause and eventually determined that the problem was on the company’s end. In the meantime, all the electricians around the building were instructed to power down so as not to blow the equipment when the electricity came back on. Within 15 minutes, power was restored.

Philipson said the worst part for him was that “as soon as the power went out, Entergy sent out a tweet from their Arkansas office saying the issue was on the user’s end—meaning us—and they never retracted that statement.” The following week, however, Entergy admitted the cause was a faulty piece of equipment in their vault that was not set properly. “That piece has since been taken out of commission,” he said. “It wasn’t our fault, but it became our problem. There was no sleep for a few days, as you can imagine.”

Ultimately, what saved the game was the contingency plan. “We had a pre-written script for the announcer in case anything like this happened,” Philipson said. “We also had planned messages for the message board and our PA was on emergency power. One message read ‘We have suffered a temporary loss of power, please remain in your seats,’ because the last thing we wanted was pandemonium and people exiting the building.” Because of that forethought, people listened and the majority stayed until the end of the game. Most importantly, they did not panic.

“Our plan worked the way it was designed, which is why we do it,” Philipson said. “If we had not written the protocols based on the shutdowns that we did, there is no telling how long we would’ve been down. Quite honestly, I don’t know if we would have finished the game. I can’t emphasize enough the need to think about those things.”

Even considering all the dire possibilities, the power outage was brief and play was only interrupted for 34 minutes, thanks to the efforts of Philipson and his team. “Nobody wants to be the guy who is in charge when the lights go out at the Super Bowl,” he said. “Of course, now I have that lucky distinction.”