Just before 5 p.m. on Monday, the team behind New York's Fog Creek Software announced in a blog post that its services would be online during Hurricane Sandy, despite the fact that the company's main data center was located in an evacuation zone in the low-lying Financial District.
"Given the preparation work that's gone into this, we are confident that all of our services will remain available to our customers throughout the weather," systems administrator Bradford Ley wrote. "Consider this the 'Everything is Perfectly Fine Alarm.'"
The storm hit. The power went out. Fog Creek’s data center, Peer1, switched to an emergency diesel generator on the 17th floor, just as planned. Fog Creek's bug tracking system, FogBugz, remained online. So did its code review service Kiln, productivity tool Trello, and sister site Stack Exchange.
But around 9 a.m. the next morning, there came an urgent phone call
But around 9 a.m. the next morning, there came an urgent phone call. Water had surged into the basement and lobby of the data center, immersing the 20,000-gallon tank of diesel fuel and wiping out the fuel pumps and monitoring equipment. There was no way to know how much fuel was left, but Peer1 estimated it at about three hours of power.
The Fog Creek family went into crisis mode. The Trello team hustled to get its service transferred to Amazon's cloud hosting within 10 hours. Fog Creek had to locate one of its systems administrators, who was incommunicado. An employee trekked to the apartment where his coworker was relaxing without power or cell service, climbed 13 flights to surprise the administrator at his door, and yanked him back to reality.
Then, employees picked up a few five-gallon buckets normally used to hold supplies for the office fish, purchased drums of diesel fuel from gas stations around the city, and headed down to their darkened, flooded data center.
"It was like a war zone," said Michael Pryor, the co-founder of Fog Creek and Stack Exchange. "Almost no one around, but cars upside down and floating out of parking garages, plus the streets smelled of diesel from all the generators and leaking fuel tanks. It was pitch-black as we walked into the lobby and stairwell of the building which were slippery with a mixture of seawater and diesel fuel. Definitely a little bit scary, like the zombie apocalypse."
The employees joined forces with a crew from fellow New York startup Squarespace, which also had its servers at Peer1. The troops filled the five-gallon buckets with diesel fuel, carefully navigated over the diesel-and-saltwater- slicked floor, and lugged their cargo up 17 flights to the generator and poured the fuel in. By 2 a.m., the tank was half-full. By 11:30 a.m., it was full.
Off-line
Similar situations played out across the city. Another Financial District data center, Datagram, flooded and took down BuzzFeed, Huffington Post, Gawker, Bloomberg News, Daily Kos, and the United Nations. HuffPo's backup data center, which was in Newark, was also taken out by the storm. All sites are back up with the exception of Gawker, which is still publishing on its Tumblr-powered backup site. Tumblr, also based in New York, suffered a data center failure mere days before the storm. But its two data centers, in undisclosed locations, appeared to carry it through Sandy.
Livestream's data center was at 111 Eighth Avenue, in the building owned by Google. The backup generators were humming and expected to last about 85 hours, until the fuel pumps failed and took the whole data center down. After a long day, engineers returned the site to the internet this afternoon.
Shapeways, the 3D printing marketplace, was also offline during the storm when its data center in the Financial District lost power. As an ecommerce startup, it’s also facing another kind of disruption: no shipments. The company is still scrambling to get back online.
Why put a data center in New York City?
It’s common for internet companies to build data centers in safe places, with lots of open space to protect privacy, and a local government eager to subsidize construction. Facebook built its first data center in Prineville, Oregon, a middle-of-nowhere desert town with a predictable climate. Google, Intel, and Amazon have also built data centers in Oregon towns with miniature populations, taking advantage of the remoteness as well as the cheap Northwest hydropower. Small towns in North Carolina and Virginia have also been attracting data centers, like Apple’s massive iCloud facility in Maiden, North Carolina.
But who knew so many startups had their data centers in New York, where expensive real estate is usually overstuffed with people, not wires?
There’s an assumption that when you host in the cloud, the datacenter is well above sea level.
— Jeremiad Lee (@JeremiadLee) October 30, 2012
In fact, there are several major data centers in Manhattan and a huge industry of them in nearby New Jersey. Their presence is largely encouraged by financial services companies seeking the speediest information transfers possible, but internet companies value proximity, too.
The offices of Fog Creek and Stack Exchange are just a few blocks from their data center, making it easy to personally perform maintenance and upgrades (and assist personally in the case of a power failure). Stack Exchange has a second data center in Oregon, but only because an employee lives there. Despite the higher cost of keeping a data center in New York, and the devastation wrought after this storm, the companies are happy with their Lower Manhattan data center and have no plans to relocate.
Online, but displaced
After the storm knocked out power for homes and office buildings, New York's tech workers are clustering in pop-up offices all around the city. They’ve gathered at coworkers' homes, in coffee shops, and at the offices of their fellow startups.
Foursquare's Dennis Crowley immediately started coordinating with fellow refugees Etsy, Tumblr, and Kickstarter in order to find places for their employees to work. Startups coordinated through various email lists in order to find temporary internet and power connections. Local venture capitalist Charlie O'Donnell started a hashtag on Twitter, #sandycoworking, to encourage this teamwork, which was converted to a crowdsourced map.
The idea of losing whole days of productivity is devastating to most startups
"We made it through just fine," said Boxee's Avner Ronen, whose office was miraculously unaffected even though it's located just above the dead zone at West 26th Street and surrounded by buildings without electricity. Boxee is now hosting employees from Kickstarter, Brewster and Workmarket. "Power never went out at the office and most of our employees came the day of and the day after." This was especially lucky given that Boxee plans to launch its new Boxee TV tomorrow, he said.
The idea of losing whole days of productivity is devastating to most young startups, where working overtime is the default.
"I emailed the team and said we'd have the company pay for taxi cab rides to locations with power and internet so that they could get productive again," Brian Wang, CEO of fitness-tracking game Fitocracy, told The Verge from his temporary perch in the office of fellow New York startup LocalResponse. "For us, the stakes are high. We're working on a big update to our iOS app and also to ship our Android app for the holidays. Getting everything done with multiple platform launches requires uninterrupted focus."
Fear of analog
All around New York City, people are struggling to stay connected: to the grid, to the internet, and just to sources of information. Lines of displaced residents stretch outside of payphones in the East Village, while people huddle in ATM vestibules and around trees that have been wired with string lights, plugging into any outlet they can find. Others headed to Tekserve, which opened today without electricity, to pick up Mophie battery packs, surge protectors, power supplies, and flashlights.
The crowding around the outlets at Starbucks looks like the withdrawal symptoms of a hyperconnected urban citizenry, but the scene lacks the panic felt by workers in the internet industry.
"I think those images seem hyperbolic, like ‘desperate’ folks wanting to check on their virtual farms," said Joe West, a Brooklyn-based entrepreneur working on an unnamed startup that is redesigning email. "But likely they just want to connect with friends and family and let people know they're okay or vice versa."