Lessons From How The Planet Communicated Their Data Center Explosion
InGameNow has been down since Saturday… So have thousands of other sites as 9,000 servers were caught in a massive fire at server and hosting company The Planet’s Houston, Texas data center. Now the good news is that no one was injured and that 6,000 of those servers were restored last night at 5:06pm CDT. The bad news: InGameNow is still down.
There are numerous lessons to be learned in this ordeal (one of which is the strain this causes on a start up, it’s users and the engineers), but I am particularly struck by The Planet’s responses - which truthfully has been mixed.
I often write about the importance of transparency. For customers, The Planet has been very transparent and responsive. They have sent out numerous mailings (see below), are reachable via phone, and in Twitter-esque style, are updating a blog / forum each hour with “Data Center Status Updates”. Considering the severity of the issue, I applaud the type of regular communication and frankly, it probably also relieves some of their CS burden.
That transparency, however, isn’t reflected to new customers - and that’s disturbing. There is no mention of any issues in either “Recent News” or “Recent Blog Posts” - in fact, their main blog is linked to directly on the homepage and hasn’t been updated since May 28th (last post: Hello, World). And much of the page is dedicated to “Learning more about World Class Data Centers”. Considering the circumstances, perhaps part of the page should be dedicated to support or at least linking to the Status Blog.
The part that really irks me is the default pop-up screen that asks you if you’d like to interact with Live Sales Representatives… I am tempted to type in “When will InGameNow.com be running?”
Update: 11:05am pst
InGameNow is now up… although, as you can tell, it is running slowly and graphics aren’t rendering as the necessary fixes we have implemented will take a few hours to propagate.
The Planet’s Initial Email:
> Dear Valued Customers:
> This evening at 4:55 in our H1 data center, electrical gear shorted,
> creating an explosion and fire that knocked down three walls
> surrounding our electrical equipment room Thankfully, no one was
> injured. In addition, no customer servers were damaged or lost.
>
> We have just been allowed into the building to physically inspect
> the damage. Early indications are that the short was in a high-
> volume wire conduit. We were not allowed to activate our backup
> generator plan based on instructions from the fire department.
>
> This is a significant outage, impacting approximately 9,000 servers
> and 7,500 customers. All members of our support team are in, and
> all vendors who supply us with data center equipment are on site.
> Our initial assessment, although early, points to being able to have
> some service restored by mid-afternoon on Sunday. Rest assured we
> are working around the clock.
>
> We are in the process of communicating with all affected customers.
> we are planning to post updates every hour via our forum and in our
> customer portal. Our interactive voice response system is updating
> customers as well.
>
> There is no impact in any of our other five data centers.
>
> I am sorry that this accident has occurred and apologize for the
> impact.
>
> Sincerely,
>
> Douglas J. Erwin
> Chairman & Chief Executive Officer
You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
June 3rd, 2008 at 9:51 am
I am a Hostgator client, who uses the planet for their server hosting as well. I have been crushed by the fire as my sites were down, then up, now back down again.
I am at a level of frustration that has made me almost giddy because I’m at the point where there’s nothing I can do but laugh.
What a complete and utter joke.
June 3rd, 2008 at 9:20 pm
[…] Lessons From How The Planet Communicated Their Data Center Explosion :: Ryan Spoon – an excellent post on how companies can handle web disasters the right way instead of trying to hide away. […]
June 4th, 2008 at 2:02 pm
I don’t see how you could expect TP to turn around and publicize to their prospective clients that one of their datacenters blew up? It does not affect new clients because they get hosted on a different DC so why would they be pushing the issue to them? I could understand if they hid it every which way they could, but their forum IS open to the public and the thread is completely accessible to anyone that would want to do some of their own research to find out more about TP.
In my opinion, ThePlanet handled the issue quite well.
June 7th, 2008 at 7:36 am
The planet told everyone to go to the forums, but when you ask questions you get no answers and if you ask what some feel are the wrong questions one of the planets managers calling customers trolls. When I posted telling this manager it was sad to see a company manager calling customers trolls as anyone that has spent any time on the Internet knows calling someone a troll is getting close to using the “N” word with someone.
After I posted that it was sad, he banned me from posting anymore comments. I have yet to receive any contact from his bosses (I sent copies of the comment to all of them) so I can only assume that they are aware or feel the same way about the customer.
Folks systems fail, it is just how it is. But to then have the company call it’s customers trolls for asking about info related to the service that company is being paid to provide is just too much for me. I have moved my server and I know many who once they get the rest of their data from their servers will be gone as well. I feel the planet should know that for some of these folks the last straw was your manager calling the customers trolls.
In years to come when things are calmed down what will be remembered about this whole incident is how at a time of crisis the customer where called TROLLS by management