Expect some downtime
Tuesday, October 30, 2012 at 3:41PM Expect Lost In Mobile to be offline for a while. The email from Squarespace below is an example of how to deal with such problems.
Dear Customer,
I have some unfortunate news to share. Our primary data center, Peer1, in Lower Manhattan lost power yesterday at about 4:30PM local time. At that time, we smoothly made the transition to generator power and took comfort over the fact that we had enough fuel to last three to four days. (Peer1 stayed online during the last 3 major natural disasters in the area, including a blackout that lasted for days.)At 8:30PM yesterday, we received reports that the lobby in the data center's building was beginning to take on water. By 10:30PM, as is sadly the case in most of Lower Manhattan, Peer1's basement had experienced serious flooding. At 5AM, we learned our data center's fuel pumps and fuel tanks were completely flooded and unable to deliver any more fuel. At 8AM, they reported that the generators would be able to run for a maximum of four more hours.
Unfortunately, this means that Squarespace will be offline soon (our estimate being at 10:45 AM today). Be assured that while this will impact our availability, there is no chance of data loss or any other permanent effects. We have simply run out of power, backup power, and cannot access our fuel in a flooded basement.
Our teams have been working tirelessly on contingency plans. We are working to bring the Squarespace systems back online as soon as possible. As you have probably read, all bridges and tunnels into and out of Manhattan are closed and large portions of the city remain without power. We will do everything in our power to get Squarespace running as soon as possible, and we will remain online for as long as it is safe.
Squarespace support will remain available 24/7 during this downtime. We will post updates to Twitter via @squarespacehelp and urge you to follow us there. Updates will also be posted to blog.squarespace.com.
Our hearts go out to the many people who have lost their lives in this terrible tragedy and also to those who continue to suffer through the consequences of this historic storm.
Thank you for your understanding,
Anthony Casalena
Founder & CEO
All News
Reader Comments (6)
Communication is key. Very impressive and I hope all gets resolved soon.
Maybe LIM can continue while offline via twitter?
Excellent statement of events. Everything's explained and it provides a source for updates.
As I read this, I couldn't help but wonder why they didn't have a backup server somewhere else. Possibly providing reduced services but still operating. Or maybe a deal with another company.
OK, severe weather conditions will keep rising around the globe and it will have some nasty quakes and other activities altogether... I guess people are starting to be aware of it now. Lots of world services are based on US ground, so I just think something bad strikes USA the world will be shot down?... have you seen recently released videos of Google's facilities? They are spread in USA and Europe and that's the way it should be...
Then again, we shouldn't really be worried... isn't everything in the cloud? How can it be affected by floods and quakes? (just being ironic)
They've handled it well with the communication, but like Bob I can't help wondering why they had all their eggs in one basket.
I worked for a fairly small company but even we had our servers spread across two different data centres. Admittedly they were both in the London area, but opposite sides of it.
Rui I know what you mean. Unfortunately the term 'Cloud' has been hijacked by marketers to mean pretty much anything they want to sell. It basically just means 'on the web' now. Unless you're looking to move your servers to The Cloud, in which case it'll probably mean a bunch of virtual machines running on blade servers. Same thing we've been doing for years but they've jumped on the Cloud band-wagon.
Can you tell this is a pet peeve? ;o)
My approach is to have mirror servers in three separate data centres (or "availability zones" in Amazon EC2 speak). A nightly snapshot of each is taken, which gives us the ability to simply bring one up with the same IP/DNS in another zone within a short period of time. We can also upgrade and downgrade the capabilities on the fly.
This "cloud" model is a revelation. No contracts, limitless power and bandwidth. Low costs. Massive flexibility. There's a big difference between the old shared hosting model, dedicated servers and the new cloud ethos - it's revolutionising the ability of small firms like mine to deliver big capacity and scalability without big investment.
I was with Rackspace for a few years. Nice servers, but it took a few weeks of negotiation to set the price, followed by an 18 month contract and a fixed capacity box which we soon outgrew. Now, I can add as many servers as I wish for just a few hours, if needed. I was able to expand computing power and bandwidth for the Olympics (LOCOG was a client) and downgrade at the end.
It's a brave new world.
Nevertheless, I would agree that the term cloud is now analogous to web services in most people's eyes.
Yes that's the type of Cloud I was most interested in (using Azure as it's a DotNet system).
Unfortunately because Commensus calls its bunch-of-virtual-machines Cloud the Finance Director thought 'box ticked'. To be fair I would probably have been able to get us onto Azure if I'd stayed there, but the hijacking of the Cloud had made it a lot more difficult to get across what real Cloud computing is.
It's not about adding more virtual machines when you get busy and then shutting them down. Having an entire instance of Windows running just to host a single web service seems incredibly wasteful; true Cloud computing doesn't have that waste.
I should mention that Commensus' bunch-of-virtual-machines seems pretty damned good as these things go; I was tempted to go that route rather than Azure as it has some distinct advantages (e.g. up-time, security, manageability).