// //

Thursday, August 24, 2017 - Welcome, guest user!

Datacenter flood; JSF running on generator power; Possibility of outage exists.

Tuesday, October 3, 2006 by  
Filed under Daily Blog

October
3
2006

Datacenter flood/evacuation: Updated October 3, 7:45 AM EDT.

Yesterday afternoon the datacenter in downtown Orlando where the JSF servers are located was evacuated due to some boneheaded construction worker who broke a massive water pipe on the 15th floor. The pipe break caused the sprinkler system to go off in the entire building. Between the broken pipe and the sprinklers, a massive amount of water was released. There was a lot of water coming down the elevator shaft and all the equipment closets. Thankfully my equipment and most of the critical equipment in the datacenter was not damaged by the sprinklers….

The authorities have cut the power to the entire building and ordered everyone out. It may be days before power is restored. Yesterday afternoon (October 2) the fire department forced the datacenter to turn off the emergency generators, resulting in a complete outage starting about 5:30 PM EDT. Thankfully around 7:00 PM last night permission was given to turn the generators back on. JSF has been up and running on generator power since then. The possibility still exists that the authorities will force the datacenter to turn off the generators again today as cleanup and repair work continues.

All JSF data is backed up and safely off-site, and I have fully redundant hardware off-site. In an effort to further minimize any possible data loss, I will be doing backups twice daily instead of once daily until this problem has been resolved. Please be aware that the I might have to shut everything down at a moment’s notice, and there could be extended downtime. I’m pleased that so far we’ve only been down for about an hour and a half – fairly impressive considering what happened! I hope by the end of the day things stabilize and power is restored. I will post updates to this thread. Thanks for your patience.

Speak Your Mind

Tell us what you're thinking...

You must be logged in to post a comment. Not yet a member? Registration is fast and free!