LON3 Outage
This morning, at approximately 09.19, our LON3 Data Centre experienced power interruptions causing downtime for a portion of our customers. These interruptions were the result of a power issue caused by a UPS module failure on UPS B. The UPS then failed to transfer the load as expected, Chloride engineers have been notified and we will launch an investigation into what caused the UPS issues.
In the meantime our engineering team have isolated the module, power has been restored and DCops are in the process of systematically bringing back up all of the affected servers.
We sincerely apologize to our customers and those who were affected by this downtime. We are working tirelessly to determine exactly what the issues were and how we can address them. Thank you for your support and we will endeavour to keep you informed.
* *Update 11:31 a.m.**
Power has been restored and we are in the process of systematically bringing back up all of the affected servers.
** Update 4.30 p.m.**
Further to the Outage in LON3 experienced today, here is more information on the incident and actions taken.
Following on from the brief outage this morning we have continued to bring any affected devices online. Although the majority of devices were down for a minimal amount of time we had particular issues with a group of about 220 Whiteboxes which have taken several hours to bring up. DCOps have had to manually intervene to bring the servers back online. In numerous instances they had to replace power supplies in servers, replace firewalls, reconfigure switches and to log on to serves to get them to boot properly.
We can only apologise for this incident. We take this type of incident extremely seriously and will work to fix the root cause.
A full incident report will follow, pending a thorough evaluation. In the meantime your account management team is available to help.
With respect rackspace, this is what you told me at 9.30 this morning. ‘Power restored, just booting up now’. Bad communications all round guys, a very unhappy bunny here.
Hi Matt
Thanks for your feedback we will take it on board as part of our evaluation of the incident, you are right power was quickly restored and most customers were back up in a short space of time. Unfortunately a minority of those customers affected have had networking and server issues as a result of the power issue and we are working at getting those restored as soon as possible.
Will you be contacting customers to inform them of when their servers are back online, or with an ETA of when this will happen? We’ve got a lot of unhappy customers demanding to know when their websites and email are back up.
Hi Nick, We’d like to get your account manager to give you a call to let you know the process, but as you responded via the blog we don’t know who you are. Can you either get in touch with your AM or let us know your customer number and we’ll get them to call you.