User:Alessia/data centers on fire
Your data is burning!
OVH Cloud Datacenter Fire
(Strasbourg, France, March 2021)
A massive fire erupted at one of OVH's data centers, part of OVHcloud's network, which includes other 17 centers in France and 32 globally.
The servers housed in these centers hosted roughly 4 million websites, including specialised government platforms belonging to agencies from France, Britain, Poland and the Ivory Coast. This event also affected a portion of the .FR web domain.
SBG1 and SBG2 went up in flames. SBG2, a five-storey, 500m2 data centre, and its contents were destroyed, while SBG1 was partially damaged. The building was eventually demolished. Two other buildings, SBG3 and SBG4, weren't damaged.
The different service branches that were affected by the fire were Web Cloud Universe (websites hosting), Bare Metal Universe (high performance servers for intensive computing, like media encoding, gaming, corporate security), Hosted Private Cloud Universe (private cloud infrastructure) and Public Cloud Universe (cloud service by OVH).
The exact cause of the fire remains unknown, but it appears to have originated from two UPS systems (devices that maintain server operations during power outages), one of these UPSs had gone through maintenance the day before the fire, during which some components were replaced.
OVHcloud nicely declined to comment on its fire safety protocol, but from the BEA-RI’s (France’s Bureau of Investigation and Analysis on industrial risks) findings we can tell maybe that was the problem, and not just that. The report from BEA highlighted issues such as the lack of an automatic fire extinguisher system in the building, and delay in electrical cutoff which made it more challenging to stop the flames. Toxic fumes from lead batteries also worsened the situation.
Initial investigation suggested that a water leak onto an inverter may have started the fire. So, the absence of an automatic fire extinguisher system, the delayed electricity cutoff, and the general building design likely worsened the damage to the buldings. However, the report still tell, curiously, that the exact cause of the fire remains uncertain. So these remain just hyphotesis.
How come just hyphotesis?
Strictly determining the cause of the fire could have had enormous financial consequences for OVH, given the ongoing legal actions from more than 140 clents, that asked for 10 milion euro in compensation, customers, so maybe it's better to keep a little bit of mystery to keep the boat afloat.
OVH's confusion regarding the location of their servers didn't improve their situation, leading some companies to believe their data was backed up on survived servers, as written on their contracts, when it was not. There were also issues with backup solutions, as some contracts inaccurately claimed that backup options were isolated from the server infrastructure. Great unluck here.
Despite suspicions that the water leak was the main issue involved in the 2021 fire, OVH continues to utilize water cooling for its servers. Eheh.
I was reading this news article, https://www.theregister.com/2022/09/13/ovh_sbg5_opens/, titled "OVH opens less flammable datacenter at site of 2021 fire" they could have easily put one ;) emoji there as well.
In 2022, OVH reconstructed their datacenter building over the SBG2 rubbles. In the new SBG5 they made significant changes to improve safety tho. They separated the datacenter from the supporting infrastructure. Batteries are now housed in fire-resistant containers outside the datacenter, along with rooms containing transformers.
The entire facility is built with fire-rated walls and surrounded by double fire barriers. Very Early Smoke Detection Apparatuses are installed in all internal spaces, with both sprinklers and inert gas available to suppress the fire.
That was a good start, mainly for the investors, as it was a good way to introduce a new cold storage service called Cold Archive, one of four mini cute data centers builded across France. This service offers a long term storage for "cold data", a cold cemetery.