User:Alessia/data centers on fire: Difference between revisions

From XPUB & Lens-Based wiki
Line 5: Line 5:
<br>
<br>
(Strasbourg, France, March 2021)<br>
(Strasbourg, France, March 2021)<br>
A massive fire erupted at one of OVH's data centers, part of OVHcloud's network, which includes other 44 centers globally.
A massive fire has broken out in one of OVH's data centers, which is part of the OVHcloud network that includes other 44 centers around the world.<br>
The servers housed in these centers hosted around 4 million websites, including dedicated government platforms from authorities in France, the UK, Poland and the Ivory Coast. This event also affected part of the .FR web domain.<br>
SBG1 and SBG2 went up in flames. SBG2, a 500 m2 data center with five stores, and its contents were destroyed, while SBG1 was partially damaged. The building was eventually demolished. Two other buildings, SBG3 and SBG4, were not damaged. The different service branches affected by the fire were Web Cloud Universe (hosting of websites), Bare Metal Universe (high-performance servers for intensive computing, such as media encoding, gaming, corporate security), Hosted Private Cloud Universe (private cloud infrastructure) and Public Cloud Universe (OVH's cloud service).<br>
<br>
<br>
The servers housed in these centers hosted roughly 4 million websites, including specialised government platforms owned by agencies from France, Britain, Poland and the Ivory Coast. This event also affected a portion of the .FR web domain.
The exact cause of the fire is not yet known, but it appears to have originated from two UPS systems (devices that maintain server operation during power outages). One of these UPSs had gone through maintenance the day before the fire, during which some components were replaced.<br>
OVH nicely declined to comment on its fire safety protocol, but from the findings of the BEA (French Bureau for Investigation and Analysis of Industrial Risks) we can conclude that this may have been the main problem, and not just that. The BEA report pointed out problems such as the lack of an automatic fire extinguisher system in the building and delay in electrical cutoff which made it more difficult to stop the flames. Toxic fumes from lead acid batteries also worsened the situation.<br>
<br>
<br>
SBG1 and SBG2 went up in flames. SBG2, a five stores, 500m2 data centre, and its contents were destroyed, while SBG1 was partially damaged. The building was eventually demolished. Two other buildings, SBG3 and SBG4, weren't damaged.
Initial investigations suggested that a water leak onto an inverter may have started the fire. So, the absence of an automatic fire extinguisher system, the delayed power shutdown and the general design of the building design likely worsened the damage. However, the report still tell, curiously, that the exact cause of the fire remains uncertain. So it is still just hyphotesis.
The different service branches that were affected by the fire were Web Cloud Universe (websites hosting), Bare Metal Universe (high performance servers for intensive computing, like media encoding, gaming, corporate security), Hosted Private Cloud Universe (private cloud infrastructure) and Public Cloud Universe (cloud service by OVH).
How come it's only hyphotesis?<br>
<br>
<br>
The exact cause of the fire remains unknown, but it appears to have originated from two UPS systems (devices that maintain server operations during power outages), one of these UPSs had gone through maintenance the day before the fire, during which some components were replaced.
Determining the cause of the fire could have had enormous financial consequences for OVH, as more than 140 clients have sued for 10 milion euros in damages. So perhaps it's better to keep a little bit of mystery to keep the boat afloat.<br>
<br>
<br>
OVHcloud nicely declined to comment on its fire safety protocol, but from the  BEA-RI’s (France’s Bureau of Investigation and Analysis on industrial risks) findings we can tell maybe that was the problem, and not just that. The report from BEA highlighted issues such as the lack of an automatic fire extinguisher system in the building, and delay in electrical cutoff which made it more challenging to stop the flames. Toxic fumes from lead batteries also worsened the situation.<br>
OVH's confusion over the location of their servers has not helped the situation. Some companies believed that their data was backed up on the surviving servers, as written on their contracts, but this was not the case. There were also problems with the backup solutions, as some contracts falsely stated that the backup options were isolated from the server infrastructure. Great bad luck here.<br>
<br>
<br> ------<br>
Initial investigation suggested that a water leak onto an inverter may have started the fire. So, the absence of an automatic fire extinguisher system, the delayed electricity cutoff, and the general building design likely worsened the damage to the buldings. However, the report still tell, curiously, that the exact cause of the fire remains uncertain. So these remain just hyphotesis.<br>
Despite the suspicion that the water leak was the main reason for the fire in 2021, OVH continues to use water cooling for its servers. Eheh.
How come just hyphotesis?<br>
<br>
Strictly determining the cause of the fire could have had enormous financial consequences for OVH, given the ongoing legal actions from more than 140 clents, that asked for 10 milion euro in compensation, so maybe it's better to keep a little bit of mystery to keep the boat afloat. <br>
<br>
OVH's confusion regarding the location of their servers didn't improve their situation, leading some companies to believe their data was backed up on survived servers, as written on their contracts, when it was not. There were also issues with backup solutions, as some contracts inaccurately claimed that backup options were isolated from the server infrastructure. Great bad luck here.<br>
<br>
Despite suspicions that the water leak was the main issue involved in the 2021 fire, OVH continues to utilize water cooling for its servers. Eheh.<br>
<br>
<br>
I was reading this news article, https://www.theregister.com/2022/09/13/ovh_sbg5_opens/, titled "OVH opens less flammable datacenter at site of 2021 fire" they could have easily put one ;) emoji there too.<br>
I was reading this news article, https://www.theregister.com/2022/09/13/ovh_sbg5_opens/, titled "OVH opens less flammable datacenter at site of 2021 fire" they could have easily put one ;) emoji there too.<br>

Revision as of 14:12, 22 March 2024

fire-ace-free-trial.png


OVH Cloud Data center fire🔥


(Strasbourg, France, March 2021)
A massive fire has broken out in one of OVH's data centers, which is part of the OVHcloud network that includes other 44 centers around the world.
The servers housed in these centers hosted around 4 million websites, including dedicated government platforms from authorities in France, the UK, Poland and the Ivory Coast. This event also affected part of the .FR web domain.
SBG1 and SBG2 went up in flames. SBG2, a 500 m2 data center with five stores, and its contents were destroyed, while SBG1 was partially damaged. The building was eventually demolished. Two other buildings, SBG3 and SBG4, were not damaged. The different service branches affected by the fire were Web Cloud Universe (hosting of websites), Bare Metal Universe (high-performance servers for intensive computing, such as media encoding, gaming, corporate security), Hosted Private Cloud Universe (private cloud infrastructure) and Public Cloud Universe (OVH's cloud service).

The exact cause of the fire is not yet known, but it appears to have originated from two UPS systems (devices that maintain server operation during power outages). One of these UPSs had gone through maintenance the day before the fire, during which some components were replaced.
OVH nicely declined to comment on its fire safety protocol, but from the findings of the BEA (French Bureau for Investigation and Analysis of Industrial Risks) we can conclude that this may have been the main problem, and not just that. The BEA report pointed out problems such as the lack of an automatic fire extinguisher system in the building and delay in electrical cutoff which made it more difficult to stop the flames. Toxic fumes from lead acid batteries also worsened the situation.

Initial investigations suggested that a water leak onto an inverter may have started the fire. So, the absence of an automatic fire extinguisher system, the delayed power shutdown and the general design of the building design likely worsened the damage. However, the report still tell, curiously, that the exact cause of the fire remains uncertain. So it is still just hyphotesis. How come it's only hyphotesis?

Determining the cause of the fire could have had enormous financial consequences for OVH, as more than 140 clients have sued for 10 milion euros in damages. So perhaps it's better to keep a little bit of mystery to keep the boat afloat.

OVH's confusion over the location of their servers has not helped the situation. Some companies believed that their data was backed up on the surviving servers, as written on their contracts, but this was not the case. There were also problems with the backup solutions, as some contracts falsely stated that the backup options were isolated from the server infrastructure. Great bad luck here.

------
Despite the suspicion that the water leak was the main reason for the fire in 2021, OVH continues to use water cooling for its servers. Eheh.
I was reading this news article, https://www.theregister.com/2022/09/13/ovh_sbg5_opens/, titled "OVH opens less flammable datacenter at site of 2021 fire" they could have easily put one ;) emoji there too.

In 2022, OVH reconstructed their datacenter building over the SBG2 rubbles. In the new SBG5 they made some changes to improve safety tho. They separated the data center from the supporting infrastructure. Batteries are now stored in fire resistant containers outside the data center, along with rooms containing transformers.
The entire facility is built with fire-rated walls and surrounded by double fire barriers. Very Early Smoke Detection Apparatuses are installed in all internal spaces, with both sprinklers and inert gas available to suppress the fire.
That was a good start, mainly for the investors, as it was a good way to introduce a new cold storage service called Cold Archive, one of four mini cute data centers builded across France. This service offers a long term storage for "cold data", a cold cemetery.

OVH?

Its name stands for “on vous héberge”, “we host you” in french.
OVH is Europe's biggest hosting provider and the third largest worldwide in terms of physical servers. It mainly serves clients in Europe and the US, and opened subsidiaries in Singapore, Australia, India.

It was founded in 1999, by Octave Klaba and his family, in Roubaix, France.

I read this enlightening article on the rise of OVH and Klaba as an entrepreneur. Behind big companies there are always human beings anyway, right?
Klaba wears t-shirts and trainers, plays guitar, works himself to exhaustion, just like a normal person, except that he runs a company for 2.07 Billion dollars, isn't that great? It could be you.

He came from nowhere to build a great company that will be a European leader in cloud computing
😬😬😬
Klaba was so dedicated that he used to sleep next to his equipment some nights
😬😬😬

It's always admirable, the resilience, I don't know Klaba personally, from how he is portrayed he seems like a guy to have coffee with (Elon Musk, for example, I don't think is a guy to have coffee with). investing in other start-ups and setting up a new wind-turbine manufacturing company, money is getting greener as we already know.

There is definitely some 'punk' characteristic in OVH and Klaba, not as 'punk' as Elon Musk, but certainly punk enough to recognise him in the entrepreneur crowd.

Funny facts:

Octave Klaba made a bold move hosting a mirror website (replica) of Wikileaks after the US government took it down in 2010. This action obviously caused some controversies. With the French government he had some quarrels as well, as he stood up for civil and digital freedoms. He opposed a new French intelligence law, criticising features like algorithms designed to monitor internet traffic, as well as interception systems targeting individuals (to research more).

Interestingly, Klaba's email address appears in a GCHQ (British intelligence agency) interception test report, adding another layer to the punk narrative. 🤘🖤🎸 (in a test aimed at knowing whether the intercept of a satellite liaison between Sierra Leone and Belgium was technically possible, why tho?)

In 2013 OVH faced a data breach when some h a c k e r s infiltrated the company's servers through mail accounts. The investigation into this breach is still ongoing, could be a case of industrial espionage, as OVH stated. "we weren't cautious enough. We're now operating in a state of heightened caution" Klaba told the media.


Returning back to OVH,
OVH is a pioneer of free cooling, which uses water instead of air conditioning to cool processors down. This innovation has been the key for OVH's cost effective approach, helping them being competitive in the great sharks market dominated by giants as Google, Amazon and Microsoft. Quite interesting the new hybrid immersion cooling system, that uses thermally and not electrically conductive liquid, that claims to lower the carbon impact, use less space, being generally cheaper.
OVH markets itself as a green data center solution, relying on nuclear power, for example the Gravelines data center sits next to the Gravelines nuclear power station. During 2021 OVH joined the Climate Neutral Data Centre Pact, aiming for climate neutrality in data centers by 2030.

OVH also promotes itself as an independence solution for Europe.
Many european politicial nests would love to see OVHcloud succeeding as they see the still present US cloud dependency as a risk for the european sovereignty and economic competitiveness.
Klaba used these concerns to push for OVH as the favourite safe alternative, as its servers being outside the US are not subject to US search warrants and law enforcement, as well as intelligence agencies infiltrations, the latter could be a little bit weak as a claim.

...we believe in an open system that keeps the users, coders, developers, partners, administrators and operators in control of their data – to manage in the right way, in a responsible way. That’s how we build freedom


In 2023 OVH made a big move by introducing a quantum computer named MosaiQ at its data center in Croix, France. It is developed by Quandela and it is the first of its kind housed in a data center. Unlike other quantum computers, MosaiQ's design uses light beams and photonics principles, removing the need for extremely low temperatures (might research a bit more on this).
IBM runs several quantum machines from a data center in New York, accessible through a specific portal. Many other quantum computing firms offer access to their systems via their own cloud platforms, although most of these systems are still kept in labs or data centers.


sources:
https://www.datacenterdynamics.com/en/news/ovh-launches-quantum-computer-at-data-center-in-france/
https://www.ft.com/content/c78ee22d-ceb3-4a31-aae4-96589bb5cddf
https://www.lemonde.fr/en/archives/article/2022/03/14/britain-spied-on-the-ceo-of-europe-s-biggest-internet-hosting-company_5978491_113.html#
https://www.ovhcloud.com/en/lp/status-services-backup-strasbourg/
https://www.datacenterknowledge.com/industry-perspectives/increasing-viability-sustainable-fire-protection-data-centers#close-modal
https://www.theregister.com/2022/06/10/ovhcloud_datacenter_fire_last_year/
https://pa-schembri.medium.com/an-opiniated-review-of-the-ovh-cloud-archive-service-f0b2d615c5ca https://blog.ovhcloud.com/new-hybrid-immersion-liquid-cooling-developments-at-ovhcloud/
https://www.climateneutraldatacentre.net/