Revision as of 16:57, 23 March 2024

OVH Cloud Data center fire🔥

(Strasbourg, France, March 2021)
A massive fire has broken out in one of OVH's data centers, which is part of the OVHcloud network that includes other 44 centers around the world.
The servers housed in these centers hosted around 4 million websites, including dedicated government platforms from authorities in France, the UK, Poland and the Ivory Coast. This event also affected part of the .FR web domain.
SBG1 and SBG2 went up in flames. SBG2, a 500 m2 data center with five stores, and its contents were destroyed, while SBG1 was partially damaged. The building was eventually demolished. Two other buildings, SBG3 and SBG4, were not damaged. The different service branches affected by the fire were Web Cloud Universe (hosting of websites), Bare Metal Universe (high-performance servers for intensive computing, such as media encoding, gaming, corporate security), Hosted Private Cloud Universe (private cloud infrastructure) and Public Cloud Universe (OVH's cloud service).

The exact cause of the fire is not yet known, but it appears to have originated from two UPS systems (devices that maintain server operation during power outages). One of these UPSs had gone through maintenance the day before the fire, during which some components were replaced.
OVH nicely declined to comment on its fire safety protocol, but from the findings of the BEA (French Bureau for Investigation and Analysis of Industrial Risks) we can conclude that this may have been the main problem, and not just that. The BEA report pointed out problems such as the lack of an automatic fire extinguisher system in the building and delay in electrical cutoff which made it more difficult to stop the flames. Toxic fumes from lead acid batteries also worsened the situation.

Initial investigations suggested that a water leak onto an inverter may have started the fire. So, the absence of an automatic fire extinguisher system, the delayed power shutdown and the general design of the building design (wooden floors) likely worsened the damage. However, the report still tell, curiously, that the exact cause of the fire remains uncertain. So it is still just hyphotesis. How come it's only hyphotesis?

Determining the cause of the fire could have had enormous financial consequences for OVH, as more than 140 clients have sued for 10 milion euros in damages. So perhaps it's better to keep a little bit of mystery to keep the boat afloat.

OVH's confusion over the location of their servers has not helped the situation. Some companies believed that their data was backed up on the surviving servers, as written on their contracts, but this was not the case. There were also problems with the backup solutions, as some contracts falsely stated that the backup options were isolated from the server infrastructure. Great bad luck here.

Despite the suspicion that the water leak was the main cause of the fire in 2021, OVH continues to use water cooling for its servers. Eheh.

I read this news article, https://www.theregister.com/2022/09/13/ovh_sbg5_opens/, titled "OVH opens less flammable data center at site of 2021 fire" they could have just put a ;) emoji there too.

In 2022, OVH rebuilt its data center over the rubbles of SBG2. However, in the new SBG5, some changes were made to improve safety. They have separated the data center from the supporting infrastructure. The batteries are now stored in fire resistant containers outside the data center, as are the rooms containing the transformers.

It was a good start, especially for the investors, as it was a good opportunity to introduce a new cold storage service called Cold Archive, one of four mini cute data centers built across France. This service offers long-term storage for "'cold data"', a cold cemetery.

OVH?

Its name stands for “on vous héberge”, “we host you” in french.
OVH is Europe's biggest hosting provider and the third largest in the world in terms of the number of physical servers. The company mainly serves customers in Europe and the US, and has opened branches in Singapore, Australia and India.

It was founded in 1999 by Octave Klaba and his family, in Roubaix, France.

I read this enlightening article about the rise of OVH and Klaba as an entrepreneur. There are always human beings anyway behind companies, right?
Klaba wears t-shirts and trainers, plays guitar, works himself to exhaustion, just like a normal person, except he's running a 2 billion dollar company, isn't that great? That could be you.

He came from nowhere to build a great company that will be a European leader in cloud computing
😬😬😬
Klaba was so dedicated that he used to sleep next to his equipment some nights
😬😬😬

I don't know Klaba personally, but the way he's portrayed, he seems like the kind of guy you'd enjoy having a coffee with (I don't think Elon Musk, for example, is the kind of guy you'd enjoy having a coffee with). He invests in other start-ups and founds a new company that makes wind turbines. The money is getting greener, as we already know and he seems to know as well.

OVH and Klaba definitely have certain punk characters. Not as 'punk' as Elon Musk, but definitely punk enough to recognise him in the crowd of entrepreneurs.

Fun facts:

Octave Klaba made a bold move hosting a mirror website (replica) of Wikileaks after the US government shut it down in 2010. This action naturally caused some controversy. He also had some disputes with the french government, as he stood up for civil and digital freedoms. He opposed a new French intelligence law and criticised features such as algorithms for monitoring internet traffic and wiretapping systems that target individuals (to research more).

Interestingly, Klaba's email address appears in a GCHQ (British intelligence agency) test report on interception, adding another layer to the punk narrative. 🤘🖤🎸 (in a test aimed at knowing whether the intercept of a satellite liaison between Sierra Leone and Belgium was technically possible, why tho??)

In 2013, OVH was hit with a data breach when some hackers broke into the company's servers via email accounts. The investigation into this breach is still ongoing. It could be a case of industrial espionage, as OVH explained. "we weren't cautious enough. We're now operating in a state of heightened caution," Klaba told the media. :)))))

Returning back to OVH,
OVH is a pioneer of free cooling, which uses water instead of air conditioning to cool processors. This innovation was key to OVH's cost-efficient effective approach and helped the company to be competitive in the great sharks market dominated by giants such as Google, Amazon and Microsoft. Quite interesting is the new hybrid immersion cooling system (wow, matrix), that uses a thermally and non-electrically conductive liquid.

OVH markets itself as a green data center solution. For example the Gravelines data center is located right next to the Gravelines nuclear power plant. In 2021, OVH joined the Climate Neutral Data Center Pact, which aims to achieve climate neutrality for data centers by 2030.
OVH also promotes itself as an independence solution for Europe.
Many european politicians would like to see OVH succeed, as they see the continued dependence on the US cloud as a risk to european sovereignty and economic competitiveness.
Klaba used these concerns to promote OVH as the preferred secure alternative, as the company's servers outside the US are not subject to US search warrants and law enforcement actions, as well as infiltration by intelligence agencies.

'...we believe in an open system that keeps the users, coders, developers, partners, administrators and operators in control of their data – to manage in the right way, in a responsible way. That’s how we build freedom'

In 2023, OVH took a big step by introducing a quantum computer named MosaiQ in its data center in Croix, France. It was developed by Quandela and is the first of its kind to be housed in a data center. Unlike other quantum computers, MosaiQ's design uses light beams and photonics principles, so it does not require extremely low temperatures (might research a bit more research on this).

sources:
https://www.datacenterdynamics.com/en/news/ovh-launches-quantum-computer-at-data-center-in-france/
https://www.ft.com/content/c78ee22d-ceb3-4a31-aae4-96589bb5cddf
https://www.lemonde.fr/en/archives/article/2022/03/14/britain-spied-on-the-ceo-of-europe-s-biggest-internet-hosting-company_5978491_113.html#
https://www.ovhcloud.com/en/lp/status-services-backup-strasbourg/
https://www.datacenterknowledge.com/industry-perspectives/increasing-viability-sustainable-fire-protection-data-centers#close-modal
https://www.theregister.com/2022/06/10/ovhcloud_datacenter_fire_last_year/
https://pa-schembri.medium.com/an-opiniated-review-of-the-ovh-cloud-archive-service-f0b2d615c5ca https://blog.ovhcloud.com/new-hybrid-immersion-liquid-cooling-developments-at-ovhcloud/
https://www.climateneutraldatacentre.net/

@@ Line 12: / Line 12: @@
 OVH nicely declined to comment on its fire safety protocol, but from the findings of the BEA (French Bureau for Investigation and Analysis of Industrial Risks) we can conclude that this may have been the main problem, and not just that. The BEA report pointed out problems such as the lack of an automatic fire extinguisher system in the building and delay in electrical cutoff which made it more difficult to stop the flames. Toxic fumes from lead acid batteries also worsened the situation.<br>
 <br>
-Initial investigations suggested that a water leak onto an inverter may have started the fire. So, the absence of an automatic fire extinguisher system, the delayed power shutdown and the general design of the building design likely worsened the damage. However, the report still tell, curiously, that the exact cause of the fire remains uncertain. So it is still just hyphotesis.
+Initial investigations suggested that a water leak onto an inverter may have started the fire. So, the absence of an automatic fire extinguisher system, the delayed power shutdown and the general design of the building design (wooden floors) likely worsened the damage. However, the report still tell, curiously, that the exact cause of the fire remains uncertain. So it is still just hyphotesis.
 How come it's only hyphotesis?<br>
 <br>