User:Alessia/data centers on fire: Difference between revisions

From XPUB & Lens-Based wiki
No edit summary
No edit summary
Line 136: Line 136:
https://www.dhakatribune.com/bangladesh/329192/millions-left-offline-as-khawaja-tower-fire-sparks<br>
https://www.dhakatribune.com/bangladesh/329192/millions-left-offline-as-khawaja-tower-fire-sparks<br>
https://www.dhakatribune.com/bangladesh/dhaka/329179/mohakhali-building-fire-death-toll-rises-to-3<br>
https://www.dhakatribune.com/bangladesh/dhaka/329179/mohakhali-building-fire-death-toll-rises-to-3<br>
='''SK C&C'''=
<br>
The SK C&C data centre is operated by the SK Group, one of South Korea's largest industrial groups.
The fire caused disruption for around eight hours. The incident occurred at the 70,000 square meter facility in Pangyo Techno Valley, near Seoul.<br>
<br>
The fire started in a room for lithium-ion batteries, presumably caused by a battery manufactured by SK itself. According to initial rumours, SK Group's BMS gave two fire warnings hours before the actual fire broke out. SK Group has denied these allegations.<br>
<br>
The outage affected not only SK Group's own systems, but also Kakao, a well known South Korean internet company, including its widely used platform KakaoTalk. This led to the shutdown of 32,000 servers and disruption to various services such as mobile payments, transportation, gaming and music streaming, affecting millions of users. Naver, another major internet platform comparable to Google in South Korea, also experienced service disruptions. Messenger, banking, transportation, payment and e-commerce services were unavailable, impaired or interrupted for more than 10 hours because the backup systems did not respond quickly enough.<br>
<br>
Kakao itself has suffered direct losses of 20 billion dollars and has received many claims for lost business from partners such as retailers using its payment system and taxi drivers who use its ride app.<br>
The South Korean government demanded explanations from Kakao, SK Group and Naver about the incident and created a 'digital crisis management headquarters' to improve responsiveness to similar accidents.<br>
While Kakao was not responsible for the fire, it was responsible for its backup system, which was not working and was located at a site where the fire and connection interruption occurred.<br>
<br>
''We can’t take any measures because we don’t have the data. We can compensate clients based on our contracts''
<br>
<br>
sources:<br>
https://www.msit.go.kr/bbs/view.do?sCode=user&mId=113&mPid=112&pageIndex=&bbsSeqNo=94&nttSeqNo=3182267&searchOpt=ALL&searchTxt=<br>
https://theregister.com/2022/10/24/blazing_south_korean_data_centre/<br>
<br>

Revision as of 19:08, 23 March 2024

fire-ace-free-trial.png

https://hub.xpub.nl/chopchop/~aleevadh/

  • Your data is burning
  • Data centers like to autocombust
  • Don’t put all your internet infrastructure in one basket
  • The internet infrastructure is fragile
  • Data center sector is shockingly bad at reporting transparently
  • There is no black box for data centers
  • If your data didn’t burn it will drown
  • Data centers don’t like people! >:( things always happen when people are around
  • People actually die during fires >:(


OVH Cloud🔥


(Strasbourg, France, March 2021)

A massive fire has broken out in one of OVH's data centers, which is part of the OVHcloud network that includes other 44 centers around the world.
The servers housed in these centers hosted around 4 million websites, including dedicated government platforms from authorities in France, the UK, Poland and the Ivory Coast. This event also affected part of the .FR web domain.
SBG1 and SBG2 went up in flames. SBG2, a 500 m2 data center with five stores, and its contents were destroyed, while SBG1 was partially damaged. The building was eventually demolished. Two other buildings, SBG3 and SBG4, were not damaged. The different service branches affected by the fire were Web Cloud Universe (hosting of websites), Bare Metal Universe (high-performance servers for intensive computing, such as media encoding, gaming, corporate security), Hosted Private Cloud Universe (private cloud infrastructure) and Public Cloud Universe (OVH's cloud service).

The exact cause of the fire is not yet known, but it appears to have originated from two UPS systems (devices that maintain server operation during power outages). One of these UPSs had gone through maintenance the day before the fire, during which some components were replaced.
OVH nicely declined to comment on its fire safety protocol, but from the findings of the BEA (French Bureau for Investigation and Analysis of Industrial Risks) we can conclude that this may have been the main problem, and not just that. The BEA report pointed out problems such as the lack of an automatic fire extinguisher system in the building and delay in electrical cutoff which made it more difficult to stop the flames. Toxic fumes from lead acid batteries also worsened the situation.

Initial investigations suggested that a water leak onto an inverter may have started the fire. So, the absence of an automatic fire extinguisher system, the delayed power shutdown and the general design of the building design (wooden floors) likely worsened the damage. However, the report still tell, curiously, that the exact cause of the fire remains uncertain. So it is still just hyphotesis. How come it's only hyphotesis?

Determining the cause of the fire could have had enormous financial consequences for OVH, as more than 140 clients have sued for 10 milion euros in damages. So perhaps it's better to keep a little bit of mystery to keep the boat afloat.

OVH's confusion over the location of their servers has not helped the situation. Some companies believed that their data was backed up on the surviving servers, as written on their contracts, but this was not the case. There were also problems with the backup solutions, as some contracts falsely stated that the backup options were isolated from the server infrastructure. Great bad luck here.

Despite the suspicion that the water leak was the main cause of the fire in 2021, OVH continues to use water cooling for its servers. Eheh.

I read this news article, https://www.theregister.com/2022/09/13/ovh_sbg5_opens/, titled "OVH opens less flammable data center at site of 2021 fire" they could have just put a ;) emoji there too.

In 2022, OVH rebuilt its data center over the rubbles of SBG2. However, in the new SBG5, some changes were made to improve safety. They have separated the data center from the supporting infrastructure. The batteries are now stored in fire resistant containers outside the data center, as are the rooms containing the transformers.

It was a good start, especially for the investors, as it was a good opportunity to introduce a new cold storage service called Cold Archive, one of four mini cute data centers built across France. This service offers long-term storage for "'cold data"', a cold cemetery.

OVH
Its name stands for “on vous héberge”, “we host you” in french.
OVH is Europe's biggest hosting provider and the third largest in the world in terms of the number of physical servers. The company mainly serves customers in Europe and the US, and has opened branches in Singapore, Australia and India.

It was founded in 1999 by Octave Klaba and his family, in Roubaix, France.

I read this enlightening article about the rise of OVH and Klaba as an entrepreneur. There are always human beings anyway behind companies, right?
Klaba wears t-shirts and trainers, plays guitar, works himself to exhaustion, just like a normal person, except he's running a 2 billion dollar company, isn't that great? That could be you.

He came from nowhere to build a great company that will be a European leader in cloud computing
😬😬😬
Klaba was so dedicated that he used to sleep next to his equipment some nights
😬😬😬

I don't know Klaba personally, but the way he's portrayed, he seems like the kind of guy you'd enjoy having a coffee with (I don't think Elon Musk, for example, is the kind of guy you'd enjoy having a coffee with). He invests in other start-ups and founds a new company that makes wind turbines. The money is getting greener, as we already know and he seems to know as well.

OVH and Klaba definitely have certain punk characters. Not as 'punk' as Elon Musk, but definitely punk enough to recognise him in the crowd of entrepreneurs.

Fun facts:

Octave Klaba made a bold move hosting a mirror website (replica) of Wikileaks after the US government shut it down in 2010. This action naturally caused some controversy. He also had some disputes with the french government, as he stood up for civil and digital freedoms. He opposed a new French intelligence law and criticised features such as algorithms for monitoring internet traffic and wiretapping systems that target individuals (to research more).

Interestingly, Klaba's email address appears in a GCHQ (British intelligence agency) test report on interception, adding another layer to the punk narrative. 🤘🖤🎸 (in a test aimed at knowing whether the intercept of a satellite liaison between Sierra Leone and Belgium was technically possible, why tho??)

In 2013, OVH was hit with a data breach when some hackers broke into the company's servers via email accounts. The investigation into this breach is still ongoing. It could be a case of industrial espionage, as OVH explained. "we weren't cautious enough. We're now operating in a state of heightened caution," Klaba told the media. :)))))


Returning back to OVH,
OVH is a pioneer of free cooling, which uses water instead of air conditioning to cool processors. This innovation was key to OVH's cost-efficient effective approach and helped the company to be competitive in the great sharks market dominated by giants such as Google, Amazon and Microsoft. Quite interesting is the new hybrid immersion cooling system (wow, matrix), that uses a thermally and non-electrically conductive liquid.

OVH markets itself as a green data center solution. For example the Gravelines data center is located right next to the Gravelines nuclear power plant. In 2021, OVH joined the Climate Neutral Data Center Pact, which aims to achieve climate neutrality for data centers by 2030.
OVH also promotes itself as an independence solution for Europe.
Many european politicians would like to see OVH succeed, as they see the continued dependence on the US cloud as a risk to european sovereignty and economic competitiveness.
Klaba used these concerns to promote OVH as the preferred secure alternative, as the company's servers outside the US are not subject to US search warrants and law enforcement actions, as well as infiltration by intelligence agencies.

'...we believe in an open system that keeps the users, coders, developers, partners, administrators and operators in control of their data – to manage in the right way, in a responsible way. That’s how we build freedom'

In 2023, OVH took a big step by introducing a quantum computer named MosaiQ in its data center in Croix, France. It was developed by Quandela and is the first of its kind to be housed in a data center. Unlike other quantum computers, MosaiQ's design uses light beams and photonics principles, so it does not require extremely low temperatures (might research a bit more research on this).

Personality test result:
You are an enigmatic and mysterious person whose personality is shrouded in an aura of uncertainty. Like the uncertain origin of the flames that destroyed OVH's data centre, an aura of ambiguity surrounds you. Even when the answers seem clear, there is always a subtle uncertainty that leaves others wondering what the hell you are doing and when they will get their compensation money from you. Like the fire that hit OVH, you are capable of generating significant impact and provoking intense reactions. However, your true nature remains wrapped in mystery, leaving others guessing about your true intentions and motivations. But despite your enigmatic aura, you are also open to growth and change. Like OVH in rebuilding the SBG2 data center with significant security improvements, you have the potential to rise stronger than before, turning difficulties into opportunities and paving the way for new and better versions of yourself. Remember: the greener, the better. Maybe next time use some nice ecofriendly materials, like inflammable cool wood, to look even more stylish and environmentally friendly.


sources:
https://www.datacenterdynamics.com/en/news/ovh-launches-quantum-computer-at-data-center-in-france/
https://www.ft.com/content/c78ee22d-ceb3-4a31-aae4-96589bb5cddf
https://www.lemonde.fr/en/archives/article/2022/03/14/britain-spied-on-the-ceo-of-europe-s-biggest-internet-hosting-company_5978491_113.html#
https://www.ovhcloud.com/en/lp/status-services-backup-strasbourg/
https://www.datacenterknowledge.com/industry-perspectives/increasing-viability-sustainable-fire-protection-data-centers#close-modal
https://www.theregister.com/2022/06/10/ovhcloud_datacenter_fire_last_year/
https://pa-schembri.medium.com/an-opiniated-review-of-the-ovh-cloud-archive-service-f0b2d615c5ca https://blog.ovhcloud.com/new-hybrid-immersion-liquid-cooling-developments-at-ovhcloud/
https://www.climateneutraldatacentre.net/


Khawaja Tower🔥


(Dhaka, Bangladesh, October 2023)

The building was, and still is, a massive storage of wires. and was operated as a busy business center, housing one of Bangladesh's largest data centers where several mobile phone companies operate.
The building itself was built in the 1990s and does not have the capacity to house so many technology companies at once. It was reported that seventy percent of the country's technology servers were managed from this location. Two prominent data centers, NRB and Dhaka Cola, were housed in the tower, each connected to 10 International Internet Gateways (IEGs) and providing services to hundreds of Internet Service Providers (ISPs). In addition, about 40% of broadband internet services across Bangladesh were provided by ISPs connected to Khawaja Tower, covering about 20% of the country's 120 million mobile internet users.

The tragic incident has shown how fragile the internet infrastructure is, especially in Bangladesh. It makes us think about how easily we take internet connectivity for granted. If your Whatsapp isn’t working, maybe a data center is burning.
The prolonged fire, which lasted more than 16 hours, resulted in many people trapped in the 14 floors building being injured and some of them tragically losing their lives.

While the exact cause of the fire has not been officially released, it is evident that the presence of numerous flammable materials and the lack of a safety plan, a safety protocol for the building, contributed to the severity of the situation, despite the presence of some fire extinguishers.

After this disaster, the fact that just a single building could meet the needs of an entire country was called into question.

The building caught fire again a month later, which fortunately was quickly extinguished.

Personality Test Result: 
Wow! You are a symbol of connectivity and innovation. Just like this iconic building, you easily attract people, connect with everyone, you are the center of attention in your social circles. People are drawn to you because you are reliable and build easily everlasting connections. BUT you are tired of it, it's understandable because you can not always be available to people (maybe you want to become a feminist data center?).

You have a strong character and play an important role in many people's lives. You think of other people's needs before your own, and that's not always healthy. Just like Khawaja Tower you excel at managing multiple responsibilities and multitasking.

However, you may sometimes face challenges due to limitations in capacity of your own infrastructure. You should take care of your infrastructure!
But you do not let this deter you. You continue to strive for excellence and adapt to overcome obstacles.

The tragic incident of the fire reminds you that despite setbacks, you will come back stronger than before embodying the best spirit of perseverance.
Remember that you are more than just a building, you are a symbol of resilience. Keep on shining!


sources:
https://www.dhakatribune.com/bangladesh/329192/millions-left-offline-as-khawaja-tower-fire-sparks
https://www.dhakatribune.com/bangladesh/dhaka/329179/mohakhali-building-fire-death-toll-rises-to-3

SK C&C


The SK C&C data centre is operated by the SK Group, one of South Korea's largest industrial groups. The fire caused disruption for around eight hours. The incident occurred at the 70,000 square meter facility in Pangyo Techno Valley, near Seoul.

The fire started in a room for lithium-ion batteries, presumably caused by a battery manufactured by SK itself. According to initial rumours, SK Group's BMS gave two fire warnings hours before the actual fire broke out. SK Group has denied these allegations.

The outage affected not only SK Group's own systems, but also Kakao, a well known South Korean internet company, including its widely used platform KakaoTalk. This led to the shutdown of 32,000 servers and disruption to various services such as mobile payments, transportation, gaming and music streaming, affecting millions of users. Naver, another major internet platform comparable to Google in South Korea, also experienced service disruptions. Messenger, banking, transportation, payment and e-commerce services were unavailable, impaired or interrupted for more than 10 hours because the backup systems did not respond quickly enough.

Kakao itself has suffered direct losses of 20 billion dollars and has received many claims for lost business from partners such as retailers using its payment system and taxi drivers who use its ride app.
The South Korean government demanded explanations from Kakao, SK Group and Naver about the incident and created a 'digital crisis management headquarters' to improve responsiveness to similar accidents.
While Kakao was not responsible for the fire, it was responsible for its backup system, which was not working and was located at a site where the fire and connection interruption occurred.

We can’t take any measures because we don’t have the data. We can compensate clients based on our contracts

sources:
https://www.msit.go.kr/bbs/view.do?sCode=user&mId=113&mPid=112&pageIndex=&bbsSeqNo=94&nttSeqNo=3182267&searchOpt=ALL&searchTxt=
https://theregister.com/2022/10/24/blazing_south_korean_data_centre/