Real men test in production… The truth about the CrowdStrike disaster

Summary

In a recent incident reminiscent of the Y2K bug, millions of Windows machines crashed due to a faulty update from cybersecurity firm CrowdStrike, affecting 8.5 million devices. The video explores possible causes, from a simple coding error to a potential cyber attack or conspiracy. It delves into technical details, revealing a logic error in a channel file update that led to the system crash, and discusses the implications of running critical software without stringent quality control measures. The video also humorously speculates on other theories, including a multi-dimensional plot and the promotion of a different programming language for driver development.

Takeaways

• 💻 On July 22, 2024, millions of Windows machines crashed due to a faulty update from cybersecurity firm CrowdStrike, affecting 8.5 million devices.
• 🔄 The incident is eerily similar to a 2010 McAfee antivirus update that caused a widespread outage, with the same CTO, George Kurtz, involved in both events.
• 👷‍♂️ CrowdStrike's software, Falcon Sensor, operates in the privileged 'ring zero' space, typically reserved for Microsoft, and requires a special certification from Microsoft.
• 🛑 The crash was caused by a logic error in an update to a configuration file, leading to a system-wide failure, which is unusual for application crashes.
• 👨‍💻 A professional C++ programmer hypothesized that the issue was due to a null pointer dereference, a common coding mistake that should have been caught.
• 🔍 The community noted that the code may have been flawed for a while, and the problematic configuration file update was the final straw that exposed the issue.
• 🚫 The incident highlights the importance of robust quality assurance processes in software development, especially for critical systems.
• 💡 The video suggests that the root cause of the disaster was likely a lack of quality control within CrowdStrike, rather than a single developer's error.
• 🕵️‍♂️ Conspiracy theories suggest that the crash was either a foreign spy's infiltration, a rogue employee's message, or a pre-planned test for a future cyber attack.
• 🌐 The video also touches on the idea that the world economic forum has predicted a worldwide cyber attack, and CrowdStrike's incident might be connected.
• 🎓 The sponsor, Brilliant, is highlighted as a platform for learning problem-solving skills essential for programming and overcoming complex challenges in software development.

Q & A

What caused millions of Windows machines to go down recently?

A bad update from cybersecurity firm CrowdStrike caused millions of Windows machines to go down.

How many devices were affected by the CrowdStrike update issue?

8.5 million devices were affected by the CrowdStrike update issue.

Who was the CTO of McAfee during the 2010 incident, and what is his current position?

The CTO of McAfee during the 2010 incident was George Kurtz, who is now the CEO of CrowdStrike.

What specific mistake did the CrowdStrike update make that caused the system crashes?

The CrowdStrike update contained a logic error in Channel file 291, which caused the system crashes.

What is the role of the CrowdStrike Falcon sensor?

The CrowdStrike Falcon sensor is software that runs in the background on machines, looking for potential security anomalies and executing code via a driver.

What mode does the CrowdStrike software run in, and why is this significant?

The CrowdStrike software runs in ring zero, or kernel mode, which is the most privileged zone around the CPU usually reserved for process scheduling and direct hardware access.

What certification must third-party code have to run in kernel mode on Windows, and did CrowdStrike have this certification?

Third-party code must have WHQL certification from Microsoft to run in kernel mode on Windows, and the CrowdStrike driver was WHQL certified.

What was the hypothesis of a professional C++ programmer about the cause of the CrowdStrike issue?

The hypothesis was that an engineer coded up a null pointer trying to access a memory address that doesn't exist, a rookie coding mistake that could have been fixed with an if statement.

What deeper conspiracy theories have emerged regarding the CrowdStrike incident?

Some conspiracy theories suggest it was the work of a foreign spy, a rogue employee, or a pre-planned event by the World Economic Forum to test for a real cyber attack in 2026.

What lesson about quality control and organizational failures can be learned from the CrowdStrike incident?

The incident highlights the importance of multiple layers of protection, quality assurance, continuous integration, and staggered rollouts to prevent such failures from reaching production.

Outlines

00:00

🔍 Y2K Revisited: CrowdStrike's Major Update Failure

Last Friday, millions of Windows machines were affected by a faulty update from cybersecurity firm CrowdStrike, reminiscent of the Y2K experience. The update impacted 8.5 million devices, leading to theories about whether it was a simple mistake, a cyber attack cover-up, or a long-planned false flag operation. The incident echoed a similar event in 2010 when a McAfee update caused a major disruption, with George Kurtz, the CEO of CrowdStrike, having been McAfee's CTO at that time. This coincidence raises questions about systemic failures and the accountability of such high-profile executives.

05:02

🛠️ Understanding the Technical Failures of CrowdStrike

CrowdStrike's recent failure was linked to an update in their Falcon sensor software, specifically a logic error in Channel file 291. Unlike typical application crashes, this incident caused system-wide failures because the software operates in the CPU's privileged ring zero. The fault lay in the driver code, certified by Microsoft, which encountered issues with the updated config file, leading to a widespread crash. An analysis by a C++ programmer suggested a simple coding error might have been the root cause, highlighting deficiencies in the company's quality control and the need for better safeguards in critical software systems.

💡 Root Causes and Conspiracy Theories Surrounding the Incident

The CrowdStrike incident revealed broader organizational failures, with inadequate quality control and potentially deeper issues. While some blamed individual developers, the problem seemed to stem from systemic lapses in testing and deployment processes. Speculation ranged from a foreign spy infiltrating the company to theories about pre-planned cyber attacks linked to the World Economic Forum. The incident underscored the importance of rigorous quality assurance in software development, especially for products operating in critical system areas.

📅 Future Cybersecurity Threats and the Importance of Robust Driver Development

Looking ahead, some conspiracy theories suggest this incident was a prelude to a larger cyber attack predicted for August 12, 2026. Despite the outlandish nature of these claims, they emphasize the need for robust development practices. The video promotes Brilliant, an educational platform that helps users develop problem-solving skills crucial for programming, offering a way to enhance one's ability to tackle complex software challenges effectively.

Keywords

💡Y2K experience

The term 'Y2K experience' refers to the widespread fear and anticipation of computer failures at the turn of the year 2000 due to the 'Year 2000 problem' or 'Y2K bug'. In the video, it is used metaphorically to describe the recent widespread failure of Windows machines, drawing a parallel to the Y2K fears.

💡CrowdStrike

CrowdStrike is a cybersecurity technology company. In the video, it is mentioned as the source of a bad update that caused millions of Windows machines to go down, highlighting the company's significant role in the recent incident discussed.

💡McAfee Antivirus

McAfee Antivirus is a well-known computer security software. The video script references an incident in 2010 where a McAfee update caused a similar widespread computer failure, drawing a connection between past and present events and noting George Kurtz's role in both companies.

💡Windows service host

The Windows service host is a critical system process in Windows operating systems. In the video, it is mentioned as being accidentally removed by a McAfee update in 2010, leading to widespread system failures, illustrating the importance of this component in system stability.

💡Blue screen of death

The 'blue screen of death' (BSOD) is an error screen displayed when Windows encounters a critical system error. The video uses this term to describe the outcome of the McAfee and CrowdStrike incidents, indicating severe system crashes.

💡George Kurtz

George Kurtz is the CEO of CrowdStrike and formerly the CTO of McAfee. The video script highlights his connection to both companies during major system failures, suggesting a pattern of issues related to his leadership.

💡Ring zero

In the context of operating systems, 'ring zero' refers to the most privileged mode of processor operation, typically reserved for the kernel. The video explains that CrowdStrike's software operates in ring zero, which is unusual for third-party software and contributed to the severity of the system crash.

💡WHQL certification

WHQL (Windows Hardware Quality Labs) certification is a Microsoft program that verifies the reliability and stability of hardware and drivers. The video mentions that CrowdStrike's driver had this certification, which raises questions about the certification's effectiveness in preventing such failures.

💡Null pointer

A 'null pointer' in programming is a pointer that does not point to any valid memory location. The video suggests that a null pointer access might have been the cause of the CrowdStrike update failure, indicating a common but critical coding error.

💡Configuration file

A configuration file contains settings and parameters for software. In the video, it is suggested that an error in parsing a configuration file may have led to invalid pointers, contributing to the system crash.

💡Organizational failure

The term 'organizational failure' refers to a breakdown in processes or systems within an organization that leads to negative outcomes. The video concludes that the CrowdStrike incident was likely due to poor quality control and organizational practices, rather than the actions of a single individual.

💡World Economic Forum

The World Economic Forum is an international organization known for its meetings and initiatives on global issues. The video mentions it in the context of conspiracy theories suggesting that the CrowdStrike incident was a test run for a larger, pre-planned cyber attack, linking it to the organization's partnerships and predictions.

Highlights

On July 22nd, 2024, millions of Windows machines went down due to a bad update from cybersecurity firm CrowdStrike.

8.5 million Windows machines were affected by the update.

The incident is reminiscent of a 2010 McAfee Antivirus update that caused a similar issue.

George Kurtz, the CTO of McAfee in 2010, is now the CEO of CrowdStrike.

CrowdStrike released an official statement explaining the technical details of the incident.

The CrowdStrike Falcon sensor is software that runs in the background, looking for security anomalies.

A logic error in an update to channel file 291 caused the system to crash.

CrowdStrike operates in ring zero, the most privileged zone around the CPU.

The CrowdStrike driver was WHQL certified, allowing it to run in ring zero.

A professional C++ programmer hypothesized that the issue was due to a null pointer access.

The driver code has potentially been broken for a long time, with the config file being the final straw.

The incident might not have been caught due to a lack of quality control and organizational failure.

Colonel Kurtz is known for testing in production and is willing to take risks.

CrowdStrike sells a very expensive product that few understand, prioritizing sales over software engineering.

There are theories suggesting the incident was not accidental but the work of a foreign spy or rogue employee.

Some believe the failure was pre-planned as a test run for a real cyber attack scheduled for 2026.

The video sponsor, Brilliant, offers a platform to develop problem-solving skills in programming.

Transcripts

last Friday the world finally got the

Y2K experience it deserved when millions

of Windows machines went down thanks to

a bad update from cyber security firm

crowd strike 8.5 million to be exact but

now the plot is thickened and multiple

theories for why this actually happened

have emerged a was it just a silly

mistake B was it actually a Cyber attack

being covered up or C was it a false

flag planned centuries ago by our

multi-dimensional lizard overlords in

today's video we'll try to find out what

really happened by taking a deep dive

into the technical details but first

here's a crazy detail you need to know

on April 21st 2010 at approximately

1,400 hours a McAfee Antivirus update

accidentally removed the windows service

host file and knocked millions of

computers running Windows XP off the

internet causing many of them to go into

an endless reboot loop the blue screen

of death shut down critical services

around the world that was 15 years ago

when Justin Bieber was only 16 years old

but it's nearly identical to the

crowdstrike disaster going on right now

here's the crazy part though the CTO of

McAfee in 2010 was none other than

George kurts the CEO of crowd strike

today that's quite the example of

failing upwards now he did just lose

$300 million in paper wealth but most

importantly we now know the embarrassing

truth about how the crowd strike

disaster actually happened almost it is

July 22nd 2024 and you watching the code

report the creator of C++ be straup once

said C++ makes it harder to shoot

yourself in the foot but when you do you

blow your entire leg off and we should

have listened to him crowd strike

released an official statement

explaining what happened come on you

guys there it is right there in front of

you the whole time you're dereferencing

a m pointer open your eyes the crowd

strike Falcon sensor is software that

sits in the background on your machine

looking for potential security anomalies

it contains a driver which is the thing

that actually executes code along with a

bunch of Channel files which are

basically just config files that contain

rules about new potential attacks that

the sensor can look for these files are

not kernel drivers and can be updated on

the Fly and when crowd strike pushed an

update to channel file 291 a logic error

caused the entire system to crash now

normally when an application crashes it

only breaks that application running in

user land or ring three in the CPU

protection ring no blue screen of death

required but crowd strike is a unique

piece of software that runs within ring

zero or kernel mode the most privileged

Zone around the CPU usually reserved for

process scheduling and direct Hardware

access ring zero is an area that

normally only microsof is are allowed to

touch and in order for any third party

to run code here they must receive a

whql certification from Microsoft to

verify that your code won't Breck 8.5

million devices and shut down the global

economy the crowd strike driver was whql

certified so it sounds like it's

Microsoft's fault well not so fast

what's unique about crowd strike is that

they can make updates to those config or

Channel files dynamically in this case

the driver had some kind of issue

reading Channel file 291 causing the

entire system to fail that's pretty much

all the detail we have from official

sources but luckily there's a guy on the

internet who's a professional C++

programmer and provided a breakdown that

went viral his hypothesis was that this

was a skill issue where some engineer

coded up a n pointer trying to access a

memory address that doesn't exist a

simple rookie coding mistake that could

have been fixed with an if statement

this tweet got a lot of traction but

since then it's been Community noted and

another security researcher explains

that this code is reading pointers from

a table in a loop and some are invalid

perhaps an error parsing the

configuration file left some entries

uninitialized what's kind of crazy here

is that it looks like the driver code

has actually been broken for a long time

and this one config file was the straw

that broke the camel's back we may not

know the full truth until there's a

congressional hearing but it looks like

some developer there wrote some bad code

said works on my machine but then made

the horrible mistake of deploying on a

Friday but we can't blame this one

person programmers write bad code all

the time but a failure like this should

never reach production the Falcon sensor

is not just some crappy to-do list app

when software operates in the critical

path like this there should be multiple

layers of protection quality assurance

continuous integration this staggered

rollouts and so on it's absolutely

insane that this wasn't caught by some

automated process before it killed 8.5

million computers heads need to roll for

this but it's not the person who wrote

the code it's an organizational failure

and it's not the first time Colonel

Curts has been connected to a worldwide

outage he knows that real men test in

production and is willing to die on that

Hill the thing is this company sells a

very expensive product that very few

people understand and if you want to

have an exotic car collection like this

your Enterprise sales team is your

highest priority not your software

engineering team those nerds therefore

the most likely root cause of This

Disaster is just a lack of quality

control at the company crowd strike but

another theory floating around is that

this wasn't an accident but actually the

work of a foreign spy who infiltrated

the company or perhaps a rogue employee

who wanted to send a message a message

that is time to switch to the Russ

programming language for Windows driver

development but the conspiracy theories

go even deeper and some think this

failure is so egregious that it was

actually pre-planned in advance the

world economic Forum has made

predictions about a worldwide Cyber

attack and crowd strike is a World

economic Forum partner this was all just

a test run for the real Cyber attack

scheduled to happen on August 12th 2026

most of us will already be dead by then

but if your goal is to write robust

Colonel drivers on Windows you'll need

to know how to problem solve like a

programmer and you can start doing that

for free thanks to this video sponsor

brilliant problem solving is a skill

that you keep forever brilliant's

platform will introduce you to essential

programming Concepts but most

importantly the handson exercises will

develop your brain to recognize and

solve complex problems that developers

need to over come on a daily basis best

of all every lesson is concise and

rewarding by investing just a few

minutes each day you'll develop habits

that can level up your programming

skills for the rest of your life and you

can do it anywhere even from your phone

to try everything brilliant has to offer

for free for 30 days visit brilliant.org

fireship or scan this QR code for 20%

off their premium annual subscription

this has been the code report thanks for

watching and I will see you in the next

one