CrowdStrike Update: Latest News, Lessons Learned from a Retired Microsoft Engineer

Summary

In this video, Dave, a retired Microsoft software engineer, discusses the recent CrowdStrike Falcon cybersecurity platform outage caused by a faulty sensor configuration update. He provides technical details, updates on conspiracy theories, and broader lessons learned from the incident, emphasizing the need for better security practices and communication.

Takeaways

• 👋 Introduction: Dave, a retired Microsoft software engineer, discusses the recent CrowdStrike Falcon cybersecurity platform outage.
• 🔧 Technical Details: The outage was caused by a faulty sensor configuration update in the Falcon platform, specifically a malformed 'Channel file 291'.
• 💥 Impact: Approximately 8.5 million Windows devices were affected, leading to significant disruptions across various industries, including banking, airlines, and emergency services.
• 🛠️ Quick Fix: CrowdStrike identified the issue and deployed a fix to prevent further machines from being affected but did not automatically fix the already impacted systems.
• 👨‍💻 Manual Intervention: System administrators and IT professionals worldwide had to manually boot affected machines into safe mode to remove the corrupted update file and reboot.
• 🤔 Microsoft's Role: Despite the issue being primarily with CrowdStrike, the reliance on kernel drivers in Windows raises questions about Microsoft's platform design.
• 🔄 Past Incidents: CrowdStrike has had similar issues affecting Debian and Linux, and Rocky Linux, indicating a pattern of problems with their updates.
• 🍎 Cross-Platform: CrowdStrike also provides security solutions for macOS, but the Falcon sensor for macOS does not install kernel extensions due to Apple's deprecation of them.
• 🛡️ Microsoft's Challenges: The Windows platform requires deep integration for security functionalities, which currently necessitates kernel-side code, posing stability risks.
• 🏛️ Regulatory Hurdles: Microsoft developed an advanced API for security applications like CrowdStrike, but EU regulators deemed it anti-competitive and prohibited its implementation.
• 📚 Lessons Learned: The incident highlights the need for better communication, crisis management, and possibly reconsidering the reliance on kernel mode code for security solutions.

Q & A

Who is Dave and what is his background?

Dave is a retired Microsoft software engineer who started working on Windows back in the early 1990s. He now runs a shop and creates content, including updates on the latest news and speculations, particularly focusing on cybersecurity issues.

What was the cause of the recent CrowdStrike IT outage?

The recent CrowdStrike IT outage was caused by a faulty sensor configuration update in their Falcon cybersecurity platform. The update involved a malformed configuration file known as Channel file 291, which triggered a logic error in the CrowdStrike kernel driver, resulting in system crashes.

How many devices were impacted by the CrowdStrike IT outage?

Approximately 8.5 million devices worldwide were impacted by the CrowdStrike IT outage, causing significant disruptions across various industries.

What was the nature of the 'fix' CrowdStrike deployed after identifying the issue?

The 'fix' CrowdStrike deployed was to prevent more machines from being affected by the faulty update. However, for the machines that had already taken the update, the fix did not automatically resolve the issue; it required manual intervention by system administrators or users to boot into safe mode, delete the corrupted update file, and reboot.

Why is it ironic that the IT outage is often associated with Microsoft despite it being a CrowdStrike issue?

It is ironic because the issue primarily lies with CrowdStrike's platform and not specifically with Windows itself. However, the perception might be due to the fact that the impact manifested on the Windows platform, which is developed by Microsoft.

What similar issues did CrowdStrike face with non-Windows operating systems?

CrowdStrike faced similar issues with Debian and Linux on April 19th, causing systems to crash and preventing normal reboots. Another issue occurred on May 13th affecting Rocky Linux servers, which experienced freezes after upgrading to Rocky Linux 9.4, linked to a Linux sensor operating in user mode combined with Pacific 6.x kernel versions.

Why doesn't the CrowdStrike sensor for macOS install kernel extensions?

The CrowdStrike sensor for macOS does not install kernel extensions because, starting with macOS Big Sur and later versions, Apple deprecated the use of kernel extensions entirely. Instead, CrowdStrike has rearchitected its sensor to use system extensions provided by Apple.

What is the role of a kernel driver and why is it considered risky?

A kernel driver has very intimate access to the system's most inner workings, allowing for low-level system access necessary for certain security functionalities. However, it is risky because if anything goes wrong with the kernel driver, the system must blue screen to prevent further damage to user settings, files, and security.

What was the impact of the regulatory body's decision on Microsoft's advanced API for security applications?

The regulatory body, concerned with fair competition, deemed the advanced API anti-competitive and prohibited its implementation. This decision was based on the fear that the API could create a dependency on Microsoft's ecosystem, effectively locking out competitors who couldn't leverage the same level of access to the Windows core.

What are some of the lessons that can be learned from the CrowdStrike IT outage?

Lessons include the potential risks of relying on a single vendor for critical infrastructure, the need for critical systems like 911 to be on an N-1 or N-2 update schedule, and the importance of proper vetting and testing of software updates to prevent widespread impact from bugs.

What is the significance of the Tylenol crisis in the context of corporate crisis management?

The Tylenol crisis is significant as it set a new standard for corporate crisis management through transparency, decisiveness, and a focus on consumer safety. It demonstrated the power of ethical leadership and the importance of maintaining open communication during a crisis.

What are some of the conspiracy theories that emerged following the CrowdStrike outage?

Some conspiracy theories suggest that the outage was a deliberate cyber attack signaling the onset of World War III, while others propose it was orchestrated by political figures to influence geopolitical events. However, these theories lack evidence and are speculative.

Why is it important for a device driver to properly vet its input?

It is important for a device driver to properly vet its input to prevent access violations and system crashes. Even if the input files are signed, the code needs to sanity check the contents to ensure they are valid and not corrupted, which can help avoid reliance on luck and prevent potential system failures.