Servers down after CrowdStrike update — How it happened and how to fix

July 19, 2024

If you’re managing servers you may need to cancel your weekend plans as a CrowdStrike update has caused servers to BSOD / boot loop.

The incident does not appear to be a security incident or cyberattack, and only affects Windows hosts, with CrowdStrike saying Linux and Mac are not affected.

The issue was first reported 19:00 UTC on July 18 and was acknowledged by CrowdStrike in the early hours of July 19.

“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts,” CrowdStrike CEO George Kurtz wrote on Twitter/X.

“This is not a security incident or cyberattack,” he added, “the issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.”

CrowdStrike Message — (Image credit: CrowdStrike)

The good news is that a fix has already been found. The bad news is that as servers are not booting it is likely many will require manual intervention. CrowdStrike gave the following instructions on how to fix the issue.

Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching C-00000291*.sys* and delete it
Boot the host normally

Microsoft later issued further advice:

We recommend customers that are able to, to restore from a backup from before 19:00 UTC on the 18th of July
Alternatively, attempt to repair the OS disk offline.
Attach a disk to VM for offline repair (Encrypted disks may need further instructions)
Once the disk is attached delete the Windows/System/System32/Drivers/CrowdStrike/C00000291*.sys file
We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance.

Who is affected by the CloudStrike update?

The CrowdStrike update has affected Windows devices and Virtual Machines running Windows Client and Windows Servers running the CrowdStrike Falcon agent. Personal PCs running Windows are not affected.

It’s not yet known exactly how many machines have been affected but it’s already had a large impact on the globe especially in Europe with Visa, Amazon, and Microsoft all reporting issues. There have also been reports of airlines and hospitals having issues. We won’t know the full extent of the impact until later in the day.

How to fix the CrowdStrike issue?

Essentially, you need to delete the file matching C-00000291*.sys

You can do that by

1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching C-00000291*.sys and delete it

You may need to manually remove /update the OS disk

What is CrowdStrike?

CrowdStrike is a cybersecurity company behind software used by some of the largest companies and institutions around the world, including hospitals, airports, banks, and many businesses listed in the Fortune 500.

Source

If you’re managing servers you may need to cancel your weekend plans as a CrowdStrike update has caused servers to BSOD / boot loop. The incident does not appear to be a security incident or cyberattack, and only affects Windows hosts, with CrowdStrike saying Linux and Mac are not affected. …