What Happened with the CrowdStrike Update?

A recent CrowdStrike Falcon sensor update has caused a massive Windows Blue Screen of Death (BSOD) outage. CrowdStrike offers endpoint protection and other services that are used on a widespread scale worldwide, and this sensor update issue is causing global issues.

Impact

There are widespread reports of BSOD error on Windows hosts, all of which are associated with multiple versions of CrowdStrike sensors. This update is believed to have sent servers, desktops, laptops and computer endpoints into a spiral of reboots that are commonly referred to as the “blue screen of death,” wilth the error message, “DRIVEN_OVERRAN_STACK_BUFFER.”

Details of the CrowdStrike Update

Symptoms of the Windows crash include hosts experiencing a bugcheck message or the Blue Screen of Death error, both of which are related to CrowdStrike Falcon Sensor update. CrowdStrike has indicated that channel file C-00000291*.sys with timestamp of 0409 UTC is the problematic version, and that channel file C-00000291*.sys with timestamp of 0527 UTC (July 19) or later is the reverted (good) version. 

CrowdStrike has also indicated that  Windows hosts that are brought online after 0527 UTC, Hosts running Windows 7/2008 R2, and Mac- or Linux-based hosts will not be impacted.

Current Actions for Remediation

Workaround Steps for individual hosts:

1. Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
   1. Boot Windows into Safe Mode or the Windows Recovery Environment
      • NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
2. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
3. Locate the file matching C-00000291*.sys and delete it.
4. Boot the host normally.
   • Note: Bitlocker-encrypted hosts may require a recovery key.

Workaround Steps for public cloud or similar environment including virtual:

Option 1:

1. ​​​​​​​Detach the operating system disk volume from the impacted virtual server
2. Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
3. Attach/mount the volume to to a new virtual server
4. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
5. Locate the file matching C-00000291*.sys and delete it.
6. Detach the volume from the new virtual server
7. Reattach the fixed volume to the impacted virtual server

Option 2:

• ​​​​​​​Roll back to a snapshot before 0409 UTC.

Get Assistance 

As many are affected worldwide, we understand that you might be impacted by the recent CrowdStrike agent issue and are working to fix it. We are here to help you. Upwind has put together a dedicated team available 24/7 to provide you with the support needed to get you back up and running. Please do not hesitate to reach out to us at any time.

For Upwind customers, please reach out to us in the console chat. For all others, please fill out the following form or email us at [email protected] and will reach out to you within minutes of submission.