Falcon Content Update Remediation and Guidance Hub

Page last updated 2024-07-22 0238 UTC

CrowdStrike is actively assisting customers affected by a defect in a recent content update for Windows hosts. Mac and Linux hosts were not affected. The issue has been identified, isolated, and a fix has been implemented. This was not a cyberattack.

Customers are encouraged to check the support portal for updates. We will also continue to provide the latest information here and on our blog as it becomes available. We encourage organizations to verify that they are communicating with CrowdStrike representatives through official channels.

We assure our customers that CrowdStrike is operating normally and that this issue does not impact our Falcon platform systems. If your systems are operating normally, there is no impact on their protection with the Falcon sensor installed.

We understand the seriousness of this situation and apologize for any inconvenience and disruption. Our team is fully mobilized to ensure the safety and stability of CrowdStrike customers.

Overview

Statement from our CEO

Sent 2024-07-19 1930 UTC

Dear customers and partners,

I want to sincerely apologize to all of you for the outage. CrowdStrike understands the severity and impact of the situation. We quickly identified the issue and implemented a fix so we could diligently focus on restoring our customers’ systems as our highest priority.

The outage was caused by a defect in a Falcon content update for Windows hosts. Mac and Linux hosts are not affected. This was not a cyberattack.

We are working closely with affected customers and partners to ensure all systems are restored so you can deliver the services your customers rely on.

CrowdStrike is operating normally and this issue does not impact our Falcon platform systems. There is no impact to security if the Falcon sensor is installed. Falcon Complete and Falcon OverWatch services are not disrupted.

We provide ongoing updates through our Support Portal at https://supportportal.crowdstrike.com/s/login/.

We have mobilized CrowdStrike to help you and your teams. If you have questions or need additional support, please contact your CrowdStrike representative or Technical Support.

We know that adversaries and malicious actors will attempt to exploit these types of events. I encourage everyone to remain vigilant and ensure you are reaching out to official CrowdStrike representatives. Our blog and technical support remain the official channels for the latest updates.

Nothing is more important to me than the trust our customers and partners have placed in CrowdStrike. As we resolve this incident, I have made a commitment to provide full transparency into how this happened and the steps we are taking to prevent it from happening again.

George Kurtz

Founder and CEO of CrowdStrike

Technical details

• Technical details about the outage can be found here: Read the blog Published on 2024-07-19 0100 UTC
• We assure our customers that CrowdStrike is operating normally and this issue does not impact our Falcon platform systems. If your systems are operating normally, there will be no impact on their protection with the Falcon Sensor installed. Falcon Complete and OverWatch services are not disrupted by this incident.
• CrowdStrike identified the trigger for this issue as a Windows sensor-related content deployment and we have rolled back those changes. The content is a channel file in the %WINDIR%\System32\drivers\CrowdStrike directory.
• Channel file “C-00000291*.sys” with time stamp 2024-07-19 0527 UTC or later is the rolled back (good) version.
• Channel file “C-00000291*.sys” with time stamp 2024-07-19 0409 UTC is the problematic version.
• Please note: It is normal to have multiple “C-00000291*.sys” files present in the CrowdStrike directory as long as An of the files in the folder have a timestamp of 05:27 UTC or later, it becomes the active content.
• Symptoms include hosts experiencing a bugcheck/blue screen error related to the Falcon Sensor.
• Windows hosts that not are affected, no action is required as the problematic channel file has been rolled back.

Unimpacted hosts

• Windows hosts brought online after 2024-07-19 0527 UTC are not affected
• Windows hosts installed and provisioned after 2024-07-19 0527 UTC are not affected Updated 2024-07-21 1435 UTC
• This issue does not affect Mac or Linux based hosts

How do I identify affected hosts?

How do I identify affected hosts via an advanced event search?
Updated 2024-07-22 0139 UTC

The queries used by the dashboards are listed at the bottom of the respective KB articles about the dashboard.

How do I identify the affected hosts via the dashboard?
Updated 2024-07-22 0139 UTC

An updated granular dashboard is available that shows the Windows hosts affected by the content update defect described in this Tech Alert. See Granular Health Dashboards to Identify Windows Hosts Affected by a Content Issue (v8.6). Note that the queries used by the dashboards are listed at the bottom of the respective dashboard KB articles.

If hosts are still crashing and cannot stay online to receive the channel file update, the recovery steps below can be used.

How do I troubleshoot individual hosts?
Updated 2024-07-21 0932 UTC

• Reboot the host to give it the opportunity to download the rolled back channel file. We highly recommend putting the host on a wired network (rather than wifi) before rebooting, as Ethernet will give the host significantly faster internet connectivity.
• If the host crashes again upon reboot:
• Option 1 – Manual
• See this Microsoft article for detailed steps.
• Please note: Hosts encrypted with Bitlocker may require a recovery key.
• Option 2 – Automated via bootable USB stick

How can I recover Bitlocker keys?
Updated 2024-07-21 1810 UTC

How to Recover Resources from a Cloud-Based Environment

Information about external suppliers
Updated 2024-07-20 2259 UTC

Additional Resources