888-685-3101, ext. 2

Navigating CrowdStrike Outage: Latest Updates and Effective Workarounds

by | Jul 19, 2024 | Azure, Blog, Cloud, Microsoft

CrowdStrike recently experienced a significant outage affecting Windows hosts due to a defect in a single content update. This issue has impacted numerous environments, including Azure-related virtual machines and on-premises workstations. In this blog post, we’ll provide the latest updates from CrowdStrike and share effective workarounds for various environments.

Latest Update from CrowdStrike and Microsoft

As of July 19, 2024, CrowdStrike and Microsoft released an official statement and technical alerts regarding the issue:

  • The issue has been identified, isolated, and a fix has been deployed.
  • Only Windows hosts are affected; Mac and Linux hosts are not impacted.
  • This is not a security incident or cyberattack.
  • CrowdStrike is actively working with impacted customers and providing updates through official channels.

Read the CrowdStrike Statement

Read the Azure Status Updates

Understanding the Issue

Symptoms include hosts experiencing bugcheck/blue screen errors related to the Falcon Sensor. Windows hosts that haven’t been impacted don’t require any action, as the problematic channel file has been reverted. The problematic channel file is “C-00000291*.sys” with a timestamp of 0409 UTC, and the reverted (good) version has a timestamp of 0527 UTC or later.

Swift Response to Customer Needs

One of our valued Virtual-DBA customers encountered this issue and promptly contacted XTIVIA. Thanks to our support system and dedicated team, we quickly engaged with the customer and implemented the necessary fixes. This swift action ensured minimal downtime and a speedy return to normal operations. Our ability to respond promptly and effectively underscores the benefits of having XTIVIA as a reliable support partner during critical times.

Workarounds for Affected Systems

For Individual Hosts:

  1. Reboot the host to allow it to download the reverted channel file.
  2. If crashes persist:
    • Boot into Safe Mode or Windows Recovery Environment.
    • Navigate to %WINDIR%\System32\drivers\CrowdStrike.
    • Delete the file matching “C-00000291*.sys”.
    • Boot the host normally.
    • Note: Bitlocker-encrypted hosts may require a recovery key.

For Cloud or Virtual Environments:

Option 1:

  1. Detach the OS disk volume from the impacted virtual server.
  2. Create a snapshot or backup of the disk volume.
  3. Attach the volume to a new virtual server.
  4. Navigate to %WINDIR%\System32\drivers\CrowdStrike and delete “C-00000291*.sys”.
  5. Detach the volume and reattach it to the original server.

Option 2:

  1. Roll back to a snapshot or backup taken before 0409 UTC.

Effective Workarounds for Azure VMs

For Azure-related issues, leveraging Azure backups has been effective. Here’s a step-by-step guide:

1. Ensure You Have Recent Backups:

  • Verify that you have recent backups of your affected VMs via Azure Backup Center.

2. Restore the Disks:

  • Use Azure Backup Center to restore the disks from your backups. Ensure you select a backup from a point in time before the CrowdStrike update caused issues.

3. Unmount the Current OS Disk:

  • Access your Azure portal and navigate to the affected VM. Unmount the current (problematic) OS disk from the VM.

4. Mount the Restored Disk:

  • Attach the restored disk to the VM as the new OS disk. This action replaces the problematic disk with a clean, functional version from your backup.

5. Restart and Verify:

  • Restart the VM and verify that it operates correctly. Ensure all services are running as expected and that there are no lingering issues.

Conclusion

Navigating through the CrowdStrike outage has been challenging, but with the right approach, it’s possible to restore and stabilize your Azure-related VMs and on-premises workstations. Leveraging Azure backups and swapping OS disks has proven to be a reliable solution for cloud environments, while using system restore points or backups, and booting into Safe Mode has been effective for on-premises systems. Stay informed with the latest updates from CrowdStrike and Azure, and don’t hesitate to seek assistance if needed.

Our ability to respond swiftly and effectively highlights the importance of having XTIVIA as a dependable partner during critical times. If you’re already our client, you’ve experienced firsthand the benefits of our dedicated support. If not, consider the peace of mind that comes with knowing you have a team ready to address and resolve unexpected issues promptly.