Outage in Cyberspace: How a Cybersecurity Update Brought a Bevy of Industries to Their Knees

Everyone not living in a cave has heard of the news: in an unexpected turn of events, a relatively simple update to CrowdStrike's cybersecurity software has led to widespread disruptions across many different industries worldwide, causing computers that operate Windows to crash - and not just crash, but display the much-dreaded blue screen of death (or BSOD, in techie terms). As mentioned, across industries, companies from all over the globe have found themselves unable to reboot their systems, seemingly paralyzed by this unforeseen glitch. One high-profile casualty of the incident included Sky News, which, due to the outage, has not been able to broadcast at all. Concerned users have even taken to Reddit to voice their concerns, insights, opinions, and in many cases, humor into the situation, with one user commenting, “Oh d*mn, what a day to be in IT. “ Others, of course, flooded forums with reports of the issue, sharing their frustrations and seeking solutions.

If you arrived at work last Friday to find chaos and malfunctioning computers with colleagues wringing their hands in (pretend) despair, rest assured, you were not alone. Let’s get a detailed look at what happened - and what steps you can take to address the issue, especially if – God forbid – it happens again next time.

What happened? Identifying the gist of the problem

The root cause of the problem lay with CrowdStrike's Falcon Sensor product, a critical component of their cybersecurity platform designed to prevent breaches and other malicious activities. CrowdStrike's engineers actively worked on resolving the issue, which they initially believed to be a simple – but problematic - update. However, as clarified by “Brody,” a Reddit user and the director of CrowdStrike Overwatch, it turned out to be a faulty channel file rather than a full-fledged update.

The impact of this IT outage has been extensive, as you may have seen. Airports and broadcasters worldwide have reported issues. In the US, as in Hong Kong, Sydney, and other airports, planes have been grounded, while in the UK, train services have been disrupted, and boarding scanners at Edinburgh Airport in Scotland have also been affected. Microsoft has been quick to acknowledge the situation, stating that they took "mitigation actions" after the service issues began at around 6 pm Eastern Time. Furthermore, the company is investigating problems with its cloud services along with several services and apps impacted by the incident.

Impact on the casino sector

But it wasn’t just flights and banking that were affected – the casino sector suffered from the effects of the outage, too. For instance, it was reported that customers had difficulty accessing the Coral and Ladbrokes website, which are both under Entain. Significantly, several other land-based casinos (albeit small ones) couldn’t get their slot machines to function. In the case of Ladbrokes and Coral, the portals displayed a message, stating “We’re experiencing temporary service interruption, and our team is working to restore normal service quickly.”

In another statement released by BetMGM and MGM Resorts, it said that they were also experiencing technical issues due to global IT outages. And TAB, Sky Racing, and Sportsbet in Australia also released statements, saying that 3rd party technical issues were disrupting their betting services along with customer support services.

The Las Vegas Review Journal reported that the outage caused a ripple effect in casinos in Vegas, shutting down slot machines at several brick-and-mortar establishments, and some casino-goers were even unable to cash out.

Official statements and initial fixes

In response to the worldwide crisis, a spokesperson from Microsoft said that they were fully aware of the situation from the beginning and even recommended to customers to follow the guidance provided by CrowdStrike.

In the meantime, CrowdStrike’s Chief Executive Officer, George Kurtz, took to X (formerly Twitter) to confirm that the issue was not a cyberattack but was a botched update instead. He went on to say that, contrary to reports, it was not a security incident in any way. He assured that the issue had, in fact, been identified, isolated, and a fix deployed. Kurtz further advised customers to consult the official support portal for updates and to communicate with CrowdStrike personnel using official channels.

What to do next?

But although the CrowdStrike CEO himself had stated that the issue has been resolved, addressing the problem may not be as straightforward as implied. For example, while a workaround exists, it involves manual intervention, which can be time-consuming and admittedly impractical for large organizations. Here’s the recommended workaround, according to Reddit user Brody:

1. Boot Windows into Windows Recovery Environment (WRE) or Safe Mode.
2. Navigate to C:\Windows\System32\drivers\CrowdStrike.
3. Locate and then delete the file matching "C-00000291*.sys".
4. Reboot normally.

However, this solution requires applying the fix manually to each affected system, which can be a monumental undertaking for larger enterprises. A director of a major cybersecurity firm, Adam Harrison, also emphasized the difficulty of resolving the issue once systems are stuck in a ‘reboot loop.’ He stated that if they do manual fixes, this is going to take time to apply. Simply put, CrowdStrike can't push a new update remotely just to fix the issue. In short, the update will have to be fixed manually on each system.

The CISO at Cyjax, Ian Thornton-Trump, suggested that CrowdStrike might develop a tool to apply the fix at the disk level using bootable media. But while this strategy could expedite recovery for some, it still doesn't offer a fully remote or scalable solution. For organizations with critical systems, restoring from backups or utilizing Microsoft's shadow copy feature might be necessary.

Communication and support

In such crises, effective communication is vital. CrowdStrike must ensure that information about the fix is disseminated quickly and accurately. Harrison noted that they can only communicate that fix as quickly and widely as they can and assume that the update is already down, so any systems which haven’t been able to update for whatever reason shouldn't still get pushed with a bad update.

But CrowdStrike’s proactive engagement with affected customers through their support portal and official channels will be crucial in mitigating the fallout from the incident. Thornton-Trump emphasized the importance of immediate response, stating that CrowdStrike will do their best to pull the update and instruct agents not to update until they can get it sorted.

Moving forward from a disastrous time

For many businesses, the immediate focus will be on restoring normal operations as we know them. However, this incident also serves as a stark reminder of the potential risks associated with software updates and the importance of more robust planning for contingencies. Additionally, organizations may need to reassess their update deployment strategies and ensure that they have effective rollback mechanisms and backup procedures in place to minimize the disruption in the event of a similar issue occurring in the future.

As CrowdStrike continues to address the fallout from this update (incidentally, its shares, along with Microsoft’s, went down significantly as a result of this glitch), the broader tech community will be watching closely and monitoring the market like hawks. The lessons learned from this incident will undoubtedly inform future practices and policies to better safeguard against such widespread disruptions.

In the meantime, if your organization is still affected by the CrowdStrike issue, follow the recommended workaround, stay in close communication with CrowdStrike support, and monitor official updates to stay informed about the latest developments and solutions.