Three important lessons from the Microsoft / CrowdStrike outage

Last Friday (17 July 2024) saw a large-scale IT outage which led to significant global disruptions. The outage affected certain Windows devices using CrowdStrike’s software and was caused by an update they rolled out causing these Windows systems to crash. More than 8.5 million Windows devices were affected across the globe as a result of the faulty update CrowdStrike rolled out, causing severe disruptions to public services and other essential services internationally. Whilst this figure amounts to less than 1% of all Windows devices, the resultant fallout from these system outages was significant.  Thousands of flights were cancelled (a greater number of which were delayed), supermarkets were unable to take contactless payments and additional impact was posed to banks, healthcare, media and other transport providers.

Microsoft explained this disproportionate impact in a message from their Vice President David Weston, published on Saturday (20 July 2024), stating that “while the percentage was small, the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services.” CrowdStrike (whilst not a household name) is a cybersecurity company that supplies anti-malware endpoint security solutions to organisations and is heavily relied upon globally by providers of critical infrastructure. CrowdStrike have published a Remediation and Guidance Hub with details of the incident and its fixes.

The resultant fallout from the incident is a clear demonstration of the fragility of our digital infrastructure. With such a global impact from a single faulty update, what lessons can organisations take away to bolster organisational resilience in the face of such incidents? And how can they minimise the likelihood of impact from similar events going forward?

Lesson 1: Ensure that all updates are thoroughly tested and quality checked before their deployment.

CrowdStrike’s stock value has plummeted by 25% over the last 5 days, as a result of the faulty update. Had CrowdStrike implemented the appropriate governance measures and thoroughly tested and quality checked any new code/update before deployment, the incident would have been prevented from ever taking place. Ideally, organisations receiving third-party updates should also have equivalent governance controls in place to ensure that third-party software updates are properly tested before being deployed at scale in the production environment (where possible and feasible).

However, this was not an option for the CrowdStrike update. This update was for an Anti-Malware (AM) application. These applications are granted a higher level of systems access that allows them to automatically push out updates without user permission. Theoretically, this allows for a prompt response to emerging threats. Nevertheless, applications being granted this level of systems access is an exception to the rule, granted where there is a proportionate justification (such as with AM software where it is necessary to be able to adapt to new threats).

Generally, third-party software updates are not automatically rolled out to devices by the software provider, and instead organisations can test such updates for bugs or vulnerabilities before deploying them at scale across their organisation. Organisations are therefore able to test the update, for example by rolling it out on a small percentage of their devices and checking for any issues or system outages before deploying the update on the rest of their devices. Alternatively, they may use sandboxes, isolated testing environments, which allows them to test the update on devices without impacting the rest of their network.

Where an organisations’ risk appetite is limited, these governance controls within the process for developing and deploying new code/updates (for internally developed, and third-party software where possible) are vital measures to bolster resilience.

Lesson 2: Prioritise your supply chain security controls.

Another key learning which can be taken away from the CrowdStrike Incident is the ever-increasing importance of supply chain security controls. This sentiment was echoed in the above-mentioned message from Microsoft’s Vice President, who stated that the incident was a demonstration of the “the interconnected nature of our broad ecosystem — global cloud providers, software platforms, security vendors and other software vendors, and customers”.

This interconnection calls for enhanced governance measures and technical restrictions to ensure the security of supply chains. Such controls should include organisations:

• Ensuring that there are appropriate due diligence checks conducted on any third parties (and where possible sub-contractors utilised by third parties) prior to engaging them.
• Setting and communicating the minimum security requirements for third parties, and where appropriate, including these requirements in contractual clauses.
• Conducting appropriate audits and testing on third parties.
• Forbidding staff from downloading unapproved applications and enforcing these rules with technical restrictions.
• Restricting the access and permissions given to third-party applications.
• Keeping up to date asset registers of all third-party application in use.

Lesson 3: Ensure your organisation has appropriate and tested Incident Response plans.

Last, but not least, the CrowdStrike incident demonstrates the importance of having robust and well-tested Incident Response (IR) and Business Continuity (BC) / Disaster Recovery (DR) plans in place. Having such processes in place allows organisations to respond promptly when faced with such incidents to mitigate the potential damage suffered.

Organisations with strong IR & BC/DR plans would have been best positioned in the face of the CrowdStrike incident. Such plans would have ensured their senior leadership team was promptly provided the critical information to allow them to take any necessary key decisions to mitigate impact. These plans would have included well-considered communications strategies and templates enabling them to communicate quickly and efficiently with their customers, thus protecting customer trust and loyalty. BC/DR plans could also have been relied upon to prioritise the continuity and recovery of the organisations most critical services, thus making the most efficient use of resources.

Developing, testing and improving such plans is a core step in ensuring the resilience of your organisation. The maturity of an organisation’s IR plan and BC/DR plans has a direct impact on how promptly and efficiently the organisation can respond and recover from such an incident, minimising the subsequent impact.

Mitigating future risks

Whilst the CrowdStrike - Microsoft Incident may have highlighted the single points of failure within our digital infrastructure and the significant consequences where they are exposed, this incident can be viewed as a reminder of the importance of implementing solid controls to prevent and mitigate such impact.

Our data protection and cyber-risk team is experienced in supporting organisations to develop robust cybersecurity controls and improve their incident response strategy. If you would like to find out more information on how we can help your organisation, please contact our advisers.