A problematic update from Crowdstrike, a well-regarded security software company, caused major unexpected disruptions to some of its customers. The patch, intended to enhance security, led to significant disruptions for several users, including public-facing companies. This incident affected customer service delivery and resulted in substantial media coverage, lost business, and the potential for lawsuits.
Risk management practitioners took note of the situation. They considered previous instances from other vendors in which patches issued by the vendor with the intention to enhance security or reliability had the opposite effect due to issues arising when the patch was applied. Audit committees and governance executives of companies using Crowdstrike played a crucial role in determining the business impacts on their organizations and what controls existed to prohibit the damage and potential lawsuits they were hearing about in the media. Of special concern to those responsible for governance were reports that issues in Crowdstrike’s customer infrastructure may have added to the complications resulting from the patch. They also wanted to ensure that the organization was doing all it could to minimize and prevent similar types of incidents from occurring in the future.
Automated Patching Is Usually Recommended
In its blog post, “Understanding Patches and Software Updates,” the Cybersecurity & Infrastructure Security Agency (CISA) defined patch management as “software and operating system (OS) updates that address security vulnerabilities within a program or product. Software vendors may choose to release updates to fix performance bugs, as well as to provide enhanced security features” (https://tinyurl.com/bdhw3a3n). The CISA recommends best practices for software updates, including “enable automatic software updates whenever possible. This will ensure that software updates are installed as quickly as possible.” As a result, many organizations, especially those with limited resources who want to minimize the security exposure from not applying patches, will implement the recommendation from the CISA.
Many consumer electronics users are already familiar with the concept of automatic software updates. For example, Microsoft Windows having the automated update feature turned on by default has given many users first-hand experience of how a patch (update) works. Although sometimes it may appear at an inconvenient time or, in some situations, users may be forced to process the patch, the benefits of preventing attacks outweigh the costs of inconvenience. Per Microsoft, “With automatic updating, you don’t have to search for updates online or worry that critical fixes or device drivers for Windows might be missing from your PC. Windows Update automatically installs important updates as they become available” (https://tinyurl.com/5eumdvan). Consumers do not usually test updates as they rely on Microsoft’s (the vendor’s) quality controls to mitigate problems arising from the update. Despite the vendor’s efforts, the update may fail or cause damage due to unique software configurations on specific hardware. Some of this risk is reduced because the complexity of consumer configuration is relatively predictable, thereby helping to enhance the probability of patch update success. The risk of not testing is considered acceptable as most consumers would fail to implement the patch, thereby increasing the likelihood of the vulnerability for which the patch was developed being exploited.
Not as Easy for Organizations
Organizations face a difficult decision. Like consumers, they are encouraged, for the same reasons, to apply automated updates. Yet, given the increased sophistication of technology resources used by an organization (no matter their size) and the potential diversion of assets across geographies, there is a greater risk for a faulty patch to have unintended consequences that could impact an organization’s network, related systems, and (of most significant concern) delivery of services to customers, including but not limited to web commerce initiatives.
The realistic challenges facing organizations, as well as the reasons why a quick implementation is recommended, are described by the National Institute of Standards and Technology (NIST) in “Guide to Enterprise Patch Management: NIST Special Publication 800-40r4” (https://tinyurl.com/2khj97m6, pg. 8).
What has made enterprise patch management tougher recently is how dynamic and dispersed computing assets are, as well as the sheer number of installed software components to patch. In addition, patch management processes and technology take different forms depending on the type of assets (e.g., IoT, mobile, cloud, traditional IT, virtual machines, containers). The result is that many organizations are unable to keep up with patching. Patching often becomes primarily reactive (i.e., quickly deploy a patch when a severe vulnerability is being widely exploited) versus proactive (i.e., quickly deploy patches to correct many vulnerabilities before exploitation is likely to occur).
So, the challenge facing organizations is to process the automated patch and assume the probability of experiencing that the risk of patch failure is lower than the vulnerability of the exploitation for which the patch was designed to mitigate. Many smaller and midsize organizations do not have the budget or personnel and rely on the automatic updating strategy. Mainly, that works unless there is something wrong with the patch like, for example, what happened with the Crowd-strike patch. The companies are then at the mercy and priority of the vendor to reverse the patch and bring their technology back online.
What Can an Organization Do?
In most situations, organizations seek to demonstrate that, despite limited options, they performed sufficient due diligence to show they made reasonable efforts to minimize disruptions and took the most appropriate strategies according to their risk tolerances. Even if an automated update process is relied upon, organizations could benefit from the following methods:
Backup everything. This is a timeless and critical strategy that enables organizations to at least revert to the environment prior to the processing of the questionable patch. Even better, organizations should have a tested recovery plan to facilitate actions and decisions if problems occur.
Maintain a current inventory. Understanding how a patch may impact the network and system infrastructure is critical to minimizing risks and damage. This includes how different software components interact with each other.
Understand unique configurations and setup. Organizations often use common technology configurations and network designs to manage their infrastructure. Yet, business needs may sometimes dictate that the organization use customized configurations or designs. Another challenge is poorly maintained technology or technology that exceeds end-of-life recommendations. These types of deviations can significantly impact the success of patch updates.
Implement change control processes. To the extent that an organization is able, having a test environment for patches or for high-risk changes (impact to infrastructure) should be considered. As discussed above, this may not always be feasible and, in some cases, may require the suspension of automated updating. Another strategy is to delay an update in order to better understand what other organizations are experiencing.
Perform a risk assessment. Understanding the risks that the patch is trying to prevent, and the risks the organization faces given its unique needs, is crucial to effectively managing the reality of patches. The risk assessment can also identify other impacts on the organization that technology specialists may not have considered, especially those related to customer relationships.
Understand vendor terms and conditions. Organizations should review (including legal review) all contracts and supplements (e.g., terms and conditions) before signing an agreement. In some cases, customers may be unable to negotiate changes to necessary contracts or services. They should consider the impacts and compensating factors needed during the risk assessment in these situations. When Crowdstrike’s customers first considered lawsuits against the vendor, a reading of the terms and conditions that customers had agreed to either limited Crowdstrike’s potential liability or the type of legal action that could be taken by the signatories to the contract.
Choose the Best Strategy
Patching is not a one-answer-fitsall strategy. Each organization will need to understand its risk appetite, its tolerance for downtime, and security exposure. As with similar risk-based decisions, organizations should document their risk assessment and rationale for their decisions, should they need to answer to stakeholders.
