Using Key Performance Indicators
In the current data-driven environment, executives appreciate using key performance indicators and metrics to describe how well the organization manages its cyber program. Numbers can be beneficial in assessing initiatives, especially ones that heavily rely on complex technologies or unfamiliar topics. The comfort of benchmarking KPIs is familiar to many executives and is a primary reason to try to do the same with cybersecurity. Some challenges do exist, however, because what constitutes an appropriate or higher risk indicator can vary by organization. For example, one organization may limit its assessment to a few metrics focused on the number of breaches or records breached. Their cybersecurity budgets are often based on an industry benchmark in which organizations invest a comparable amount in cyber as their competitors, and in some cases, even lower. Others take a broader view and are interested in effectiveness, process, and ultimate results. For these executives, the development and monitoring of KPIs represent a powerful solution, as they can monitor progress.
In cybersecurity, defining KPIs typically involves benchmarking against what other organizations do, focusing on issues of paramount importance to the executive team, and determining indicators using data or a combination of these factors. These KPIs are then presented in some scorecard presentation. These presentations could be as simple as a one-page scorecard highlighting critical considerations using a heat map (red, yellow, and green). Others use a point system to score their practices (for example, Stanford University’s can be found at https://tinyurl.com/hvdh49pk). Some limit their reporting to the extent that the organization complies with and addresses corporate policy requirements, including KPIs related to how well the organization addresses supporting programs such as cybersecurity and vendor management. Those seeking a more comprehensive metrics program to support extensive investments can review the National Institute of Standards and Technology’s (NIST) recently revised (December 2024) two-volume “Measurement Guide for Information Security.” The first volume focuses on selecting metrics, while the second focuses on designing a measurement program to support the monitoring, analysis, and reporting (https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-55v1.pdf and https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-55v2.pdf)
The ISACA Journal (https://tinyurl.com/mazmm7wn) provides a high-level summary of using KPIs for security governance. The article offers excellent considerations on the challenges of trying guidance. The author, Andrej Volchkov, identifies the following categories for which KPIs can be developed:
- ▪ Financial metrics—Concerned with the financial impact of security controls.
- ▪ Maturity assessment—Evaluates the state of security as a whole or in some specific domain, such as cyber-security, continuity, application security, or management.
- ▪ Modeling—Intends to simplify analyses in the absence of accurate data.
- ▪ Assumption-based measurement—Consists of limiting and focusing measurement efforts.
- ▪ Progress toward objectives—Consists of establishing KPIs to measure the degree of progress toward a set of given objectives.
- ▪ Operational metrics—Articulates the effectiveness of security controls and processes.
- ▪ Cost analysis—Provides relevant indicators for security spending governance.
- ▪ Benchmarking—Makes it possible to compare the process in place with those of other organizations or best practices.
From a risk manager’s perspective, risk metrics can result in benefits and challenges for the organization. On the one hand, metrics can be used to obtain needed program funding and justify current investments. Metrics can also be helpful for outside parties, including insurance carriers, as well as to demonstrate due diligence for regulators and attorneys. Metrics can also demonstrate program strength to potential merger or acquisition suitors.
Risk managers face several challenges, the most prevalent being choosing relevant metrics. Generally, many risk managers would agree that monitoring outstanding vulnerability remediation efforts is critical. Yet statistics used to show activity (e.g., the cost of individual attacks prevented) may not be appreciated, given the time and effort needed to gather and prepare the information for what some believe are bureaucratic purposes used to justify the investment in required security from business leaders. Others believe cybersecurity program effectiveness cannot be assessed in a number or series of numbers comparable to the analysis performed to determine the organization’s financial performance.
Maturity Assessment Provides a Popular and Balanced Approach
Of the categories identified above by Volchkov, maturity assessment represents a popular approach to assessing and reporting on the effectiveness of a cybersecurity program. The Software Engineering Institute (SEI), a government-supported project at Carnegie Mellon University, developed maturity guidelines to assess the software development process and quality. Eventually, they expanded these processes and methodologies into technical security areas. Through its COBIT information technology framework, ISACA introduced maturity models to the audit and accounting communities. Reputable consulting firms such as Gartner and Forrester also leveraged these models in their assessment and related benchmarking tools provided to clients.
Eventually, the NIST adopted a maturity model (four tiers) for its widely adapted Cybersecurity Framework (https://tinyurl.com/4h8548xv). Per the framework, “the Tiers describe a progression from informal, ad hoc responses to agile, risk-informed approaches and continuously improving. Selecting Tiers helps set the overall tone for how an organization will manage its cybersecurity risks” (p. 7). The four tiers (from worst to best) are Partial, Risk Informed, Repeatable, and Adaptive. The tier-level approach provided a practical scale and, in some cases, granularity for non-technical users (e.g., executives) to appreciate how their organization manages cyber risk.
Executives must confirm the maturity scale used when overseeing or assessing cyber efforts based on maturity. For example, as discussed above, the CSF uses a four-tier scale. Yet COBIT 2019, ISACA’s premier IT governance framework, uses a six-tier CMMI-based process capability scheme. These tiers range, from worst to first: Incomplete, Initial, Managed, Defined, Quantitative, and Optimizing. Those responsible for governance should confirm the scale used for actual and targeted performance.
Measure What Matters Most
Each organization is different. As a result, what matters and gets measured for each is different as well. It is often said that what gets measured gets done, yet the cost of obtaining the measures should not exceed the benefits derived from using the measures. In this author’s opinion, a combination of measures focusing on board-approved policies (metrics) and the maturity of cyber security programs provides the optimal mix for many small to mid-size organizations.
