How well is your counter insider threat (C-InT) program working? How do you measure its effectiveness? If it’s like most C-InT programs, your organization struggles to employ metrics that accurately assess the contributions of its C-InT program and its value to the organization. We generally use the terms “measures of effectiveness” (MOE) and “return-on-investment” (ROI) to capture these metrics. In this blog, I briefly describe some of the challenges and desired characteristics of InTP metrics.
A Science-Based Approach to Assess Measures of Effectiveness
There’s a story of a man who claps his hands every ten seconds. When asked why, he says:
“I’m clapping to scare away the elephants.” When we point out to him that there are no elephants around, he says, “See? It works!”
This example of non-scientific or “magical” thinking is not helpful in assessing the impact of a C-InT program.
One type of science-based approach focuses on business processes, reflecting operational performance or throughput. It’s useful to compare operational performance before and after deploying methods or decision-analytic tools such as AI or expert systems applications that augment human analysts. An informative INSA whitepaper describes numerous operational MOEs including:
- number of cases generated per week/month
- average time required to detect a threat
- number of cases processed per week/month
- average time required to resolve a case
- number of cases assigned per analyst per week/month.
Going forward, tracking one or more of these process and workflow metrics provides a formal way to assess the effectiveness of your program over time.
Cogynt Enhances and Informs Program Effectiveness
In previous blogs, I and other Cogility team members have described the Cogynt streaming intelligence platform for C-InT analysis and decision making. Cogynt’s integrated Case Management system tracks workflow and displays case details, risk history and traceability using customized, tailorable displays; and Cogynt’s integrated Superset tool provides important business process data and insights including time-based plots of business process metrics like those listed above.
These measures provide excellent sources of information about the efficiency of one’s C-InT program in processing potential threats; but of course, this is but one (important) challenge facing insider threat analysts who are overwhelmed by ever-increasing volumes of streaming data. Much time and resources are spent gathering and examining incident evidence from diverse sources including security information event management (SIEM), user behavior analytics, performance or personnel records, security or travel records, etc., to identify the very rare at-risk individuals who will become the subject of a case—this is referred to as the triage process.
A typical host/network monitoring system can generate billions of events per week, but while the sheer volume of data has been increasing rapidly over time, the staffing of qualified analyst resources has not kept up with the demand. Thus, most C-InT programs require automated decision-making support to address this information processing deficit. As depicted in the figure at right, effective automated threat analysis and case management support, such as provided in Cogility’s Cogynt platform, allows a finite-resourced C-InT team to meet the challenges of ever-increasing volumes of data—it’s a force multiplier!
Evaluating Return-on-Investment
To maintain the scientific integrity of the ROI measurement process, the selection of ROI metrics must satisfy the criterion that measures taken before and after deployment of the model or tool being assessed must gauge the same things—we adopt ROI metrics that may be acquired in both a baseline period and post deployment phases.
Metrics used in ROI studies are largely the same as measures defined for MOE assessment. Examples of ROI-related measures include various counts (e.g., number of cases resolved per week, number of cases assigned to analysts per week, … ) and/or various time measurements (e.g., average time case is in threat analysis).
Once the measures are established, the C-InT team must begin the record-keeping process of collecting these measures during a baseline period prior to deployment of the C-InT tool(s). If historical records are available, that’s all the better.
Then, for a period of post-deployment, these same measures must be recorded—ideally, the automated support (case management) tools will collect these data without human intervention. Once the baseline measures and the post-deployment data have been collected, the ROI analysis is straightforward. For example, for each of the selected ROI metrics there will be baseline data and one or more sets of post-deployment data available that can be compared. The degree of improvement may be identified as a ratio or the obtained data can be transformed into cost figures by converting hours to labor costs.
Conclusions
In this blog, I have briefly reviewed science-based approaches to assessing the impact of C-InT program methods or tools. I described several MOE metrics that convey business process improvements (e.g., throughput), and then I discussed science-based methods for establishing ROI metrics.
Because Cogynt addresses the full range of challenges faced by C-InT programs—encompassing intelligent support for triage, case analysis, case management, and business process/program oversight—the Cogynt platform offers a complete C-InT solution that meets the needs of an ever-increasing and changing insider threat landscape. For more information, please see the Cogility website.
