Introduction
At the 2025 DSI Symposium, I gave a presentation that argues for greater integration of historically distinct approaches to risk analysis: the idea of combining qualitative and quantitative modeling methods to exploit each of their advantages. Here’s a synopsis of this presentation.
Qualitative models use descriptive, non-numerical constructs to represent processes or relationships. They produce comparative predictions such as “y increases with x” or causal hypotheses like “B is more likely to occur if A is observed.” Qualitative decision-making models incorporate descriptive data such as expert opinions, contextual knowledge, experience, and subjective factors. They are well suited when exploring complex phenomena and in developing theories or hypotheses. A limitation of qualitative models is the possible influence of the researcher’s perspective and biases.
Examples of qualitative models are the SOFIT knowledge base of insider risk indicators1 and Eric Shaw’s Critical Pathway to Insider Risk.2 A premier example of a qualitative modeling approach applied to risk assessment is the Structured Professional Judgment (SPJ) process. SPJ tools provide a systematic, fact-based assessment of contributing factors to arrive at an overall risk judgment and a plan to mitigate the risk. Major steps in the SPJ process include:
- Information gathering to determine the presence of risk factors (behaviors that may indicate planning, preparation, or intent) and protective factors (behaviors or conditions that may tend to decrease risk)
- Determining the relevance of these factors in assessing where the subject is on the pathway to insider threat
- Developing a risk formulation that explains the available evidence and identifies the most likely risks
- Developing risk management strategies such as interventions and safety measures to protect targets
- Preparing the final judgment that documents the determinations and reflects the analyst’s degree of concern or assessment of the level of risk (e.g., “none”, “low”, “medium”, and “high”).
Steps in Structured Professional Judgment
Quantitative models generate numerical predictions of phenomena being studied. My research and presentations on insider risk assessment have been heavily oriented toward quantitative, predictive analytic models. I have discussed the development and testing of mathematical models for the propagation of insider risk. Typically, the parameters used in these predictive analytic models are informed by expert judgments.
The Cogynt model is a premier example of a quantitative approach that reflects expert judgments. Its Hierarchical Complex Event Processing, or HCEP, is a pattern-based representation of expert knowledge that is, by design, a model of the expert’s decision analysis process. For insider risk assessment, it ingests available behavioral and technical data continuously and uses HCEP to identify relevant insider risk indicators based on the SOFIT taxonomy. Further pattern-based analyses, defined in the HCEP model, propagate the risk up through the hierarchy to produce behavioral profiles and risk scores for all the threat behaviors of interest.
Cogynt Decision Intelligence Platform
The predictive analytic modeling approaches that I have studied should not be confused with actuarial models, which represent a rather different approach to threat assessment. Actuarial models were advocated for violence risk assessment as an alternative to clinical judgments. Proponents argue that these test instruments are not subject to the limitations of unaided human threat assessment professionals who may have varying experience, cognitive abilities, resources, or biases. On the other hand, SPJ advocates remind us that actuarial methods depend on statistical properties or averages in the target population. As such, actuarial tools do not recognize the uniqueness of an individual case that will be evident to a human expert. The SPJ approach emerged as an alternative framework that addresses the limitations of both individual clinical judgments and actuarial methods.
Comparing Approaches
One might ask how these methods compare in terms of performance. This is a complicated question because they have been evaluated using different contexts and statistical methods. An important metric for SPJ concerns how well it lives up to the goal of increasing consistency of output, and it produces relatively high scores for inter-rater reliability. Metrics that focus on predictive validity are more problematic for SPJ methods, since their performance as measured by area under the ROC curve has yielded only fair results. For quantitative models, as I have reported in several papers and talks, we have found that some models, such as Cogynt, perform very well with predictions showing high agreement with expert judgments of risk, which are taken as proxies for ground truth.
Here's another way to compare the contributions of the different approaches:
- Because actuarial models focus on population characteristics, this is not the most useful approach for assessing the risk of an individual case.
- Qualitative models rely on expert judgments, and their strengths include the use of rigorous procedures to encourage consistency of outcomes that also capitalize on incorporating the expert’s insights on a specific case.
- Predictive analytic models – especially those that apply quantitative methods that are informed by expert judgments – provide rigorous, testable predictions.
- AI/Expert System models allow domain experts to define patterns of behavior that represent what to look for in dynamic data streams.
Computer based methods that integrate AI/Expert System models with predictive analytic models provide a knowledge-driven approach that exploits the experience of experts in a transparent decision intelligence environment (Gartner, 2024) that is distinguished from purely data-driven machine learning models.
I would like to suggest that a more fully integrated approach would combine predictive analytics, AI/Expert Systems, and qualitative methods such as SPJ. This integration of qualitative models that reflect expert judgments and insights with quantitative, AI/Expert System based predictive analytic models represents a combination of approaches that may provide the most powerful risk management solution.
How the Cogynt Predictive Analytic Platform Can Enhance SPJ
Now I’d like to briefly point out some of the ways that Cogynt can support and strengthen the SPJ approach. Here I will use as an example the widely applied WAVR-21 tool3 that assists in assessing workplace violence. This method first asks the analyst to examine the evidence or input data and decide whether or not each of 21 risk factors is present. This information gathering and risk indicator accounting can be supported by the Cogynt model, which is built upon an expert-informed and empirically calibrated set of potential risk indicators—a mapping between the SOFIT PRIs and the WAVR-21 indicators can implement this strategy.
WAVR-21 Factors can be Mapped to SOFIT PRIs
Looking at the next step in the SPJ process, we can see that Cogynt not only helps to identify the risk indicators, but it also supports their mappings to threat behaviors. The steps that follow produce a narrative framework that describes the nature of the threat, and then describe possible mitigation strategies, yielding a final judgment and a rating of the degree of concern. These steps can be facilitated by Cogynt’s hierarchical complex event processing that reveals complex relationships among the indicators and behaviors. Cogynt’s integrated AI Chatbot can collect and summarize the information and provide reasons why certain conclusions were reached.
In conclusion, both qualitative and quantitative modeling approaches utilize human judgment. Quantitative models like Cogynt represent expert thinking, and its parameters are informed by expert judgments that are tailored to the organization’s mission and priorities. Qualitative models are informed by expert judgments but can also be supported by the output of predictive analytic models.
There is no need to implement one method exclusively: Leveraging the most effective features of each method can yield joint cognitive systems – Decision Intelligence solutions that perform better than either approach alone.
You can view a video of the presentation here: https://cogility.com/resources/insider-risk-webinar-human-judgment-and-ai/
References
- Greitzer, FL, J Purl, YM Leong, & DE Becker. (2018). SOFIT: Sociotechnical and Organizational Factors for Insider Threat. IEEE Symposium on Security and Privacy Workshops, 197-206.
- Shaw, ED & L Sellers. (2015). Application of the critical-path method to evaluate insider risks. Studies in Intelligence, 59(2), 41-48.
- Meloy, JR, SG White & S Hart. (2013). Workplace Assessment of Targeted Violence Risk: The development and reliability of the WAVR-21. Journal of Behavioral Science, 58(5), 1353–1358. https://doi.org/10.1111/1556-4029.12196
