Judgements about loyalty: A noise audit in the context of Swedish protective security vetting
(2025) In Judgment and Decision Making 20(e44). p.1-37- Abstract (Swedish)
- Judging an individual’s loyalty in security-sensitive roles is a high-stakes task, yet little is known about the extent and sources of variability in such judgments. This study examined how 58 participants with experience in personnel security assessment evaluated applicant profiles connected to five different countries. Each participant reviewed five protocols and judged whether the case contained information relevant to a personnel security clearance, then rated the applicant’s loyalty on a 7-point scale. Using Bayesian probit regression and an ordinal item response model with hierarchical structure, we analyzed both binary judgments and rating patterns, accounting for country of connection, applicant gender, and participant-specific... (More)
- Judging an individual’s loyalty in security-sensitive roles is a high-stakes task, yet little is known about the extent and sources of variability in such judgments. This study examined how 58 participants with experience in personnel security assessment evaluated applicant profiles connected to five different countries. Each participant reviewed five protocols and judged whether the case contained information relevant to a personnel security clearance, then rated the applicant’s loyalty on a 7-point scale. Using Bayesian probit regression and an ordinal item response model with hierarchical structure, we analyzed both binary judgments and rating patterns, accounting for country of connection, applicant gender, and participant-specific variability. Results revealed substantial between-participant variability (‘noise’) in how likely judges were to flag foreign ties as relevant. Pattern noise, reflecting idiosyncratic differences in how individuals interpret the same case, was evident in loyalty ratings. Connections to Brazil and Thailand were associated with systematically lower loyalty ratings and heightened disagreement between judges, reflecting both bias and pattern noise. Contrary to policy guidance, fewer than half of foreign connections were judged as relevant, and this tendency did not vary by participant gender. The findings underscore the risk of inconsistency in high-stakes assessments and highlight the need for structured conceptual calibration in personnel vetting. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/96c1fd4f-b0a4-413b-82b9-cca6266d95f3
- author
- Malmgren, Daniel LU and Dahl, Mats LU
- organization
- alternative title
- Lojalitetsbedömningar: En brus-revision inom svensk säkerhetsprövning
- publishing date
- 2025-11-17
- type
- Contribution to journal
- publication status
- published
- subject
- keywords
- personalsäkerhet, säkerhetsprövning, judgement variability, Bayesian modeling
- in
- Judgment and Decision Making
- volume
- 20
- issue
- e44
- pages
- 37 pages
- publisher
- Society for Judgment and Decision Making
- external identifiers
-
- scopus:105024481661
- ISSN
- 1930-2975
- DOI
- 10.1017/jdm.2025.10020
- language
- English
- LU publication?
- yes
- id
- 96c1fd4f-b0a4-413b-82b9-cca6266d95f3
- date added to LUP
- 2026-02-16 21:10:04
- date last changed
- 2026-02-17 11:19:39
@article{96c1fd4f-b0a4-413b-82b9-cca6266d95f3,
abstract = {{Judging an individual’s loyalty in security-sensitive roles is a high-stakes task, yet little is known about the extent and sources of variability in such judgments. This study examined how 58 participants with experience in personnel security assessment evaluated applicant profiles connected to five different countries. Each participant reviewed five protocols and judged whether the case contained information relevant to a personnel security clearance, then rated the applicant’s loyalty on a 7-point scale. Using Bayesian probit regression and an ordinal item response model with hierarchical structure, we analyzed both binary judgments and rating patterns, accounting for country of connection, applicant gender, and participant-specific variability. Results revealed substantial between-participant variability (‘noise’) in how likely judges were to flag foreign ties as relevant. Pattern noise, reflecting idiosyncratic differences in how individuals interpret the same case, was evident in loyalty ratings. Connections to Brazil and Thailand were associated with systematically lower loyalty ratings and heightened disagreement between judges, reflecting both bias and pattern noise. Contrary to policy guidance, fewer than half of foreign connections were judged as relevant, and this tendency did not vary by participant gender. The findings underscore the risk of inconsistency in high-stakes assessments and highlight the need for structured conceptual calibration in personnel vetting.}},
author = {{Malmgren, Daniel and Dahl, Mats}},
issn = {{1930-2975}},
keywords = {{personalsäkerhet; säkerhetsprövning; judgement variability; Bayesian modeling}},
language = {{eng}},
month = {{11}},
number = {{e44}},
pages = {{1--37}},
publisher = {{Society for Judgment and Decision Making}},
series = {{Judgment and Decision Making}},
title = {{Judgements about loyalty: A noise audit in the context of Swedish protective security vetting}},
url = {{http://dx.doi.org/10.1017/jdm.2025.10020}},
doi = {{10.1017/jdm.2025.10020}},
volume = {{20}},
year = {{2025}},
}