The role of catalogues of threats and security controls in security risk assessment: An empirical study with atm professionals

[Context and motivation] To remedy the lack of security expertise, industrial security risk assessment methods come with catalogues of threats and security controls. [Question/problem] We investigate in both qualitative and quantitative terms whether the use of catalogues of threats and security controls has an effect on the actual and perceived effectiveness of a security risk assessment method. In particular, we assessed the effect of using domain-specific versus domain-general catalogues on the actual and perceived efficacy of a security risk assessment method conducted by non-experts and compare it with the effect of running the same method by security experts but without catalogues. [Principal ideas/results] The quantitative analysis shows that nonsecurity experts who applied the method with catalogues identified threats and controls of the same quality of security experts without catalogues. The perceived ease of use was higher when participants used method without catalogues albeit only at 10% significance level. The qualitative analysis indicates that security experts have different expectations from a catalogue than non-experts. Non-experts are mostly worried about the difficulty of navigating through the catalogue (the larger and less specific the worse it was) while expert users found it mostly useful to get a common terminology and a checklist that nothing was forgotten. [Contribution] This paper sheds light on the important features of the catalogues and discuss how they contribute into risk assessment process.