The first step towards implementing a successful suite of security metrics is to understand the nature of security metrics. The Oxford Dictionary of English defines the word “metrics” to mean:
“a set of figures or statistics that measure results.”
We can therefore view security metrics as referring to:
“a set of figures or statistics that measure the effectiveness of a security capability or control.”
These definitions show that there is a difference between a measurement and a metric. A metric usually consists of a combination (or series) of measurements which together provide a discrete status of the behaviour or control being monitored. These metrics can then be monitored over a period of time to show whether performance is improving, deteriorating or remaining stable over time.
A well-designed security metrics programme offers a number of benefits, including:
- providing senior management with visibility of the outcomes from existing investment in security; justifying further investment where necessary
- providing advance notice of required changes to security controls via trends in security metrics that indicate negative impacts on the business
- enabling cross-charging for security services via meaningful Operating Level Agreements (OLAs)
- providing worthwhile accurate input to compliance and audit exercises
- identifying unnecessary security controls that either hamper business effectiveness or cost too much for the protection that they offer
- monitoring defined risks to the business – particularly relevant to supporting innovative new ways of working, such as working ‘in the cloud’.
Organisations evolve; new opportunities arise, acquisitions and divestments happen, strategies change. Metrics designed to cater for the business needs of 18 months ago may no longer be appropriate today or tomorrow. Metrics must be maintained to ensure that the metrics solution is able to evolve alongside the organisation.
Now, implementation of security metrics is not without risk. One issue that can adversely affect a metrics programme is that stubborn cause of many security misfortunes – human beings. For example, how would an organisation make use of a metric measuring the number of reported security breaches in specific business areas? Should a manager be rewarded for reporting breaches (and perhaps inadvertently encourage more breaches)? Or should the manager be punished for suffering the breaches in the first place? In the latter case, the individual will almost certainly become more reluctant to report security issues - definitely not a desired outcome. Beware the law of unintended consequences!
Another potential pitfall is the production and dissemination of metrics information that is of no relevance to the organisation; it can be very tempting to collect information simply because it can be collected rather than because it should be collected. This is a symptom of a wider problem of failing to scope the metrics programme adequately but the consequences can be catastrophic in terms of:
- loss of stakeholder engagement
- losing sight of the valuable information amongst the noise
- increased cost and performance impact
However, I believe that a well-designed, business-focussed and tightly scoped security metrics programme can be an invaluable mechanism for fostering better understanding of the value of the security controls in place at an organisation. This enables our clients to identify those controls which either offer real business value (e.g. fraud prevention) or operational excellence (e.g. spam filters) and to distinguish these controls from those which are failing or redundant. You can start to make a business case for the metrics programme through the retirement of tools or processes that demonstrably no longer add any value to the business.
Furthermore, metrics can be used as a management tool to incentivise both the security function, and the business as a whole through performance management processes, to improve the security regime within the organisation.
There is little alternative to the use of security metrics for those organisations that wish to manage their security risks in an objective, empirical manner. Without visibility of the effectiveness of their controls they are forced to default to blind trust in the capabilities of their staff and implemented security products. Some organisations will be fortunate enough to maintain highly competent security functions, highly disciplined staff and top notch security products. Do you know if your organisation is so fortunate?