I am proud of my Board of Directors. They understood their accountability, and that cybersecurity needed to be a top focus of their oversight function. They dedicated the time to getting up to speed, and brought in an excellent outside consultant, who educated them not only on the risks, but helped them articulate their requests for me. They expressed their top concerns (How aligned are we with best practice frameworks? How effective is the staff security awareness training? How well do we do on external testing and audits? etc.), and asked me to produce metrics to track our performance against them.
As we sat down to consider the data points that best addressed their questions, we had a few guiding principles:
- Transparency – the data sources and methodology used to generate the metrics should be clear
- Repeatable – we should be able to track these measures consistently over time
- Reproducible – if asked, we should be able to re-generate the metrics for a previous period and produce the same result
We had been tracking, logging and reviewing metrics internally for some time, so creating tailored reports for the Board’s concerns was fairly straight forward – except for one request:
“How effective are our cybersecurity controls?”
That is simple and understandable question – but an unexpectedly tricky one to express.
We had a robust testing program, but third party assessment results were already being presented as a standalone metric – and were still point in time measures.
Producing the raw blocked attacks was meaningless – they weren’t interested in thousands of things we knew we blocked. They wanted to know whether there were attacks we didn’t detect or couldn’t prevent.
For some time, we produced a ratio metric (attacks blocked at perimeter vs. internal network detections) – showing that any internal detections were miniscule compared to those prevented at the edge – but as our tools and networks changed, this was not a repeatable or consistent value.
If we could only run continuous red team exercises, leveraging MITRE ATT&CK to ensure we covered a wide range of tactics and techniques – across varying vectors – both north \ south, as well as lateral movement – and of course, emerging and newsworthy threats would also have to be addressed – and if we could do all that in repeatedly in cycles – I could provide our Board the assurance they desired.
Well, hiring such an army, with the varying and expert skills sets required was not possible – even if I could find the range of talent required.
Thankfully, Breach and Attack Simulation (BAS) emerged, and while still developing in the marketplace, the players and products are now mature enough for real deployments and reliable reporting.
BAS is a force multiplier. We can run tests with thousands of samples, against our real defenses and provide meaningful insights to my team and our stakeholders. Even dedicated red team testers can benefit by automating a large quantity of attacks to uncover a potential flaw – where they can then focus their attention on developing or detecting a true exploit.
And quite frankly, if there is one group I worry about right after our adversaries – it is ourselves. I know I have human engineers – even when supplemented with lots of machine learning – all it takes is one configuration mistake, and we are toast.
And momentum is still the unseen enemy. Even if our tools are not misconfigured – their policies can become stagnant. Some of the tools that had my highest confidence – and continued to prevent the attacks they always did – were a little exposed by our BAS tests. The “old reliable” just did not warrant my critical eye until BAS testing reveled opportunities for simple tweaks that reduced our attack surface considerably.
And wouldn’t you know it – just weeks later those tweaks prevented the delivery true, outright attacks. Perhaps our defense in depth would had addressed these attacks anyway – but why not just block them straight away against a basic policy criteria. Sometimes a simple rule reduces the risk aperture enough, there is no need to send the attack to advanced tools for interrogation at all.
Does BAS replace a robust, third party testing program – no, not at all. We are still running the BAS tests, and disinterested third party testing is an important validation. But if you do believe your security posture and controls are fairly mature – BAS can help you uncover potential flaws and policy improvements – and help answer the surprisingly tricky question:
“How effective are your security controls?”