top of page

Measuring your Defenses for Reals...

So what’s the problem?

I hate stating the obvious but in this blog I’m going to anyway. Regardless of the advancements in security controls, attackers continue to successfully complete their cyber missions against organizations whether it’s to steal sensitive information, hold your files hostage in demand for a ransom or wipe your data completely. Earlier this year we had the city of Atlanta get infected with ransomware that took their systems down for a few days until they were able to get back to normal operations. The sad part is a spokesperson stated that they now know they have to increase their security defenses. Why did it take an event like that to figure that one out?

This year we also had the Orangeworm threat actor group compromise many healthcare entities. It’s not 100% clear how they got in, but most likely it was through vulnerable services that were unpatched and exposed to the Internet. These attacks seem to just exfiltrate sensitive data found on devices that had MRI and X-ray software/equipment.

British Airways had their defenses penetrated impacting customer data and about 380,000 customer booking transactions. And the list goes on…

With breaches continuing to flow steadily it begs the question. “Are security professionals (we) just on autopilot going through the motions? Now this is a very broad statement and I am sure it’s not true for everyone, but I bet for many of us, there’s some truth in it. With limited (skilled) resources, increasing compliance regulations, and I am sure budget issues, we end up going through the same actions we have done for many years to build and test our security defenses.

We typically will perform any one or all of the below tests to measure the effectiveness of our defenses:

Security Assessment - Here we simply test that the controls we defined to be put in place are in fact in place and are performing as planned. Or you may measure your controls based on a framework like the Cyber Security Framework or security best practices.

Penetration Testing - Another popular testing procedure is to either have your internal penetration testing department or an external company to try and break into to your environment. There is usually a clear scope for the tests and many times the testers have their own testing techniques they use to execute on the scope that was defined.

Threat Hunting - This is a fairly new service that has cropped up over the last few years. With this service you either give the entity your log data or allow them to collect on their own for a period of time. They will then analyze the logs looking for indicators that the adversary has evaded your defense and is in your network. A bit more reactive but a service that can prove useful. Larger organizations will have this capacity in-house and will implement this as one of the SOC functions.

These are all good testing procedures that need to be completed throughout the lifetime of your security program, but there is one common problem with each one and that is they are not testing your security defenses against real world techniques that adversaries are using today. Sure, your penetration testing company may use some real world techniques but some may not.

What we really should be doing is understanding what the adversaries are doing to complete their cyber mission. Understand the Kill Chain and the techniques that are used within each phase of the attack. Now this sounds super cool and I advocate it, but there are two distinct issues with this.

  • Most security professionals are not Threat Intel Analysts and do not have the time to learn all the techniques the adversary may use to complete a cyber-mission.

  • It's very hard to measure, track and document all the Tactics and Techniques even if you did have some knowledge.

Intro to the MITRE ATT&CK Project

If you have not checked out this project, I highly advise you to do so. ( It went live around 2015, but it wasn’t until the later part of last year where it started gaining traction. As a matter of fact, they just had their first ATT&CK conference that had close to 2000 attendees both in person and online. The goal of the ATT&CK project is to provide a standard way of showing all the known tactics and techniques that adversaries have used to complete their cyber missions. They researched all the freely available information and came up with 11 different tactics and over 200 techniques which the community keeps adding to. In addition, they have documented the known threat actors and the software/tools used in the attack.

Figure 1

You can drill down into each one of the techniques and you will get a brief description on what it is and how it’s used along with examples, mitigations, detections and references. It will also give you information like what platform its used on, permissions needed to run, data sources needed to detect it, etc. There is so much valuable information here for both the non-threat Intel person as well as the folks focused on this stuff.

Figure 2

This is the first type of project that gives you all the known tactics and techniques the adversary uses in one location. It’s a tremendous amount of information that can really help assess how effective or ineffective your security posture is against real world attacks. This will also allow you to continuously assess instead of just once a year which really only gives you a point in time view.

Measuring your Defenses for Reals

Now I want to walk you through one example of how you can use the framework to measure your security posture against a specific attack you might have heard of in the news. In this case, you are a healthcare organization and you heard about the attack (Orangeworm) I mentioned in the beginning of this blog and you want to know how well you are protected and if it can even be detected.

The first steps may be to simply search the Internet looking for blogs or Intel reports that discuss the attack. Once you have enough information you typically want to put it into the various phases of a cyber kill chain. Figure 3 below is an example of what that would look like.

Figure 3

The above gives you a decent overview but now we need to apply the defined adversary techniques within the ATT&CK framework for more detailed understanding of the attack. This will allow you to better determine if you have the right tools in place to detect and prevent the threat. Figure 4 below provides you an example of what that may look like. Keep in mind I am only showing details of some of the techniques.

Figure 4

Now that we have defined most if not all the techniques in the attack, the next step is to ask the following questions for each technique.

  • What is the technique? You need to make sure you fully understand what the technique does.

  • What platforms does the technique work on? This helps to determine if you even have the OS in your environment. You may not have to worry about that specific technique if you don’t have the right platform for it to run on.

  • What are the permissions needed for the technique to execute? Is it user mode or do you need administrative privileges to run.

  • What data needs to be collected to detect the technique? You need to make sure you are collecting the right log data so you have an opportunity to detect it.

  • What tools do you have or need to detect/protect against this technique? If you’re not collecting the right data, its possible you don’t have the right tools in place.

  • How can I reproduce this? Typically, you want to be able to generate the same digital dust that the technique leaves behind so you can build your detections and test them to ensure they work properly.

I don’t want to go through each one, but let’s choose “rundll32.exe” as an example. Figure 5 below is an example of what that may look like.

Figure 5

If you’re not familiar with rundll32.exe it’s a valid executable that is resident on all default Windows machines. Its typically used to run program code in dll files. The bad guys will use this to load their malicious dll in the hope they can evade security controls that might not be monitoring this trusted executable.

After you iterate through each technique to determine if you can detect or protect against it, you can then document your findings. These findings help you identify your weaknesses and assist with prioritization. To document of course you could use a spreadsheet to start, but I am sure you will want to migrate to something more robust over time. Also Mitre has developed a nice ATT&CK Navigation tool ( for better visualization and customization of your results. Below in figure 6 you can see an example of what the results would look like using the Navigator tool. You could use something as simple as Red, Yellow and Green to depict your detection status.

Figure 6

This example is for a specific threat. However, what I would recommend is to go through each technique and ask yourself the questions I discussed above to determine of you can detect that technique or not. Which ones should you do first? I recommend doing the ones that are the easiest to detect and only require one data source. Many of the techniques are complicated and the adversary has a variety of methods to execute the technique so it may be hard to account for them all at once in some cases.

Open Source Tools Related to ATT&CK Framework

To generate the digital dust for each technique I mentioned previously, you could have your offensive team perform them or if you don’t have a dedicated team, there are a few open source tools and projects that can help you get started. Below are a few:

Sigma – ( - is a generic and open signature format that allows you to describe relevant log events in a straightforward manner. The rule format is very flexible, easy to write and applicable to any type of log file.

Atomic Red Team – ( - allows every security team to test their controls by executing simple "atomic tests" that exercise the same techniques used by adversaries (all mapped to Mitre's ATT&CK). It is worth noting, they have a commercial version called Red Canary as well.

APT Simulator – ( - is a Windows Batch script that uses a set of tools and output files to make a system look as if it was compromised. In contrast to other adversary simulation tools, APT Simulator is designed to make the application as simple as possible.

Caldera – ( - CALDERA is an automated adversary emulation system that performs post-compromise adversarial behavior within Windows Enterprise networks. It generates plans during operation using a planning system and a pre-configured adversary model based on the Adversarial Tactics, Techniques & Common Knowledge (ATT&CK™) project.

There are some commercial products out there as well. One you might want to take a look at is AttackIQ ( The marketing slick states that it leverages the MITRE ATT&CK Matrix to analyze adversaries’ TTPs against your existing security controls. It can also run through an API interface connect with various security controls like EDR tools and SIEMs that automate the detection.

Things to consider when attempting to implement the Framework

After attending the ATT&CK conference, there were many practitioners sharing their experiences implementing the framework. Below are some of the common ones:

  • While the framework provides a lot of benefits when implemented, it can be a challenging process. As mentioned before, some techniques can be tricky to detect as some of them can be executed in a variety of ways. Implementing in phases and understanding this will be a continuous improvement process.

  • Understanding your organization’s daily activity is important. It helps to have a baseline to help with creating proper detections.

  • Look into the open source projects I referred to earlier to help create the digital dust to determine if you can detect the technique or not.

  • Start with the simplest techniques for detection with the least variations and potentially only using a single data source. This will allow you to make more headway in the beginning of the project and give you some experience to deal with the more complex ones.


The amount of information the ATT&CK framework provides can greatly benefit you in your efforts to properly measure the effectiveness of your security defenses against real world attacks. In addition to helping you measure your security posture, it has many more use cases such as:

  • Assist with Threat Hunting – Knowing these techniques can help you get started with adding a basic threat hunting capability to your Security Operations Center. Just knowing the techniques allows you to do basic searches for those artifacts within your network. You could simply start by querying your SIEM.

  • Act as a Training Guide – The framework is a great way to introduce someone into both the offensive and defensive sides of security. You could create some basic offensive playbooks that progressively get harder on both sides.

  • Vendor Selection – You can use the framework to help validate the usefulness of the vendor solutions you already have and or help with creating criteria when doing a bake off with various products.

  • Improve Communications – The framework provides a common language to communicate within teams, external groups and to management.

Lastly I would highly recommend you consider taking some time to investigate the ATT&CK framework and include it as a focused project in the near future before you invest in the next whiz-bang product.

Who's Behind The Blog
Recommanded Reading
Search By Tags
No tags yet.
  • Facebook Basic Black
  • Twitter Basic Black
  • Black Google+ Icon
bottom of page