What Is an Adversary Simulation?

Adversary simulation is often marketed as “red teaming”; there is nothing wrong with this, but over the years it has become a vague term that means very different things to different organizations.

At Icebreaker, we define adversary simulation as a goal-based assessment with a focus on realistic tactics and tools, in which the target organization’s defenses and procedures are actively tested. Outputs from an adversary simulation will include technical vulnerabilities and misconfigurations. However, the primary focus is to evaluate risk to specific, high-value assets and measure effectiveness of defensive policies and controls. This service may also be marketed as a red team or threat simulation. The important thing is that it’s clearly distinct from other types of offensive assessments such as penetration testing, purple teaming, or application security testing.

Adversary simulation is very different from the penetration tests that most organizations regularly receive in a number of key ways. First, in a penetration test the goal is to find as many technical vulnerabilities and misconfigurations as possible within the target. Vulnerability and port scanning tools are commonly used to facilitate this type of testing and the blue team/IR capabilities are generally ignored. Outputs from a penetration test are immediately actionable and clearly defined. Penetration testing is a cost-effective way to identify as many technical, fixable problems as possible in a short time frame.

However, many companies have started to invest millions annually into their detection and response capabilities, which are ignored and sometimes intentionally disabled during a penetration test to facilitate testing. Metrics for blue teams are notoriously hard to track and it can be difficult to evaluate current capabilities, identify gaps in detection and response, and prioritize future improvement.

The consequences of modern detection capabilities are that adversary simulations will require a longer time period than a penetration test, generally around two months of operating time. Scoping is generally less restrictive than penetration tests since they are meant to simulate a real attacker and because the testers aren’t using loud, potentially hazardous tools such as vulnerability scanners.

While during a penetration test, many of the defensive and IT team members will be aware it’s occurring, during an adversary simulation the number of people “read in” on the assessment tends to be smaller and more restrictive. This allows a company to better see how their team reacts to an unexpected incident and to identify gaps in policy and procedures in a way that an announced exercise cannot. At least one person in the incident response chain of escalation should be aware of the adversary simulation so that the client can appropriately limit impact on security personnel in case of detection.

What Value Is Provided by an Adversary Simulation?

The value output of an adversary simulation can generally be split into three different elements.

1. Measuring the risk of compromise for specific, high-value assets.
2. Finding and remediating technical, exploitable misconfigurations and vulnerabilities.
3. Training, evaluation, and improvement of blue team capabilities.

Adversary simulations are always goal based. This objective of the assessment is the driving force behind all other elements. Additionally, because the defensive team is not aware of the testing or will be treating the testers as a real threat if detected, this allows an overview not just of the engineering controls protecting an asset but the defensive controls and company policies as well. This provides a much better view of risk for a specific asset than other types of offensive testing.

Like in a penetration test, adversary simulation testers will be actively attacking and exploiting the target environment. However, during this type of assessment, the testers will be solely focused on achieving a realistic goal, in the same way an actual attacker is. Because of this, generally there will be fewer findings of this type in a final report. But those findings will almost always be able to be tied directly to high organizational impact, making them generally higher-fidelity findings.

Perhaps most importantly, an adversary simulation gives an organization the chance to evaluate and train their blue team capabilities. Depending on how scoping is handled and how experienced the defensive team is, this can be used in several ways. Testers can intentionally trigger alerts at specific points in time to force a team to handle a real hands-on keyboard attacker.

We find that this is commonly one of the most valuable things organizations get from an adversary simulation, as it sheds light on gaps in incident response procedures and/or logging. For more experienced teams, this is an opportunity to test their capabilities to track and keep out an advanced threat actor in a safe and controlled manner.

How to Get the Most Value out of an Adversary Simulation?

Due to the required time and expertise, adversary simulations are more expensive than penetration tests. Because of this, it is important to understand when you are prepared for one and how to get the most value out of it.

1. Be Clear on What You Want out of the Assessment

Unlike a penetration test, adversary simulations have a lot more “wiggle room” in terms of focus and execution. Each adversary simulation is highly bespoke, with the focus on what the customer needs at that time.

A good adversary simulation team will have the capability to operate as a highly sophisticated threat actor. But they should also have the knowledge to operate more in line with the day-to-day mid-tier threats that many organizations are more worried about.

If you have a clear threat model, communicate it with the adversary simulation team and they will work to be a threat of similar capabilities. There is a saying to keep in mind: “train hard, fight easy.”

These assessments are a great opportunity to really push your defensive team and force a “worst-case scenario.” However, adversary simulations can also be an opportunity for gap analysis of coverage against lower-tier actors. The important thing is to communicate with the assessment team what you are honestly worried about, what you want to prioritize in the assessment, and what types of outputs you most care about. They should be able to determine the best way to handle the assessment to maximize value.

2. Plan for IR Before the Assessment Begins

A core element of an adversary simulation is the defensive team. The assessment generally shouldn’t end because of a single detection; more commonly than not when our team is detected we find that the customer is unable to kick the team out of the network before we accomplish the objectives.

Before the assessment begins, make sure the assessment team and select core individuals are in the IR escalation path and establish how thoroughly IR should be conducted if and when the adversary simulation team’s activity is detected. This is not something that should be figured out “as it occurs,” as it can have serious consequences on the business, on individuals handling the incident, and on the assessment itself. We recommend deciding ahead of time how much IR should occur to maximize the value to the blue team without impacting daily cyber defense requirements.

As previously discussed, determine how important a full IR is to the organization and clearly understand the benefits and costs. For instance, you may choose to have the blue team initiate full incident response procedures if a detection occurs, with the goal of completely kicking the testers out of the environment. This provides great training and can help identify gaps in the incident response process. But if they are successful, you will lose any findings that would otherwise be discovered after the point of the team being kicked out and it may require a lot of employee hours not applied to real-world operations. Communicating your priorities and concerns ahead of time can help the adversary simulation team craft appropriate “threat level,” goals, and testing procedures to fit your needs.

3. Have a Backup Access Plan for the Testing Team

One of the biggest misconceptions we see is that this type of assessment must be 100% black box, external, no insider information, and with the testers operating at full stealth capabilities. This is absolutely a valid way to approach an adversary simulation, but often one of the least cost-effective.

Sophisticated, nation-state-level phishing campaigns are most useful and cost effective as evidence to sway skeptical stakeholders that an organization’s perimeter can be breached. If a more mature understanding that the perimeter is not impermeable pervades at your organization, we recommend planning for an “assumed breach” scenario after two weeks of initial access attempts. This still allows the testers to adequately test your external and phishing defenses while limiting cost.

This is counterintuitive to many of our customers because phishing attacks are so common. However, legal and ethical restrictions limit the ability to replicate many attack scenarios. Some examples of real attacker methods frequently used are email thread hijacking, cross-company movement, impersonating other real companies, supply chain attacks, bribes, black mail, zero/n-days, and unethical phishing pretexts. The reality is that on a long enough timeline, an attacker will gain initial access to your network.

Choosing an Adversary Simulation Provider

“Red teaming” has become a marketing buzzword in the last few years, and it’s been overused to describe many types of offensive security testing. It can also be hard to find reviews on these teams since not many client organizations want to announce a successful adversary simulation. Here are some things to consider when choosing a team to provide your adversary simulations.

1. Do They Provide True Adversary Simulation (Threat simulation, Red Team) Services?

Different companies label these services differently, but a big red flag for a company is if they cannot clearly articulate the difference between an adversary simulation and penetration test. There is a clear delineation in value output and approach between a penetration test and an adversary simulation. If a company’s “red team” service is two weeks long “but stealthy,” it probably isn’t the type of service you are looking for.

2. Do They Have Quality, Published Work?

In the absence of reviews or references, a guidepost can be to look at the type of content a team provides outside of assessments. Most adversary simulation teams have research and development that they publish on GitHub and/or blog posts. Some teach training courses or are active in the adversary simulation/red team community. These companies demonstrate they care about their craft and often have more experience in the field.

3. Do They Have Technical Chops?

Adversary simulation operators should have a strong understanding of the tooling they use throughout these assessments and can be invaluable in after-action reporting and follow-up purple team exercises in helping build detections for the gaps identified. Ask to speak with the actual operators who will be performing the adversary simulation and ask them about their recent experiences at organizations like yours.

Can they walk you step by step through how the code they executed works, why it bypassed the client’s defensive stack, and how to detect it? Do they have the capability to run their own custom tools and techniques? Can they confidently discuss the latest published techniques, the most relevant TTPs from threat actors, and how they relate to your company?

This is really where the value comes from in an adversary simulation. Beyond just “your server is vulnerable to X exploit,” a good team can be a valuable resource for insight into your current and future defensive capabilities.

Working Together

While a dynamic of friendly competition may emerge between the adversary simulation team and blue team, mutual trust and respect is always essential. As such, it’s vital for the adversary simulation team to strictly adhere to its scope, report vulnerabilities with fairness and tact, encourage an atmosphere of cooperation at the client organization, and empathize with the defender’s difficult job. Both the adversary simulation team and the blue team ultimately share the goal of improving security.

It’s common for a first-time customer to be wary and even mistrusting in initial calls, only to have them astounded by the value that can be provided by a well-executed adversary simulation. Often the impact demonstrated by an adversary simulation leads to transformation of clients’ security programs. At Icebreaker, we’ve seen first-hand the positive impact a well-executed adversary simulation can have on an organization.

Please contact the Icebreaker team to discuss how an adversary simulation can benefit your organization.