Measuring Security Metrics

The Enterprise System is basically the set of all the interconnected devices, and the way in which they are connected (largest perspective). It is the whole system that support the enterprise in its objectives.

It is a subset of the system, with an focus on a particular host, or small networks.

When dealing with security metrics, there exists a general approach that tries to link the definition of the metrics, based on the notion of attacker and defender. Trying to capture the dependencies between what the attacker can do, and what the defender can do. From this point of view an attacker is an entity that represents the entry point of an attack (IP address, privilege over a specific machine, a person); the incident is the outcome of the attack.

We can represent the attacker-defender interaction under the enterprise system, or under the computer system

We can represent the Attack-Defense interaction with this figure:

The attacker has different means that He/She can use to attack the system, some of them are easy, some are more involved; also the attacker can use different attack vectors to get into the system.

Then there are the defenses, they are effective when considered together with an attack vector and an attack. We can try to model, using one or multiple metrics, or the attacker can get into the system, measure the vulnerabilities that allow the attacker to get in, and measure the probability of success of some attacks, and finally we need to collect the evidences that we can use to express numerically those concepts.

If we consider the attack-defense interaction considering a single device, we consider low level details, like filtering barriers around the hosts, attack detection mechanisms, and then the Host and only the successful vulnerabilities.

This perspective is not easy to measure, because all the aspect over which the vulnerabilities needs to be considered must be taken into account.

A situation metric is the aggregation of all the aspects that can describe a system on a given time. We have to quantify the presence of vulnerabilities, the capability (strength) of the attacker, and the capabilities of the company to defend itself.

The situation is nothing more than a generic combination of those three aspects: Vulnerabilities, Attack, and Defense.

The central point of the analysis is the system, from there we can identify all the possible related aspects:

No we can define metrics to quantify vulnerabilities, defenses, attack capabilities, and some for quantify the current situation.

There are three main types of metrics to consider: some quantify the susceptibility to phishing and malware, and other metric concerns Password Vulnerabilities. The first two metrics tries to highlight the user cognitive bias, the fact that the user’s decisions can be impacted by his own background and knowledge (this weakness is leveraged by Social Engineering attacks); on the last point, we consider problems deriving from a bad management, so we try to measure how the user learn security measures.

Here we try to measure something that’s between the user and the software, so we would like to evaluate how much the interface is robust with the respect to the user behavior.

The metric typically used is the attack surface, from this point of view we try to quantify the subset of resources that the attacker need to use in order to enter in one, or multiple, hosts.

We define a tuple of values that is possible to analyze:

\(\langle \sum_{m \in \mathcal{M}} R_{d,e} (m), \sum_{c \in \mathcal{C}} R_{d,e} (c), \sum_{i \in \mathcal{I}} R_{d,e} (i)\rangle\)

\(R_{d,e}\) represents the ration of the potential damage and effort related to:

• The set of attack methods \(\mathcal{M}\)
• the set of channels offered by the software \(\mathcal{C}\), and
• the set of data item of the software \(\mathcal{I}\).

The tuple is composed by: the set of attack techniques that the attacker may use, how much the attacks could be effective on the considered entry point. More input the software needs, more attack vectors the attacker could use.

Considering software vulnerabilities, we have three different types of metrics that we can use:

• Temporal Attributes,
• Severity of individual software vulnerabilities, and
• Severity of a collection of vulnerabilities.

We are trying to evaluate and quantify the strength of all the elements deployed that should protect the system a priori. Typically the main mechanisms evaluated in literature are: blacklisting, data execution prevention (how we manage inputs), and finally control flow integrity, that deals with the software that we execute inside the organization.

Of course this is not an exhausting list, but there are many other prevention techniques, the problem is that they do not have any evaluation criteria, and corresponding metric.

We want to evaluate the capabilities to identify anomalies, correctly classify them and trigger a fast reaction. We want to evaluate monitoring and detection mechanisms.

This category provides only two classes of metrics:

• Address space layout randomization metrics, and
• Moving target defense

We try to evaluate the robustness of the overall defense mechanism. First we evaluate ow much we are able to contain a specific kind of attack: penetration resistance is measured by running a penetration test to estimate the level of effort required for a red team to penetrate into a system.

Network diversity measures the least, or average effort an attacker must make to compromise a target entity based on the causal relationships between resource types to be considered. This mechanism is effective only if is done with a good criteria, replication is not enough, because simply cloning a system will increase performance, but not security. So all the instances must be functional components that will delay the attack. Network diversity must be taken into account during a design phase.

We can try to measure two main elements: the lifetime and the victims of a zero-day attack. The first measures the period of time between when an attack was launched, and when the corresponding vulnerability is disclosed to the public; the latter measures the number of computers compromised (or that may be compromised by it).

This metrics measures the success of APTs. The success of targeted attacks often depends on the delivery of malware, and the tactics used to lure a target to open a malicious email attachments.

We define the threat index metric as \(\alpha \times \beta\), where \(\alpha\) is the sophistication of the social engineering tactic used, and \(\beta\) is the technical sophistication of the malware (or attack vector) used.

Botnets have been studied for a longer period of time, for this reason different metrics are present in literature.

The metric used is the infection rate metric that indicates the average number of vulnerable computers infected per time unit.

In adversarial machine learning, attackers can manipulate some features that are used, changing the model. The attacker may act at different level:

• At training level, compromising training data
• Introducing noise into the algorithm, or changing parameters of the latter.

The strength of the attack can be measured by the increased level of false-positive, and the increased level of false positive rates. (How much the performance of the learning algorithm are degraded)

From the point of view of obfuscation attacks there are two metrics that can be used:

• Obfuscation prevalence metric. How much are we able to obfuscate particular behaviors.
• Structural complexity metric. Measures the runtime complexity of packers in terms of granularity.

Here the idea is that that we abstract the system using a model, and then evaluate the state of the system given the model. There exists two metrics that can be computed from the Attack Graph. Identifying hosts, and communication links, mapping compromised hosts and links, and then counting them. The second metric models the system as a Markov Chain, representing the number of actual compromised hosts at time \(t_k\), and estimating the probability of certain number of hosts are compromised at time \(t_i\).

First of all we can try to evaluate the frequency of security incidents.

Two following metrics can be used:

• Delay in incident reaction, measures the time between the occurrence an detection
• Cost of incidents, including both direct and indirect costs.

Those two metrics must be use together to provide a full picture of the incident itself.

The two metrics of the security investment comes from a business perspective. The first index used is the security spending, that indicates the fraction of the budget allocated to ICT department spent to address security issues. Try to balance the budget.

Security Budget Allocation estimates how the security budget is allocated to various activities and resources. Security has many different phases to consider: investment on technology, training of personnel, create new procedures, and so on.

Return on security investment is the last metric, it is a common index used to evaluate the financial aspect of a company: quantify the benefit of the investment did. Since security is not a real investment, the ROSI metric actually measures the reduction of the loss caused by incompetent security. This index is very useful at the strategical management, giving fundamental information on how to modify the current investment strategy.