Getting More Value from Your Security Awareness Program: Hypothesis Based Testing

Posted by Mark Chapman—President & CEO of PhishLine 


As part of a risk-based continuous improvement process, the results of an effective information security awareness program should provide actionable observations to improve the people, processes, and technology layers of security controls.

At PhishLine, a big component that makes our approach unique is the emphasis on “Hypothesis Based Testing.” Rather than depending solely on external studies and benchmarks, our customers leverage specific tests to improve the security posture of their organization.

However, before designing a test to validate the hypothesis, it is imperative to identify what actions will be taken if the test results support the hypothesis, negate the hypothesis, or are inconclusive. If you cannot identify meaningful actions, there is little justification for performing the test. Risk-based methods should be used to identify more actionable tests.

The following graph shows three sets of results: a hypothesis (what we expect to see), and two theoretical outcomes (what we might actually see when we perform the test). Consider the various actions that could be taken based on the “Actual 1” versus “Actual 2” results when compared to the hypothesis in this hypothesis-based test:

As we can see by the bright blue hypothesis line, we expect the Failure Rate to decrease after “Security Awareness Day” and then slowly begin to increase after time passes. In the results line of “Actual 1” we see no change in the Failure Rate. This means our hypothesis was not supported; thus, the company could decide to reallocate the money spent on the ineffective “Security Awareness Training Day.”

In the results line of “Actual 2” we see a significant decrease in the failure rate. In this example the hypothesis was supported; thus, the company could decide to double-down on this specific type of training while performing higher-granularity tests to further measure and maximize the effectiveness of particular content for different audiences.

Here are a few more examples of simple hypothesis-based testing with potential actions.



We strongly feel that hypothesis-based testing is a critical component to any security awareness program. It provides context, ensures relevance, and enables appropriate remediation actions based on the resulting metrics.


Related posts