Does Security Awareness Work? Some Answers from Experimental Research

Shortly before the 2004 Infosecurity Europe trade show was held in London, a small group of researchers gathered at a major rail station in that city and proceeded to approach the mass of morning commuters. The researchers offered a deal to each individual: If you tell me one of the passwords you use at work, I will give you a bar of chocolate.

The bribe was effective. More than 70% of the commuters divulged their passwords in exchange for the candy. Another 34% were willing to provide their passwords, even without being offered the chocolate!

All right, let’s be charitable. Let’s assume that all of these chatty commuters were employed in organizations without any security awareness program. Let’s pretend that not one of the early morning, sugar-starved workers had ever learned that they should not share their password with fellow employees or, for that matter, with a complete stranger. If we assume and pretend these conditions, we gain insight concerning the security practices of users who have not been exposed to a well designed security awareness program. And this insight is most definitely unsettling: A vast majority of employees are eagerly willing to exchange their private passwords for a bit of candy. Surely, we hope, had these commuters been exposed to a carefully designed security awareness program, the researchers would have distributed far fewer chocolates.

Or would they? Would the vaunted, hypothetical “security awareness program” have transformed the workers into knowledgeable users whose lips remained sealed and whose passwords kept secret? Actually, we don’t really know. Credible research concerning the effectiveness (or ineffectiveness) of security awareness is surprisingly skimpy, and our “best practices” for conducting awareness programs is often based on little more than past practice and, occasionally, anecdotal evidence culled from conferences, websites, and magazines (bloginfosec excluded, of course).

During the past few years, however, at least three published, experimental studies have been conducted that may provide some answers to the question: “Does security awareness work?” These studies are “experimental” in that they involve one of two research designs:

· Users are exposed to a treatment (some form of security awareness training). Later, they are confronted with a situation that involves use of the knowledge imparted via the treatment. Other users, who were not exposed to the treatment, are also confronted with the same situation. Results of the groups’ behaviors are then compared to determine if the security awareness treatment was an effective cause.

· Users are exposed to a security awareness treatment. Following the treatment, data relevant to the users’ security practices are collected. If the follow-up data indicate that the users’ behaviors reflect good practices, then the awareness program is considered to be a possibly contributing influence.

This article will explain the three studies and present their results. However, the results themselves, and their possible implications for the design of security awareness programs, will be discussed in a second article to be posted in a few weeks.

What is Security Awareness?

Before discussing the experiments, it is useful to understand the concept of “security awareness” adopted by the researchers. All three studies define “awareness” as described in NIST Special Publication 800-50, “Building an Information Technology Security Awareness and Training Program” (2003):

The purpose of awareness presentations is simply to focus attention on security. Awareness presentations are intended to allow individuals to recognize IT security concerns and respond accordingly.

Thus, an awareness program—according to the NIST definition—is intended to change behavior or reinforce good security practice.

Interestingly, this is not the same definition proposed by NIST in its 1989 publication, NIST SP 500-172. In that document, security awareness is described as creating “the [employee’s] sensitivity [my emphasis] to the threats and vulnerabilities of computer systems and the recognition of the need to protect data, information, and the means of processing them.” Apparently, at some point between 1989 and 2003, the NIST researchers determined that the ambiguous term “sensitivity” was not sufficiently clear to provide guidance for program designers. By 2003, the definition is focused on behavioral outcomes, not on an amorphous sensitivity.

Here, then, are the three experimental studies that examine the effectiveness of security awareness in terms of measured behavioral outcomes.

The Carnegie Mellon Study

Researchers at Carnegie Mellon University established three groups, each consisting of 14 student volunteers.

All members of Group 1 were sent a bogus “phishing” email. When the students clicked on an embedded link, they were sent to a website that contained educational materials describing how to identify suspicious email messages and reminded to delete these messages without opening them.

Group 2 students were not sent the bogus email. Instead, they were simply asked to read the information concerning suspicious email messages. Group 3 did not receive the email, nor were they exposed to the educational materials.

One week later, members of all three groups were sent another bogus “phishing” email. Sixty-four percent of students in Group 1 recognized the message as suspicious; only 7% of the members in Groups 2 and 3 were able to identify the email as an example of “phishing.”

The German Study

The CISO of a large German financial services organizations implemented an extensive and costly security awareness program. All employees received awareness newsletters, and security-oriented posters (some including the slogan “Security is Everyone’s Problem”) were placed in prominent locations throughout the Company. Following this effort, consultants were hired to test the program’s effectiveness by means of social engineering. Within a few days, the consultants were able to access the CEO’s email and had obtained confidential information from several systems. The consultants, wearing T-shirts emblazoned with a flashy logo, were permitted to wander through Company headquarters unimpeded; not a single employee questioned whether these oddly attired strangers were authorized to access the physical premises.

The Military Academy Experiment

At the commencement of each semester, each cadet attending one of the military academies in the United States receives security awareness instruction. Freshmen cadets are provided an additional four hours of awareness instruction. This instruction includes material pertaining to network security, focusing upon the identification and avoidance of viruses, worms, and other malware.

To examine the effectiveness of this instruction, a professor in the Department of Electrical Engineering & Computer Science sent an email message to 512 of the Academy’s 4200 cadets. Participants were randomly selected, and an equal number of freshmen, sophomores, juniors, and seniors were included in the study. The email was sent from “Robert Melville, Col., USCC” (a bogus name), and his address was described as located in “Washington Hall, 7th Floor, Room 7206.” (Washington Hall, known well to all Cadets, has no 7th floor.) “Colonel” Melville’s message informed each cadet that “There was a problem with your last grade report.” The recipient was requested to click on an embedded link in order to ensure that the cadet’s grade information was accurate and to “report any problems to me.” If the gullible cadet clicked on the link, they received a modified “HTTP 404-Page Not Found” web page.

Eighty percent of all cadets clicked on the embedded link, and 90% of the freshmen fell victim to the ruse. The professor concluded his published study by stating that security awareness instruction had clearly not attained its objective. “The traditional classroom model,” he stated, “is necessary but not sufficient when it comes to learning.”

What Are the Lessons from Experimental Research

The three studies are important because they attempt to measure the effectiveness of security awareness programs. Other measures—such as the number of individuals participating in programs and the quantified opinions of employees who participate—are useful, but they do not directly indicate behavioral changes associated with awareness instruction. Additional metrics, particularly associated with the occurrence of security incidents, may offer an indication of awareness effectiveness; however, incident trends may be influenced by many factors (such as the sudden contagion by a virus) other than awareness treatments.

In the second part of this article, I’ll attempt to analyze the results of these three studies from the perspective of lessons learned concerning awareness effectiveness. However, while awaiting my babblings on this subject, you may already have developed some thoughts of your own. Do you discern patterns among the studies? Do any of these experiments confirm—or refute—your own experiences with security awareness programs?

One Comment

  1. Scott Wright Apr 15, 2008 at 12:34 pm | Permalink

    Great article! I’m looking forward to the next installment.

    This is getting very close to my own research on lost USB drives, also known as the Honey Stick Project. I have so far distributed about 20 USB sticks around various publicly accessable areas in and outside of Ottawa. I plan to do more in other cities, through my own travels and with the help of others.

    For current result stats and more background please visit the website:

    While my experiment is not as controlled as the above institutional initiatives, it is providing some interesting results about the psychology of people who find lost devices.

    – Scott

One Trackback

  1. By Does Security awareness work? « Ideasfoundry on April 21, 2008 at 5:14 am

    […] Filed under: Infosec, Manage — ideasfoundry @ 3:43 pm At Bloginfosec an article on security awareness cites some experimental studies on effectiveness of security awareness. In all the three studies […]

Post a Comment

Your email is never published nor shared. Required fields are marked *