- Open Access
Factors influencing network risk judgments: a conceptual inquiry and exploratory analysis
Security Informatics volume 4, Article number: 1 (2015)
The Erratum to this article has been published in Security Informatics 2015 4:5
Effectively assessing and configuring security controls to minimize network risks requires human judgment. Little is known about what factors network professionals perceive to make judgments of network risk. The purpose of this research was to examine first, what factors are important to network risk judgments (Study 1) and second, how risky/safe each factor is judged (Study 2) by a sample of network professionals. In Study 1, a complete list of factors was generated using a focus group method and validated on a broader sample using a survey method with network professionals. Factors detailing the adversary and organizational network readiness were rated highly important. Study 2 investigated the level of riskiness for each factor that is described in a vignette-based factor scenario. The vignette provided context that was missing in Study 1. The highest riskiness ratings were of factors detailing the adversary and the lowest riskiness ratings detailed the organizational network readiness. A significant relationships existed in Study 2 between the level of agreement on each factor’s rating across our sample of network professionals and the riskiness level each factor was judged. Factors detailing the adversary were highly agreed upon while factors detailing the organizational capability were less agreed upon. Computational risk models and network risk metrics ask professionals to perceive factors and judge overall network risk levels but no published research exists on what factors are important for network risk judgments. These empirical findings address this gap and factors used in models and metrics could be compared to factors generated herein. Future research and implications are discussed at the close of this paper.
An information system is a discrete set of information resources organized expressly for the collection, processing, maintenance, use, sharing, dissemination, or disposition of information . All information systems, which we use interchangeably with the term networks, have inherent network-related risks that cannot be eliminated completely because of operational resource constraints. Risks, defined as “the possibility of loss, injury, or other adverse or unwelcome circumstance” , must then be prioritized based on the relative risk level and addressed according to the feasibility of mitigation strategies given both context and resource constraints. Network professionals must evaluate risk throughout the network lifecycle but we focus on risk evaluation during network design and control configuration.
Within the U.S. Department of Defense (DoD) and civilian government networks, organizations must undergo a network certification process— i.e., ISO27000 series (information security standards published jointly by the International Organization for Standardization, ISO, and the International Electrotechnical Commission, IEC) and NIST 800 series )—to evaluate network risk. This process of certification involves a designated approving authority (DAA) who engages with the organization’s information assurance (IA) officer to determine the organization’s network risk level, based on risks identified and control configurations established to address those risks. Some risks are reduced, but not eliminated, through the implementation of various security controls—i.e., management, operational, and technical safeguards or countermeasures employed within an information system to protect the confidentiality, integrity, and availability of the system and its information (a listing of such controls is found in ). Ultimately at the end of the certification process, the DAA certifies that the network, with certain controls implemented, meets an acceptable standardized level of network risk.
This network risk certification process, which requires the perception and judgment of network risk, is difficult to standardize across network professionals because in part, different people believe that different factors are important to judgments of risk ([4,5]). A factor is a perceived circumstance, event, influence, fact, etc. that is related to a particular outcome. Both risk perception and judgment can be influenced by network context ([6,7]), which we operationally define as perceivable factors physically and temporally surrounding an event or circumstance (e.g., the organizational policies, the types of adversaries targeting that organization, the types of adversaries, etc.). With respect to judgments of network risk, little is known about what factors are consistently important and unimportant to network professionals like the DAA and IA officers. Guidelines included in network risk metrics for assessing and assigning risk levels are often generic and not tailored to the conditions and contexts of a given network. Consequently during the decision process, individuals like the DAA may have to ignore certain network attributes not covered in the guidelines or ignore the guidelines altogether. Under circumstances where guidelines cannot be clearly applied, the DAA most likely relies on his/her own perceptual capabilities, work experiences, etc. to judge the network risk level, yet little prior research has investigated exactly what factors are actually being considered. Using a mixed-method approach  that combines qualitative and quantitative research methods to achieve study objectives, we attempted to identify and validate the factors people believe are most important to network risk judgments.
The challenges with judging network risk levels are due in part to the semantic complexity of the term itself. Underlying this semantic complexity is the lack of an agreed upon definition of risk from professionals in industry and academia ([6,9]). Consequently, risk miscommunications may arise  because interlocutors may have different semantic meanings of the term, risk. Risk is a psychological construct , an idea constructed in the human mind from the aggregation of dimensions or categories of abstract or tangible perceived phenomenon. The dimensions constituting network risk that we are familiar with include likelihood, vulnerability, resilience, impact, etc. but it is not clear whether network professionals all believe the same dimensions comprise network risk. Dimensions can be derived from perceived factors that include environmental information, past experiences, and other psychological phenomena such as attitudes and belief systems . For example, a “likelihood” dimension of the network risk construct might be driven by perceived environmental factors and historical experience factors indicating the “likelihood” of successful implementation of security controls prior to an attack. The risk research literature provides varying definitions of risk or perceived risk across different domains (e.g., [11-14]) but no consensus exists about the dimensions underlying risk in general or factors used to construct these dimensions ([6,15-21]). We conjecture that a relationship exists between how network risk is defined, what underlying dimensions are important to a network risk definition, and the relevant dimensional factors used in judgments of network risk. Investigating these dimensions and relevant factors might offer clues to how network risk is defined by network professionals. To our knowledge, no foundational research exists that documents the network risk dimensions and respective factors important to network professionals who design and secure networks. This is the impetus for our research.
We used an exploratory, mixed-method approach to identify what factors are important and unimportant for risk judgment in general (Study 1) and what factors are commonly and most consistently judged as more safe or more risky (Study 2) across our sample of network professionals. Because no prior research has identified network risk dimensions important to risk judgments in the context of a network, Study 2 was designed to address this. Prior research indicates that risk perception and judgment can be influenced by context ([6,7]). Therefore, we were interested in identifying robust dimensions and respective factors that are not susceptible to the effects of different contexts.
This paper is structured to review each study’s objective, the method, the results and conclusions. We close this paper with an overall discussion that includes the implications of our findings, the limitations of our research and future directions.
Study 1: factors that impact network risk perception
Study purpose and research overview
The purpose of Study 1 was (a) to generate a comprehensive list of perceived factors that were relevant to judgments of network risk and (b) to determine which factors were considered most and least important to judgments of network risk. A focus group of cybersecurity professionals first generated a list of relevant factors, which were then validated with a broader sample of cybersecurity and network professionals using an online survey method.
Focus Group Demographics. Five cybersecurity professionals plus one moderator comprised the focus group. All focus group members were employees at a single organization with a variety of cybersecurity expertise. The self-reported expertise included acquisition support (1 participant), cyber threat and vulnerability analysis (1 participant), cyber enterprise and workforce management (1 participant), and enterprise threat/vulnerability management (2 participants). No other demographic information of this sample is permitted to be disclosed.
Survey Sampling and Demographics. The target population included cybersecurity and/or network professionals, who either designed, implemented, supported, and/or tested networks for security purposes or who trained individuals to do these functions. We used a snowball sampling technique , first soliciting colleagues at our institution for study participation; they subsequently invited others inside and outside our organization. The mean sample age (n=38) was 47 years with a standard deviation (SD) of 10.3 years. The mean number of years worked in computer science professions was 13 (S D=9.7), and the mean number of years in their current job was 9 (S D=7.7). We did not require participants to report additional demographic information on the DHS sectors supported. Consequently, we had low response rates for this question and did not report on these questions.
Materials and procedure
Focus Group. During three sequential two-hour meetings spread over the course of a week, we conducted a moderated focus group using a brainstorming and consensus building technique  to identify all factors (at any granularity of detail) that impact network risk perception. We did not collect related information about why each factor was important, or why some factors had very specific language; we were just generating a comprehensive list. Focus group discussions about “why” a factor was provided often lead to desultory discussions and long debates about the validity of the factor so we discouraged those discussions. Each factor offered by each group member was then recorded on a single Post-it Note™ and organized taxonomically by the focus group on butcher paper using an affinity diagramming technique .
Online Survey. All factors generated in the focus group sessions, regardless of their granularity, were placed in an online survey to assess their validity on a broader sample of network professionals. The purpose of the online survey was to assess the consensus on the importance level of each factor. The survey was hosted by SurveyGizmo™, a browser-based survey design and deployment tool, and it comprised three survey subsections: informed consent, factor ratings, and demographics.
Informed consent was obtained in accordance with ethics guidelines for research with human subjects (Institutional Review Board approval HS12-571). After providing consent, participants read the online instructions and then rated each of the factors, presented in random order, one factor per survey page. Each factor was presented in a standardized sentence structure, bolded for quick identification (e.g., “Generally, how important is <factor> to your overall perception of network risk?”); no additional network information or context was provided. Participants were asked to adjust a slider to reflect a level of importance on a continuum between 0 (not at all important) and 100 (extremely important). If a participant could not understand the meaning of the factor or had no experience with it, this person was instructed to refrain from making a rating and to write “don’t know” in a comments box below the factor. After rating all presented factors, participants could suggest additional factors.
Participants then answered demographics questions about (a) job title, (b) job-related expertise, current employer(s), (c) whether their current job supported the US government, military, private industry or whether he/she was a private consultant, and (d) which of the 18 DHS ISAC sector(s) he or she supports (e.g., healthcare, banking, energy). The order of the demographics questions was not randomized. Survey participants were financially compensated with a $15 gift card at the close of the survey.
This section first discusses common distributions of ratings for each factor and then compares the factor means. All factors identified by the focus group and used in the online survey are listed in Additional file 1: Table SA-1, for quick reference.
Participants did not typically use the response scales consistently (e.g., some use the entire range of the response scale while others use a small portion) so we characterized these first. The mean importance ratings obtained from our 38 survey participants ranged from 38.8 to 83.5 on the 0 to 100-point scale. Three common distributions of factor ratings (for Factors 8, 5, and 51) are shown in Figure 1: unimodal, bimodal and multimodal. Factor 51 (The maturity of the organization’s system capabilities for network defense), an example unimodal distribution, has a high level of agreement with most ratings clustered at the high level, indicating that this factor is very important to most participants. Factor 8 (whether the facility uses “SCADA” supervisory control and data acquisition systems) has a bimodal response distribution because scores were clustered around the low or less important end of the scale and around the high or more important end of the scale. Scores for Factor 5 (whether the network is for the military, government, or civilian sector) are distributed across the range of importance ratings. The Additional file 1: Table SA-2 lists all factors, rank-ordered in descending order according to mean importance ratings, and provides density plots of the rating distributions of each factor. Inspection of all density plots for each factor indicates that 27 factors have roughly unimodal distributions, 16 factors have roughly bimodal distributions, and the remaining 26 factors have multimodal distributions.
Table 1 provides five factors with the highest mean importance ratings and five factors with the lowest mean importance ratings. The highest rated factors detailed the adversary capabilities and the complexity of the organization’s network defense. For example, the factor with the highest mean importance rating was the adversary’s knowledge about the organization’s deployed network and security technologies (Factor 18, rank = 1). Other highly-rated factors include the skill of the adversary (Factor 31, rank = 3), how desirable the information on the network is to the adversary (Factor 63, rank = 6), whether the adversary has access to information needed to stage an attack (Factor 28, rank = 11), and whether the attack is persistent or casual (Factor 2, rank = 23).
The standard deviations (SDs) of the importance ratings (also shown in Additional file 1: Table SA-2) were computed to assess agreement across participants. Table 2 shows the five factors wit h the least inter-subject agreement (highest SDs) and the five with the most inter-subject agreement (lowest SDs). By comparing Tables 1 and 2, we note that three of the five factors with the most inter-subject agreement (Factors 18, 45, and 51) were also deemed to be among the most important factors. We assessed whether a relationship existed between the level of importance of each factor (mean importance rating) and the agreement (SDs) and no linear relationship was found.
To determine whether dimensions emerged from our data, we used an affinity diagramming method to hierarchically classify the factors generated by the focus group. Three broad categories or dimensions emerged and were named by the research staff after reviewing each the underlying factors of each dimension (see Figure 2): (1) organization hosting the network, (2) threat/adversary, and (3) contractors (prime contractor, sub-contractors). Specific factors mapped to these dimensions were identified in Additional file 1: Table SA-1. An exploratory analysis revealed that of the ten highest-ranked factors, only one was associated with contractors (Factor 1), four were related to the adversary/threat (Factors 7, 18, 31, and 63), and five were associated with the organization (three of these five organizational factors are related to the network environment—Factors 23, 51, and 66). Also, of the ten factors with the lowest rankings, none were associated with the adversary/threat; five were associated with contractors and five with the organization (the low-ranking organizational factors are related to programmatic, policy, and workforce factors—none are associated with network environment). Thus, factors associated with the adversary/threat and those associated with the organization’s network environment appeared to be ranked high in importance; factors associated with contractors tend to be ranked lower in importance, as do other general organizational factors unrelated to the network environment. This is shown in Figure 3, which was generated to assess the relationship between dimensions and risk judgments. The bar chart in Figure 3 displays the percentages of factors in each of the three dimensions that were rated as high (top 1/3), medium (middle 1/3) and low (bottom 1/3) importance. The organization dimension was populated by the highest number of factors compared to the other two dimensions. The matrix in Figure 3 breaks down counts by dimension and level of importance. The highest rated factors (14) and most of the lowest rated factors (13) involved the organization. To assess whether there was a significant association between factor dimension and level of importance, a chi square test of association was conducted and found to be non-significant (χ 2(4)=8.99, p=0.06).
Participants were asked whether any important factors needed to be added to our list and these were suggested:
Does the organization support “doing the right thing” when the situation warrants?
Morale of the IT, security, and general staff?
Access to historical intrusion/failure data
Commitment to IT hygiene
The presence or absence of a solid knowledge management system.
A focus group of network security professionals generated a comprehensive list of factors considered relevant to network risk perception. Our survey results indicated that factors relating to the adversary or to the complexity of the organization’s network defense were considered most important. We assessed whether a relationship existed between the level of importance (importance ratings) and the level of agreement SDs) amongst our sample participants. While a statistically significant linear relationship between SDs and mean ratings of all factors did not exist, the five most agreed-upon factors (lowest SDs) were also judged as relatively more important [mean ratings between 70.7 and 80.5]. Three emergent dimensions of factors were found (organization, adversary and contractors) in the absence of context.
The importance of each of our 69 factors was assessed but we could not ascertain whether each factor was safe or risky or whether context changes the emergent dimensions of factors. This became the impetus for the second study.
Study 2: context and network risk perception
Study purpose and research overview
We used a subset of factors from our original list of 69 factors and analyzed the degree of riskiness or safeness of each. Given the lack of contextual information related to each factor in Study 1, our subset of factors was analyzed using a vignette-based factor scenario method to provide context.
The target population comprised network professionals versed in the practices of cybersecurity. As shown in Table 3, the 105 participants who completed the survey represented a variety of software engineering, IT management, and information security occupations. The overall mean number of years spent in the computer science professions was 9.6 (SD = 7.6) with a range between 1 and 36 years. We did require participants to report additional demographic information on the Department of Homeland Security (DHS) critical infrastructure sectors supported. The top five DHS sectors participants self-reported to support were Information technology (62), Academia (21), Communications (18), Banking & Finance (17), and Public health (11). Five supported the military, 15 supported the government and 11 were contractors.
Materials and procedures
Three vignettes were generated to represent different network contexts: Vignette 1 described a hospital network, Vignette 2 described a military network, and Vignette 3 described a software development firm network. The context of each vignette differed on attributes like the history of the network and adversarial activity, how the network is manned, the type of information stored on that network and how the network is controlled and configured. Immediately after the participant read the vignette, he/she rated the overall network risk level using a slider (0= low risk and 100= high risk) and then offered a ranking (low, medium, or high network risk) according to the NIST SP800-30 guidance . Then, ratings on individual factors were solicited. Originally, we designed the vignettes to depict factors using a few descriptive sentences to depict each factor, but in our survey beta testing, respondents believed that the factors were not the ones we originally intended. Instead, participants believed the main idea of each descriptive sentence was a single factor. Therefore, we obtained ratings for each sentence’s main idea using a bipolar response scale between 0 = extremely safe and 100 = extremely risky. A rating of 50 was labeled neither safe nor risky.
SurveyGizmo™ hosted the online survey, which was divided into three sections: informed consent, vignette scenarios, and demographics (job title, the DHS critical infrastructure sectors supported, whether the person was working for the government, military, academia or in training). Informed consent was obtained in accordance with ethics guidelines for research with human subjects (Institutional Review Board approval HS12-571). After providing informed consent, each participant was randomly assigned to one of three vignettes using the survey randomizing tool. Each participant reviewed only one vignette. After reviewing a vignette and providing ratings, participants answered a set of un-randomized demographic questions and were awarded a $20 compensatory Amazon gift card.
Exploratory data analysis of overall risk ratings and rankings
Figure 4 summarizes the vignette risk ratings and risk rankings. In the boxplot on the left, the x-axis represents the vignette number (1 = hospital network, 2 = military and 3 = software development firm) and the y-axis represents the risk ratings. The stacked bar chart on the right shows, for each of the vignettes on the x-axis, the frequency of participants that ranked the risk as either low, medium or high. As shown in the boxplot on the left, Vignette 2 had the highest mean risk rating denoted by the horizontal bar in the middle of the box; but the mean ratings were not significantly different. Though participants believed Vignettes 1 and 3 were relatively less risky, the stacked bar chart on the right of Figure 4 indicates that participants did not rank any of the vignettes as predominantly low-risk.
Factor grouping by risk impact
One of the goals of Study 2 was to determine which factors, described by certain contextual features, were perceived as risky or safe. We used Bonferroni corrected t-tests to identify factors affecting on risky/safe ratings (i.e., tests conducted against the null hypothesis that population risky/safe means were 50). Those factors with a corrected p-value above 0.05 were removed from further consideration. We then divided the remaining factors (those with mean ratings significantly different from 50) into four groups based on their median scores: VERY SAFE (median rating below 30) includes factors such as Machines are not connected to both the private network and the internet; SOMEWHAT SAFE (median rating between 30 and 45) includes factors such as The IT staff are fully trained; SOMEWHAT RISKY (median rating between 55 and 72) includes factors such as All patient records are digitized; and VERY RISKY (median rating above 72) includes factors such as Hackers in the past few weeks have been attacking various medical centers nationwide. The factors with the median ratings between 46 and 54 were not included in Table 4 because they were not significantly different from 50. Table 4 lists the factors in each of these groups, along with their median and mean risk scores, SDs, and vignettes to which they belong.
Table 5 displays the most agreed upon factors (lowest SDs) with respect to the risky/safe ratings and the least agreed upon (highest SDs). The values of SDs for these 85 factors, across all three dimensions, ranged from 10.2 to 24.4 (median = 18.05). To assess whether a relationship existed between the number of high vs. low standard deviations and the risky/safe ratings for those factors, we compared the risk levels assigned to factors that exhibited SDs below the median (high agreement among respondents) with those that exhibited SDs above the median (low agreement across respondents). For risk levels, we used the same categories shown above for Table 4: Safe (Rating <45), Neutral (45< Rating <55), Risky (55< Rating). Table 6 provides the 2 × 3 matrix of high/low agreement by safe/neutral/risky importance ratings. The resultant significant chi-square test of association (χ 2(2)=7.06, p=0.029, n=85) indicated that our study participants agreed more about factors that were judged as riskier, compared to those that relate to lower network risk.
Factor groupings based on correlation
One way to summarize the ratings data for each single vignette is to group the ratings into clusters of factors that vary together across participants. Then, determine whether an underlying conceptual or semantic commonality exists amongst a group of factors that cluster; likened to a principal components analysis used with larger sample sizes. If commonalities exist, they may provide clues about ‘agreed upon’ underlying parameters or dimensions related to network risk judgments.
For each vignette independently, we computed the correlation matrix between all the factors (all pair-wise correlations). We used [1−c o r r e l a t i o n] as a distance measure and performed hierarchical clustering with the Ward agglomeration method to divide the factors into groups that might be interpreted as dimensions of the perceived risk construct. We chose the number of clusters, k, for each scenario based on examining several heuristics. Other choices of k could be equally valid.
The resultant correlation matrices revealed clusters of factors for Vignettes 1 and 2 with relatively strong correlations within the group and low correlations between factors in different groups. Because no such relationships were observed in Vignette 3, it was not included in further analyses. While the groupings reflect statistical structure in the data, that structure does not always correspond to a semantic representation of single dimensions. When the majority of the constituent factors in a group shared a common semantic interpretation, we adopted this semantic interpretation as a label for a network risk construct dimension. Four labeled dimensions across the two vignettes emerged:
Information factors related to the information stored on the network and the consequences of the information being compromised — inferred from Vignette 1 (hospital network)
Infrastructure factors related to the infrastructure of the network and the compliance of the network with established protocols.)—inferred from Vignette 1 (hospital network)
Personnel Skill factors related to the skill and training of network personnel — inferred from Vignette 2 (military network)
Adversary Skill factors related to the skill, resources, and motivation of the adversary — inferred from Vignette 2 (military network).
Table 7 lists the factors (with associated mean risk ratings and SDs) in each of these emergent dimensions. Also shown in Table 7 (last column) is the risky/safe grouping (VS = Very Safe, SS = Somewhat Safe, SR = Somewhat Risky, and VR = Very Risky) to which the factor belongs (if any). We note that of the four dimensions of the perceived risk construct, the adversary skill dimension contains the highest percentage of risky (SR or VR) factors (90%), while the infrastructure dimension contains the highest percentage of safe (VS or SS) factors (60%).
As in Study 1, there tended to be higher agreement on the most risky factors. For example, of the 13 factors with the most agreement in ratings (lowest SDs), nine were judged as risky while two were rated neutral and one was rated safe. On the other hand, for the 16 least agreed upon factors (high SDs), no discernable differences were observed in risky/safe ratings. Moreover, factors that were associated with the adversary/threat tended to be rated as more risky in general. Of the 13 factors with the most agreement in ratings (lowest SDs), six involved descriptions of the adversary (five were rated as very risky and one was rated as somewhat risky). Factors relating to the organization’s network infrastructure tended to be associated with lower risk as well. The semantic groupings, or dimensions, derived from inter-factor correlations have similar trending as discussed in the preceding paragraph. Of the four emergent network risk dimensions (i.e., information, infrastructure, personnel skill and adversary skill), the highest proportion of the most risky factors comprised the adversary skill dimension (90%) while the highest proportion of the most safe factors comprise the infrastructure dimension (60%). Somewhat safe factors often comprise the personnel skill and infrastructure dimensions and somewhat risky factors span all four dimensions.
Study 1 initially used a focus group method to produce a list of possible factors believed to influence network risk judgments and then used an online survey method to investigate the importance of these factors with a broader sample of network professionals. Study 1 did not ask participants how risky each factor was, just the level of importance. Study 2 extended the Study 1 findings by asking network professionals to review one of three vignettes and judge how risky or safe each factor was in the vignette. We understood that network risk judgments are difficult to make without contextual information so Study 2 provided contextual information that Study 1 lacked.
Study 2 was designed to help refine our understanding of the factors that were identified as important in Study 1. For example, factors relating to the adversary (knowledge/skill/capabilities) were considered highly important in Study1 and were also associated with higher levels of network risk. Many of these factors detailing the adversary formed the adversary/threat dimension in Study 2. Also, in Study 1, factors detailing the organization were found to have different levels of importance. Study 2 helped us refine our understanding of the organizational factors that were and were not important. Specifically, factors relating to the organization’s network infrastructure and its ability to defend against attacks were relatively more important than others in Study 1 and were associated with lower levels of network risk in Study 2. Both studies had two dimensions in common; threat/adversary and the organization. The two dimensions discovered in Study 2, information and personnel skill, were related to the subset of Study 1 organizational factors that seemed to have high variability in importance ratings. Hence, whether or not context was present, our sample of network professionals believed that dimensions of network risk should include both organizational network infrastructure (the preparedness for attack) and the threat/adversary (the attack). This provides some clues about the dimensions of network risk definitions that network professionals, rather than risk metric designers, endorse.
The finding that network risk judgments are strongly influenced by information about the adversary/threat is important because network certification generally neglects threat/adversary factors. For example, the NIST CVSS v2.10 metric focuses mainly on other factors associated with information, infrastructure, and personnel skill. One reason why the threat/adversary factors were both very important and risky to our study participants is that the existence of unknown, dangerous entities (i.e., the adversary) which cannot be easily perceived and controlled is anxiety provoking. While the factors we provided about the adversary were of known qualities (e.g., the adversary has excellent cyber offense skills, the adversary is highly motivated), the adversary still poses an uncomfortable uncertainty in network defense because one cannot predict when an attack will occur (and by whom), how the attack will be executed, and what the adversary wants. Prior research has indicated that people are generally uncomfortable with uncertainty and typically avoid it [25,26], and when uncertainty cannot be avoided, a fear response is invoked . When fear increases, perceptions of risk also increase [28-30].
The importance of our results is that we used research methods from the social sciences to devise a list of factors that impact network risk judgments from network professionals. This information is important for risk metric designers who require metric users to subjectively interpret various factors as part of the metric output. Given little information is published on how certain factors were chosen for these network risk metrics, it is possible that these factors were chosen according to the opinions of the risk metric designer rather than the opinions of network professionals. Factors that our sample agrees are more important to risk judgments may not be the factors the metric designer includes; which we detail in the subsequent paragraph. We make the argument herein that network risk is difficult for one person to accurately judge given the technical knowledge diversity required. Therefore, consensus on factors important to risk judgments from a sample of network professionals may inform risk metric designers. In addition, future research on network risk perception and judgment can build upon our findings.
The factors our sample agreed were risky and important to network risk judgments were not necessarily factors included in computed risk models like the NIST CVSS V2.10. For example, the CVSS V2.10 includes factors describing organizational and network readiness rather than the adversarial capabilities. While it could be argued that the NIST CVSS V2.10 metric assess vulnerabilities, the metric is being used for security risk management (http://www.first.org/cvss/cvss-guide). Our research identified drivers for high network risk levels that are missing or not well articulated in this NIST metric: From Study 1, “the adversary’s knowledge about the organization’s deployed network and security technology” and “the adversary’s level of skill (professional vs. amateur)”. The importance of these missing factors was confirmed in Study 2 when participants rated the adversary’s skill and training as factors that greatly increase network risk levels. Other missing factors that we identified as contributing to risk perception were perceived adversarial motivation, success rate of the adversary exploitation in recent history, and the importance of the targeted data to be exploited. Factors that reliably increase or decrease perceived risk are likely to be important for an accurate computed risk model.
Limitations and future directions
A few limitations are worth mentioning that may have impacted our results. First, the target population in these two studies was difficult to persuade to participate in our studies. Network professionals familiar with adversarial techniques used to penetrate a network, may mistake the emailed survey links for a phishing campaign. Therefore, sampling was challenging, which was reflected in our low sample sizes and consequently, we were limited in the types of statistical analyses we could conduct. For example, we wanted to relate the risky/safe ratings for each factor to the overall network risk rating of each vignette but the low sample sizes made that impossible. We also wished to assess group differences (military, private industry, government) but again, the sample sizes encumbered that effort. One way to reduce the time burden for study participation was to only require questions that were central to the study objective. Consequently, not all demographics questions were required in both studies, which resulted in low question response rates that could never characterize the sample.
Another limitation is that it is unclear whether our findings reflect the judgments of DAA and IA officers. The process of network certification described in our study may be of little relevance to the broader sample of individuals involved in network defense. The derived judgments and perceptions may not align with those of DAA and IA officers who conduct network certification in the public sector, especially military organizations (the relatively low representation of public sector respondents in Study 2 underscores this limitation). Similarly, construct dimensions derived in our data-driven approach may be a reflection of the views and experience of the participants in our study. While it was impractical for our studies to sample exclusively from personnel responsible for configuring and certifying networks, future research should include validation studies to determine if our results are consistent with judgments obtained from individuals directly responsible for network certification.
In addition, future research should continue to flesh out which factors are significantly risky and safe in various network contexts and why. Our results were intended to serve as a foundation upon which future research and operations can be built. For example, network risk metrics in operations could improve the validity of network risk metrics by including some of our most agreed upon risky and safe factors. Researchers could investigate whether commonly agreed upon dimensions relate to factor perception and definitions of network risk.
Joint Task Force Transformation Initiative, National Institute of Standards and Technology (NIST), Security and Privacy Controls for Federal Information Systems and Organizations, NIST Special Publication 800-53, Revision 4. (Washington, D.C., National Institute of Standards and Technology, 2013). http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r4.pdf.
Oxford English Dictionary (online) (Oxford/New York, Oxford University Press, 2014). risk, n. http://www.oed.com/view/Entry/166306?rskey=Z0aceK\&result=1\&isAdvanced=false (accessed November 17, 2014).
Joint Task Force Transformation Initiative, National Institute of Standards and Technology (NIST), Guide for Conducting Risk Assessments (NIST Special Publication 800-30 Revision 1). (Washington, D.C., National Institute of Standards and Technology, 2012). http://csrc.nist.gov/publications/nistpubs/800-30-rev1/sp800_30_r1.pdf.
P Slovic, B Fischhoff, S Lichtenstein, in Perilous progress: Managing the hazards of technology, ed. by RW Kates, C Hohenemser, and JX Kasperson. Characterizing perceived risk. (Boulder:Westview, 1985), pp. 91–125.
B Fischhoff, SR Watson, C Hope, Defining risk. Policy Sci. 17, 123–139 (1984). doi:10.1007/BF00146924.
B Fischhoff, in Oxford Textbook of Public Health, Fifth Edition.ed. by R Detels, R Beaglehole, MA Lansang, and M Gulliford. Risk Perception and Communication. (Oxford:Oxford University Press, Sage;2009), pp. 940–952.
EC Poulton, Bias in Quantifying Judgments (East Sussex, UK:Laurence Erlbaum Associates, Ltd., 1989).
JW Creswell, Mixed-method research: Introduction and application. In C Ciznek (Ed.), Handbook of educational policy (San Diego, CA:Academic Press, 1999).
O Renn, Three decades of risk research: accomplishments and new challenges. J. Risk Res.1, 49–71 (1998). doi:10.1080/136698798377321.
LJ Cronbach, PE Meehl, Construct validity in psychological tests. Psychol. Bull.52(4), 281–302 (1955).
T Aven, Foundations of Risk Analysis: A Knowledge and Decision-Oriented Perspective (West Sussex, UK:John Wiley & Sons Ltd, 2003). ISBN 0-471-49548-4.
LG Epstein, A definition of uncertainty aversion. Rev. Econ. Stud.66(3), 579–608 (1999).
WW Lowrance, Of Acceptable Risk: Science and the Determination of Safety (Los Altos, CA:William Kaufmann, 1976).
A Pollatsek, A Tversky, A theory of risk. J. Math. Psychol. 7(3), 540–553 (1970).
RA Bauer, Consumer Behavior as Risk Taking. In RE Karp (Ed.), Issues in Marketing (New York:MSS Information Corporation, 1999). ISBN 0-8422-5165-0.
AH Crespo, IR del Bosque, MMG de los Salmones Sanchez, The influence of perceived risk on internet shopping behavior: A multidimensional perspective. J. Risk Res.12(2), 259–277 (2009). doi:10.1080/13669870802497744.
GR Dowling, Perceived risk: the concept and its measurement. Psychol. Mark. 3(3), 193–210 (1986). doi:10.1002/mar.4220030307.
HG Gemünden, Perceived risk and information search: a systematic meta-analysis of the empirical evidence. Int. J. Res. Market. 2(2), 79–100 (1985).
YY Haimes, On the complex definition of risk: a systems-based approach. Risk Anal. 29(12), 1647–1654 (2009). doi:10.1111/j.1539-6924.2009.01310.x.
CA Ingene, MA Hughes, Risk management by consumers. In EC Hirschman (Ed.), Research in Consumer Behavior, Vol. 1 (Greenwich, CT:Emerald Group Publishing Limited, 1985).
I Ross, Perceived risk and consumer behavior: a critical review. Adv. Consum. Res.2(1), 1–19 (1975).
P Biernacki, D Waldorf, Snowball sampling: problems and techniques of chain referral sampling. Sociol. Methods Res. 10(2), 141–163 (1981). doi:10.1177/004912418101000205.
DE Hartley, Job Analysis at the Speed of Reality (Amherst, MA:Human Resource Development Press Inc., 1999).
H Beyer, K Holtzblatt, Contextual design: defining customer-centered systems (Oxford, UK:Elsevier;1998).
G Hofstede, Culture’s Consequences: International Differences in Work-Related Values, (Beverly Hills, CA: SAGE Publications, 1980).
PW Dorfman, JP Howell, in Advances in International Comparative Management.3, ed. by EG McGoun. Dimensions of National Culture and Effective Leadership Patterns: Hofstede Revisited. (Greenwich, CT: JAI Press,1988), pp. 127–150.
G Hofstede, Culture’s Consequences: Comparing Values, Behaviors, Institutions, and Organizations Across Nations, 2nd Edition. (Thousand Oaks, CA: SAGE Publications, 2001).
B Fischhoff, et al., How safe is safe enough? a psychometric study of attitudes towards technological risks and benefits. Policy Sci. 9, 127–152 (1978).
JS Lerner, RM Gonzalez, DA Small, B Fischhoff, Effects of fear and anger on perceived risks of terrorism a national field experiment. Psychol. Sci. 14(2), 144–150 (2003).
P Slovic, E Peters, Risk perception and affect. Current directions in psychological science. 15(6), 322–325 (2006).
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute. No warranty. This Carnegie Mellon University and Software Engineering Institute material is furnished on an “AS-IS” basis. Carnegie Mellon University makes no warranties of any kind, either expressed or implied, as to any matter including, but not limited to, warranty of fitness for purpose or merchantability, exclusivity, or results obtained from use of the material. Carnegie Mellon University does not make any warranty of any kind with respect to freedom from patent, trademark, or copyright infringement. This material has been approved for public release and unlimited distribution. Carnegie Mellon®; and CERT®; are registered marks of Carnegie Mellon University. DM-0001925.
The authors declare that they have no competing interests.
JC conceived, designed, and carried out the studies, oversaw analyses that were performed, and was the primary author of a technical report documenting the studies. FG supported analyses that were performed, participated in interpreting results and implications of the work, and drafted the version of the manuscript for publication. BW conducted statistical analyses and helped to interpret results. All authors read and approved the final manuscript.
An erratum to this article is available at http://dx.doi.org/10.1186/s13388-015-0020-1.