The 2013 DBIR has been released and it represents an amazing amount of work by a great number of organizations and people. We did a couple of interesting things this year from a data analysis and management perspective and I want to touch on a few with the hope that others may catch on to some of these and do similar things in their work or even take it the next level.

A Matter of Perspective

Even without running the numbers, just looking at figures 2 and 5 (industries and org sizes) in the DBIR, we can see that there are patterns in this data.  Not all industries are equally targeted and not all organizations face the same type of threats.  While it’d be great if we could all do all the best practices out of the gate, it just isn’t feasible in most cases.  Organizations have to make tradeoffs and prioritize how they implement information security.

We are trying to help with that priority by breaking down the variables we look at by the size of the victim organization and the type of threat actor.   For example, the financial and retail industries didn’t see a lot of state-affiliated attacks therefore, readers in those industries should focus on the darker green sections in the DBIR graphics.  Manufacturing and professional services (consultants) saw very little financially-motivated attackers, so they’d want focus on other things in the report.  While we still have a long way to go in how we break down and prioritize this landscape, we have started heading in that direction.  Hopefully others will realize that too and begin to focus more and generlize less.

The “n” Crowd

It’s plastered nearly everywhere: 621 confirmed data breaches in this report.  But if you look at all of the horizontal bar charts you’ll see a gray number in the lower right, of which some say 621, and many show less.   This is showing the “n” of the sample we’re looking at, which is “out of how many”.  For example, if we look at figure 24 (“Vector for hacking actions”) the “overall” is looking at the hacking vector of 326 breaches, which should give us some good confidence in the overall proportions. We can then narrow down our view to larger organizations which shows 93 breaches.  

As another example, let’s take the web application vector.  Overall we see 22% of hacking attempts at the web application layer, and in large orgs we see 27% percent.  We could easily conclude that 27% is larger than 22%, right?  If we calculate something called the binomial confidence interval we see that the 95% confidence for overall is between 18-27%, while the interval for the larger orgs is 19-37%.  See the wider range when the “n” goes down?  As we increase the number of observations our confidence increases in the accuracy of our observations. This is why it's so important to share the "n" value, more data generally means more confidence.  The interesting thing about this is that most people intuitively know this and we naturally have less confidence in smaller samples (unless there is a good story wrapped around it, but that’s a whole other topic). 

The point here is that we’re trying to convey what the data is saying, and how much data we have is an important part in that story. Please join us in the "n" crowd!

Releasing Released Data

Finally, there are many challenges in sharing data and information.  But we had the realization that we’re already releasing our aggregated data in visual form, so why not take the next step and release the data the visualizations are made from?  And this is what we’ve done this year.  Let’s take figure 24 again.  We see the 22% out of 326 breaches leveraged the web vector.  But if we go to the DBIR download page and select “Raw data for DBIR graphs and charts“ (zip file), we can see the break down in figure 24: 45 breaches attributed to financially motivated attackers, 1 to espionage, 14 to activism and 12 to “other”.   The data we are releasing is already in the visual, it’s just less precise.   Enjoy!

So those are some of the major things we focused on this year: more focus, less generalization; specifying "n" values; and releasing the specific data behind the visuals in the report.  Hopefully this may raise some awareness for any consumers of research like this, but perhaps other folks who publish research and similiar reports will pick up on these things in their own work, or who knows, perhaps take their research to the next level, which would benefit us all.

One of the recurrent themes in risk intelligence this week was misuse of ICT, both intentional and unintentional resulting in losses. Goldman Sachs reported the Bloomberg terminals they permitted in their offices have been used to track some of their employees.  To their credit, Bloomberg acted quickly to correct the problem and issued a mea culpa.  PHH Corporation had a temporary employee abuse system access and has since been indicted. Two hospital employees pleaded guilty to identity and tax fraud to the tune of more than US$300K.   An Alabama state employee misused a state database and, with two others, also attempted tax return frauds. Dawn Cappelli from CERT did an interview and podcast: Mitigating Insider Threat From the Cloud. Another theme was targeted attacks: Trend Micro, ESET, Symantec and FireEye published reports on separate targeted malware campaigns.  Adobe, Microsoft and Mozilla each added security updates to our to-do lists.  Adobe's security bulletin on Cold Fusion addresses a vulnerability that has been used to compromise web sites.

The paragraph above is taken from the executive summary of the RISK Team's weekly INTSUM report. Verizon security product customers should access the full INTSUM via your portal.

May 7, 2013 was touted by Anonymous and its allies as the day they would hit the United States where it hurt by carrying out #OpUSA. Not surprisingly Anonymous’ bite didn’t match its bark and May 7 came and went with a very limited number of DDoS attacks, breaches and defacements. Even the Izz ad-Din al-Qassam Cyber Fighters, the thorn in the side of U.S. banks, didn’t join the operation. RISK Team assessment confirmed; #OpUSA was a dud. What wasn’t a dud were the attacks against WTOP Radio, Federal News Radio and other media sites which resulted in redirects to exploit kits serving up FakeAV and Zeroaccess. In vulnerability news, Microsoft pre-announced 10 bulletins for May’s Patch Tuesday and also released a fix-it for the vulnerability in Internet Explorer 8 (CVE-2013-1347) recently exploited by watering hole attacks on a U.S. Department of Labor site. We’ll have to wait until Tuesday to see if Microsoft releases a permanent patch for that vulnerability this month. Adobe dealt with its own vulnerabilities this week by releasing a security advisory for a flaw in ColdFusion and pre-announcing updates for Reader and Acrobat due to be released on May 14. And if you’re interested in reading something serious from the satirical news organization The Onion, check out its tech team’s post-mortem on how the Syrian Electronic Army compromised its Twitter account. If you’re looking for a laugh, check out The Onion poking fun at itself over what happened. 

The paragraph above is taken from the executive summary of the RISK Team's weekly INTSUM report. Verizon security product customers should access the full INTSUM via your portal.

Being a co-author of the Data Breach Investigations Report has certainly been a great experience this year. It’s exciting to work with dozens of people (and 19 contributing organizations) who are all dedicated to the common goal of collecting, documenting and analyzing thousands of breaches for the sole purpose of understanding more about our threat landscape than we knew before. Since we released the 2013 report though, I’ve also realized that the saying, “you can’t please everyone” is an unbelievable understatement. Overall, the amount of constructive criticism we receive is healthy. We are continually challenged year over year to do better, and we hope the report reflects our dedication to improvement. But a few opinions are either misinformed or a misunderstanding (or both) and those types of things we like to talk about.

There was such a post from Marcus Ranum this week. Even though he states early on, “please don't take anything from this point forward very seriously”, we felt obliged to respond, since not only is there some truth in the write up, but also some common misunderstandings as well (that aren’t limited to just Marcus) and we wanted to cover these things.

Ermahgerd Cherner!

I was somewhat confused when I read, “Obviously, the case is being made that China is a problem.” I could see how someone may think that if they were to just look at the pictures. Clearly someone would read the report before criticizing it, so this is good feedback. We must have been unclear when we stated the presence of China on the list “may mean that other threat groups perform [espionage] activities with greater stealth and subterfuge.” Plus, the breaches that were attributed to China this year represent less than 20% of all of the confirmed breaches we collected and analyzed: not the majority. We did put a lot of effort into not sounding alarmist in the report, because there are a number of possible and contributing reasons we saw breaches from China, only one of which is related to the actual amount of breaches from China. Perhaps a heightened awareness of Chinese actors within the intel community caused a higher rate of detection, or perhaps the addition of partners like ES-ISAC and ICS-CERT brought more visibility to our data set.

What we tried to get across is that organized intellectual property theft was one of several clear trends we saw emerging in our breach data this year. The majority of that class of attack in our data set originate from China and the chart shows that China, Romania and the U.S. are major sources of breaches in our data.

(click "see more" below to see images)

On Communicating Just Enough

Another challenge year over year is the assumption that we loosely define our terms or that we haphazardly slap the data together in our free time. Even normally intelligent and lucid thinkers are quick to assume our process is less than rigorous and that they are the first to envision challenges in data analysis. We certainly do have our fair share of challenges (such as measuring or reducing our uncertainty with the impact of breaches or presence and strength of controls). But we’ve spent many long hours developing and refining terms and our classification techniques over the years.

We would encourage anyone with doubts or questions about the classification techniques in the report to spend some time surfing around http://veriscommunity.net (and even join the mailing list there and ask questions). Even though we call out the VERIS website in a sidebar in the methodology section (“A brief primer on VERIS”) on page 8, and again in the “Results and Analysis” section on page 12, we still get comments like ‘the vagueness of "external actors"’ which is rather clearly defined:

External threats originate from sources outside of the organization and its network of partners. Examples include criminal groups, lone hackers, former employees, and government entities. Also includes God (as in “acts of”), “Mother Nature,” and random chance. Typically, no trust or privilege is implied for external entities.

Hopefully, the initial concerns about “vagueness” are removed with that definition. Now, we could very easily spend a healthy portion of the report defining our terms in great detail, but that’d be difficult to do while keeping readers awake, plus it’d make a long report even longer. By putting the terms on veriscommunity.net, we can still be very specific in our classification techniques and practices where they matter, but more conversational in the report. Obviously this leaves the report open to misinterpretation and criticism, as we’ve seen, but that’s what posts like this are for.

To anyone who has read “How to Lie with Statistics”:

As Marcus points out, this is a fantastic book. Even though the samples and illustrations are now quite dated, it still applies to the presentation of statistics. But please, please, please do not let this be the only book on statistics you read, as it may cloud your perception and probably give you an unhealthy perspective on statistics. Yes, it’s easy to lie with statistics, and it’s even easier to lie without statistics. But want to know what is not easy? Telling the truth with statistics. We have collected some very unique and powerful data here. We not only have to communicate to an audience that is generally highly intelligent and equally highly skeptical (especially of anyone remotely labeled as a “vendor”), but this same intelligent and skeptical audience has also generally not focused their studies on statistics. That’s a tough audience any way it is sliced and presents us with a lot of tough decisions throughout the report creation.

Take the figure that Marcus calls out, Figure 13, showing the country of origin for external actors. We are analyzing 621 confirmed breaches in the DBIR, and of those, we were able to attribute 398 breaches to an external actor with a known country of origin. Being fully aware of our methodology, we never made the (clearly incorrect) statement of “30% of all attacks come from China.” But we do want to show that 30% of the breaches involving external actors were attributed to actors in China in our data. See the difference? We don’t have a whole lot of options for showing that, especially to a skeptical audience. For those interested, we did release the data behind the visuals (zip file). Please feel free to offer improvements.

We did toy around with showing confidence intervals, but this opens up many more challenges and so we ultimately decided against it. But let’s dig into that option here. Take this updated Figure 13 as an example (click "see more" below to see images if none show up):

This is showing a 95% binomial confidence interval for each of the top 10 origins of external actors that we recorded. How many of our readers can understand what this truly means? This certainly communicates something different and in my humble opinion something more powerful than the original. But even people who have taken a statistics course in college misread 95% confidence intervals as a 95% probability (it is not).

One of the main purposes of a confidence interval is to account for sampling error in our inferences. Since we have both sample error and bias, it’s safe to label the 95% confidence interval as an overly conservative estimate of our confidence. But even knowing that, we can still be fairly confident in saying Armenia is much less likely to show up as an origin for external attackers than China or Romania. But what do you think? Does this type of chart just raise the likelihood of being misinterpreted or does this help communicate the data? We’d certainly like more feedback on this type of question.

Wrapping up with Sample Bias

Finally we get to the final point of sample bias. Marcus drops a reference to sample bias in the DBIR in passing, and rightly so. Sample bias is certainly a challenge for us, as it is for everyone in this industry collecting data and producing reports and we often talk about it internally. Because of the sample bias, we are limited in our inferential ability (or more appropriately, it limits the confidence we should have in our inferences - see the example above). The good news here is that we are not unique. Multiple other industries face the same types of challenges (researchers can only study disease in patients with that disease for example).

All in all, Marcus has some valid points, though the reference to self-selected surveys was a bit out of place. The DBIR is not a survey. It’s true that we must be careful to not ask the wrong questions of our data (as Marcus does tongue-in-cheek), but even with the right question we have to be careful how much we can infer from the data. Hopefully, we have walked that line carefully in the report and didn’t misrepresent the data in anyway. But the benefit of doing a report like this is that we get to learn year over year, and we’ll have an opportunity to do better next year just as we do every year.

And Marcus: if your network is compromised with a Trojan (malware), that would be a single security incident. If more than one asset was involved, there would be more than one asset associated with the incident (and probably other hacking actions, and other malware varieties) in the incident and each variety of asset would be recorded for what it is (from the asset enumerations). If we can show that one mint lemonade recipe was stolen, that’d be one record of a trade secret and we’d now mark it as a confirmed data breach affecting the confidentiality attribute. If there is uncertainty in how many records were stolen but we know one or more of your Sisters of Mercy MP3s (copyrighted data) were stolen, then that’d be marked with “unknown” quantity (see Figure 37, we had many unknowns there this year). We also may begin the actor attribution process by looking at the malware used and seeing if it communicated with a known infrastructure or if it could be linked with previous actions or related tactics (geo-location of IP is insufficient for attribution). Hopefully you get the picture: a single event is not a simple set of key-value pairs. It is a relatively well-defined collection of lists and associations, and if we have unknowns, we have unknowns, and nothing is “made up out of thin air”.

See, all you had to do was ask, and we’ll help you classify your incident using the VERIS framework.