Are you afraid of the dark?

“Dark” and scary just go together, right?—Darkness at the Edge of Town, Dark Shadows, Beware of Darkness, Wait Until Dark, Heart of Darkness, House of Dark Shadows. . . you get the idea.

With Halloween on the brain, it might be a good time to take out your Zak Bagans “Ghost Adventures” Trifield Meter and investigate a phenomenon that’s scaring the tech out of IT professionals everywhere: dark data. While not completely new—dark data has been around since the dawn of the big data era—understanding and properly dealing with dark data could be a major part of any potential disruptor’s strategy.‍

What is dark data?

While the “Ghost Adventures” crew was silent on the subject, Gartner describes it as: “the information assets organizations collect, process and store during regular business activities, but generally fail to use. . . . [D]ark data often comprises most organizations’ universe of information assets.” 

60% of executives think 50% or more of their data is left in the dark, but they’re wrong. The reality is that 99.5% of all collected data is never used.

‍

Scary thought number one: While dark data comprises most of the information in a company, it is the least analyzed and acted on.

So, where did all that dark data come from?

Another take, according to TechAlpine’s Kaushik Pal: “Dark data is a subset of big data, but it constitutes the biggest portion of the total volume of big data collected by organizations in a year. Dark data is not usually analysed or processed because of various reasons by companies, but that does not lessen its importance in the context of business value.”

You’re definitely not the only one in the dark, if that’s any comfort. Dark data is an issue that most organizations struggle with, and the primary cause is lacking the infrastructure and resources to manage the sheer amount of data currently available.

Here are a few of the most common ways businesses are leaving data in the dark. ‍

1. Data without a strategy‍

Most organizations collect and store as much data as they can, often without a clear strategy in place for what they’re going to do with it. This isn’t totally bad—the more data, the better—but it leads to amassing data at a rate that can't really be processed and digested. Much of it soon becomes dark data, stored, taking up space and dollars, but never used.‍

2. Stakeholder misalignment‍

Various team members decide what data is tracked, how it gets analyzed, and what analysis is needed. Lack of communication and alignment between data engineers, analysts, and business people can quickly turn large quantities of data into dark data that, for lack of agreement, never gets touched again.‍

3. Lack of resources‍

Very few organizations actually have the resources to leverage all of their stored data. There’s just too much—and it just keeps coming. Very few agile organizations actually have the people, tools, and infrastructure in place to ever put that data to use—thus relegating it to the dark data heap.

Why is dark data so scary?

Alan Dayley, research director at Gartner, states that: “Increased data growth over the past decade has created an unstructured data nightmare."

Scary thought number two: Unlike the boogeyman in your closet, you can’t will dark data away. It’s there, and ignoring it will cost you.

Dark data assembles a fragmented view of your business because it disregards data that would create an otherwise complete picture of your customer, how they engage with your business, and—most importantly—how this impacts your bottom line. All of this results in massive opportunity loss that you can’t even quantify because you simply don’t know what you don’t know. How scary is that?

How can I shine a light on dark data?

Fear not, though, because there are ways to shine some light on all that dark data—or any data, for that matter—and it starts with what you proactively collect and analyze, and ends with insights that help you make decisions and drive your business forward.

1. Track what matters

Instead of tracking anything and everything because you can, decide what you need to track and why. Then track what’s most impactful to your business.

2. Hire the right team

Invest in resources that make sense for your business based on data complexity, insights needed, and how quickly you need those insights to run your business.

3. Invest in tech that works for you

Determine what you need in your tech toolbox to collect, analyze, and democratize data insights companywide.‍

4. Keep asking new questions

Get a handle on the data that matters and infuses curiosity into your BI strategy. Fresh insights guide where you focus next, which fuels further, deeper,  and increasingly more valuable analysis to reveal answers that help disruptors drive agility into their business practices.

We like to call the proactive exploration of data, “data navigation.” Next week,  we’ll share some details about this tech treat in part two of our dark data series.