“Dark data” may be a relatively unknown term for many, even though all contribute to its growing presence. It represents data that is accumulated continually by the interconnected devices and systems used to run businesses, homes, and personal lives. It is information that is unused and often highly labor-intensive to access. A recent global survey by TRUE Global Intelligence estimated that an average of 55% of accumulated data is dark, unexplored, and virtually useless without the proper systems to analyze it.
The integration of big data is reshaping the way global organizations make critical business decisions. Well-executed data analysis has become a key driver in operational efficiency, and the methodology behind proper data analysis—applied data science—is now being widely taught at universities. In this digital world, the constant influx of data can become overwhelming and unmanageable, and the surface has barely been scratched.
The survey, which drew responses from more than 1,300 business and information-technology decision-makers globally, explains that most dark data is generated as a byproduct of our online lives, including what it calls “data exhaust.” Other studies have estimated that the data that people access and use regularly could be only 20% of what is available. In a modern world, where actionable information is a high-value commodity, why are so many unprepared to take full advantage?
According to the TRUE Global Intelligence report, the answer is inadequate processes, lack of resources, outdated technology, and neglect. The report states that 90% of surveyed respondents agree that every organization will need to extract value from data to be successful and adds that 21% say that recovering dark data is difficult because of a lack of interest from organization leaders. “They know that data is useful to mine, in other words, but they don’t all have the tools to go digging,” the report says. “And, in some cases, they don’t even know where to dig.”
Companies that are unable to tap into their wealth of dark data face the risk of getting passed by competitors who can. Without the proper investment into dark data and understanding its true potential, companies will continue to fall short of their ability to make the most highly informed business decisions.
“If you want to be ahead or not eliminated in a fierce market, you need the support of intelligence, and the vast majority of this information comes out of this chaotic data,” said one survey participant.
The solution to this underutilization could lie with academia and the next generation of professionals. “People should be worried about how we train the next generation not just of scientists but people in government and industry,” said Alex Szalay, an astrophysicist at Johns Hopkins University and data storage pioneer. In 2018, Szalay launched a project to construct one of the world’s largest science data storage networks, which would allow scientists to more efficiently store, analyze, and share vast caches of data with other researchers. If successful, this research network could be applied to other organizations. To extract and use untapped dark data, current students need the tools to navigate today’s mountains of disorganized information.
The oil and gas industry has been quick to embrace big data with widely successful results. Geological and chemical data received from wellsites allow companies to produce more, spend less, and avoid unnecessary risks. Organizations that do not prioritize their unmined dark data could face serious long-term implications. The TRUE Global Intelligence report states, “Often, organizations ignore potentially valuable data because they don’t have the time or resources to prepare it for use or they may not understand its full potential.”
The continued pioneering of artificial intelligence (AI) provides an opportunity to revolutionize the way information is processed. Engineering effective AI relies on a tremendous amount of data, and, while only 10–15% of companies currently use AI to augment their processes, that number is expected to grow to 60–70% in the future. This growth may reveal a way to shine a light on dark data, which could permanently change companies’ ability to make effective decisions.