Despite the connotations the name might conjure, Dark Data is not evil or nefarious and is like any other benign information companies might store. Typically, it’s what you might call the everyday data that most companies accrue whilst conducting their daily business. Because of its volume and velocity, it’s very rarely catalogued or filed in such a manner as to be useful in the future. It’s a little like the allen keys you keep collecting from your many adventures at IKEA. You’re sure you’ll never need them again but you keep them…just in case.
Of course with the proliferation of such open-source distributed file systems on commodity based hardware such as Hadoop and Cloudera, it’s now becoming much more economically viable to feed your inner hoarder.
So what’s the problem you ask? Well, all that data is a little like the seemingly innocuous tyre dump that sits dormant in an empty field until the right set of events (Lightening strike or Ebays hack of 145 million users) ignites the pile quicker than you can say “are we compliant” and you then have a blazing inferno with nothing left to do other than stand by and watch your brand reputation go up in flames.
So the issue now becomes one of compliance and the key to remaining so is to know what data you have. Unfortunately many companies have no way of doing so as they have failed to catalogue 90% of their Dark Data.
The Australian Privacy Act, and in particular the Australian Privacy Principles which form part of that Act, place stringent controls and requirements on the collection of personal information, the purpose for which it was collected, retention of it, security measures applied to it, and use of it. By not auditing your Dark Data, you also run the risk of keeping records or communications relating to what might once have been acceptable conduct or practices, but which have subsequently been legislated against.
It is bad enough when you know what is taken. If your company suffers a security breach and the data taken contains sensitive information that you did not even realise was there or are unsure about, it could give rise to major liability issues and significant damage to your reputation.
In some instances it can be a false economy placing large quantities of non-contemporary data onto cheaper data storage solutions, where there is inadequate or no knowledge of the contents of that data. Ironically, your data can be your greatest asset AND liability at the very same time.
Know what data you have, keep that information up to date and always be mindful of the above when creating your companies data policies.
Is your data hiding untapped opportunities? Axon are passionate about information and would love to talk to you about how to unlock the value of your data.