Several months ago, I decided to write an article about a passion of mine, what I refer to as "Other Data”. It has been an interesting struggle to articulate in words what I would normally be able to explain verbally. The challenge is there are two aspects to “Other Data”; one a business culture issue, the other technical. The business issue deals with understanding what “Other Data” is, its significance within a corporate environment, and why it is sometimes overlooked. The second is more technical, and discusses an approach to organize and manage “Other Data”. To simplify my life, and direct the discussion to the appropriate audience, I have decided to write the article in two parts. This being the first, deals with what “Other Data” is, and how it is often derived.
What is “Other Data”?
To be clear “Other Data” is a term that I have coined. There may be a proper name and definition for it, but I believe the term to be very apt. Mainly it refers to data other than what is generally focused on by core business functions. To understand what “Other data” is, it may be best to discuss what it is not first.
Core Corporate Data
Anyone who has ever dealt with data, either from a financial, managerial or technical perspective has likely heard the terms mission critical, enterprise, or core data. These terms are used frequently to grab the attention of both technical and non-technical members of an organization and are the often used by a number of technology based industries such as Enterprise Resource Planning (ERPs), database management, Business Intelligence (BI), and Spatial Management systems. One of the technologies that frequently generates “Other Data” are ERPs.
The basic ERP Business model
An ERP system is effectively designed to ensure that a business’s key operational data is captured and available to all stakeholders as efficiently as possible. These systems can be very expensive to plan, deploy, operate and maintain. Ensuring that the data that goes into them is standardized as much as possible will ensure that a company’s core information is reliable and accurate. Failure in the planning or proper implementation of the system could cause a catastrophic business failure. We need only look at the example of the implementation of the Canadian Federal Governments Phoenix system to grasp this.
The 80/20 rule, Core and Other Data
The classic 80/20 rule can be used to help define “Other Data” within an ERP environment. With an ideal ERP scenario 80% of a business’s overall data would be captured with 20% of an overall estimated effort required to capture all data. The 80% would be the core corporate data, considered High Value and High Volume, whereas the 20% would be the “Other Data”, considered to be of low volume and low value by comparison.
Now, before someone gets all offended that their data is not considered either high value or low volume, bear in mind the value of the data is a function of the cost and associated risk to a company if they didn’t have it. Cost can be quantified in terms of missed opportunities or loss of revenue.
Being Relegated to Other Data
It goes without saying that an ERP deployment is not fun for anyone, each level of an organization will participate in it, and not everyone will be happy. However, within a deployment several steps will be taken to evaluate whether data is included or not within the 80%. Data will be evaluated to determine if it is sufficiently standardized to be included within the system. If not, it should be evaluated to determine if it can be standardized easily enough that it can be included within the system. If it cannot then it will be evaluated to determine if it is critical enough that it must be included in the system. This is of course an over simplification, but it does highlight that a process of identification is taken, and that at the end there is likely to be “Other Data” left over. Data that would cost four times the capital to capture as the standardized data.
Relegated May not be a Harsh Word
It may appear that having data relegated out of a primary ERP implementation is a bad thing, but the primary reason for its being excluded has more to do with the fact that the data is complicated with a lower risk factor. In many cases groups and individuals managing this data may be requested to continue to do so. This is generally a good thing as the alternative would be to attempt to fit that square data into a round hole.
The Silver Bullet Myth
I am often surprised at the number of people that are convinced that they have a system that is, or will be the all-encompassing system for a company. This is the Silver Bullet Myth, where one system will do everything, and unfortunately that is misleading. Essentially, there is no Silver Bullet, even if you had unlimited funds available, I am not certain one could be built, and it would certainly not be practical. Quite possibly one of the least productive things that could be told to persons involved in managing “Other Data” is that their data will be incorporated into the system in the next version’s implementation. Again, this is misleading as the Return on Investment (ROI) on the data will not have changed; ERPs are designed to streamline business systems, essentially saving companies money and investing in that additional 20% is likely to be counterproductive.
The Reality of “Other Data”
As things stand, organizations that have “Other Data”, which is every organization I have encountered in 20 years of data management, are left with both management issues and an opportunity. The challenge is the longer “Other Data” is stored the more unmanageable it will become. Take for example a simple census type survey; over the course of years surveys change as different questions are introduced or modified. These types of surveys and data sets exist across all industries and companies dealing with regulatory reporting deal with this type of situation often as both regulations and the questions they need to answer change over time. The opportunity conversely, is that when organized and managed properly, it enables the “Other Data” to be leveraged, giving an organization the ability to tap into other potential revenue streams or mechanisms for cost saving efficiencies.
Moving Forward
This article has focused upon “Other Data”, where it is derived from, and some of the inherent problems with it. The Other Data Part 2 will provide a technical option for how to manage “Other Data”, focusing on how it can be stored within a database, and using that database to leverage value from it, that may not otherwise have been realized. As mentioned within this article, there is no Silver Bullet solution, however, data may exist in a place where when a system suitable for its use is available, the data will be ready.