Digital Asset Repository - Large Oil and Gas Company Case Study
How D3Clarity architected a Digital Asset Repository that auto-cataloged a million+ assets, provided individual transparency to all of the client’s data assets, and created the foundation for analytic dashboards and visualizations
How D3Clarity architected a Digital Asset Repository that auto-cataloged a million+ assets, provided individual transparency to all of the client’s data assets, and created the foundation for analytic dashboards and visualizations
Business Challenge
For this business, D3Clarity was asked to create a data catalog of their data assets from the myriad of source systems (primarily in SAP) and translate technical database terms into business-friendly asset names and terms. This company had over 10 million assets to be cataloged and was overwhelmed by the work. Data assets for this customer were scattered across over ten different source systems. The initial estimates to provide a comprehensive data catalog (in-house by the client) were over nine work months to complete, which equated to a cost of $1.6million
The Solution
Through consulting, D3Clarity articulated the real business challenges facing the customer. Over the years, experience has shown that the best solutions come from asking better questions. In this case, the customer thought data assets had to be cataloged. In reality, D3Clarity discovered that 85% of the customer’s assets were essentially junk. In fact, only 15% (1.5 million) of the assets needed to be cataloged. Whenever you catalog a data asset, time, effort, and money are spent to do that work. Reducing what you have to catalog to the true elements leads to significant cost savings.
The next problem discovered through consulting with D3Clarity was that the customer was facing a lack of trust in the data when it was presented using analytics and visualization. In the past, this client was used to consuming data from PowerPoint presentations, which meant the presenter controlled the narrative. This style of communication proved to be dangerous as individuals lost trust in the narratives because there was never a correlation between the presentation language and the derived results.
Live analytics and visualizations force a powerful change to the narrative. It puts the power of discovery in the hands of everyday people. This client bravely decided to change the approach around data from reporting to analytics. Still, unfortunately, the battle was not yet over as the mistrust of data from the past flowed into every analytics dashboard and visualization. Can you imagine how frustrating it is to convince each viewer that your data is sound constantly? What is the solution to this problem? The answer
– Complete data transparency through total data lineage! D3Clarity provided a tool using graph database technology to show the data genealogy that assembled each data asset visually.
The Result
D3Clarity provided an initial prototype that handled 70% of the client’s needs in 7 weeks. The system automatically cataloged over 90% of the critical assets (1.5 million of 10 million). This means that the system auto- cataloged 1,350,000 assets. Only 150,000 assets needed human intervention in some manner to be cataloged. Based on the client’s original assessment, they thought they needed 25 full-time equivalent people to handle the original cataloging of the entire asset set of 10 million assets. The client now only needs one part-time person to catalog the remaining 150,000 assets
D3Clarity deployed a complete data lineage system that provided individual transparency to all of the client’s data assets. Now every user can see which data sets, queries, and aggregates are used to create analytic dashboards or visualizations. The data lineage system was an intangible benefit. No longer did the project team, project sponsors, or the digital transformation leadership need to answer the question of whether data was trustworthy. What do you feel is the value derived from the peace of mind coming from the fact that your users trust the systems you deploy?
In summary, this effort was projected to cost over $1.6 million and take over nine months if done in-house by the customer. D3Clarity provided an initial minimum viable product (MVP) covering over 70% of the requirements at $210K (a savings of over 86%) in 7 weeks (a time saving of over 80%).
