Dark data is information we are collecting but not using. It can cause leakage of proprietary knowledge and reputational damage. It increases future compliance challenges and legacy costs. It hinders making good use of data and is unnecessarily increase resource usage without any productive outcome. What are organizations doing to handle these dark data perils?

Contributor

• • Prosenjit Sengupta, Executive Vice President & Chief Digital Officer, Thermax Limited

Download Podcast
Apple Podcast, Google Podcast, Spotify, Pandora, iHeartRadio, SoundCloud, TuneIn, and Stitcher. Find other syndication channels here or search CIO Talk Network podcast on any other app.

Explore More

• • Enabling Curiosity at Scale
  • Modernizing Data Management
  • Steps to Making Data Your Greatest Asset
  • Three Data Technology Trends to Watch in 2022
  • Using data to create a safe environment
  • Data Analytics as a Team Sport
  • Steps to Data Literacy and DataOps
  • Steps to Future-proofing Data and Analytics Environments
  • Building Digital Data Strategy, Architecture, and Infrastructure
  • Building Data and Analytics Driven Culture
  • Making Sense of Big Data

Transcript

Sanjog Aul [00:00:00]:
Hello and welcome to this segment on CTN to learn more please visit CIO talknetwork.com and we have a very interesting topic today that we’re going to discuss. The topic is Handling Dark Data Perils and we have with me Prosenjit Sengupta who’s the Chief Digital Officer with the Thermax Group. Hey Prosenjit, how are you?

Prosenjit Sengupta [00:00:19]:
I am fine, thank you very much.

Sanjog Aul [00:00:22]:
Thank you so much for joining us here. So the fact that we have dark data as the topic, which of course I’d like to give a background essential. We talk about dark data, about the information, it’s about the information we are collecting but not using and it stays obviously in front of us but also in those dark crevices of the organization and it is causing as we speak today, leakage to the proprietary knowledge. It can cause reputational damage. It could create compliance related challenges and of course costs related to what you would have similar to any other legacy applications and systems and data. There’s so much that is banking on making sure that we use our data

Sanjog Aul [00:01:10]:
but dark data exists and there is not much being done or at least there’s not a formal program in many cases. So why is it? What can we do to make sure it doesn’t cause us damage or cause us leakage? That’s why we are talking. So the first question for you, Prosenjit, is we have these many risks I mentioned about security, reputational and financial risks and we have a very fast changing environment and yes, we would like to have a full control on everything possible, but it’s sometimes not practical. So are we considering dark data as just a nuisance and living with it and that’s why it is called dark data or it is going to at some point become and will be treated as a real threat.

Prosenjit Sengupta [00:01:58]:
Today we face the problem of lengthy data collection and data storage have become much easier than before. Not to mention that the fact that they have become very cheap as well. In fact, if I quote the statistics, they say that data storage cost have steadily declined over the years at approximately 40% year on year over the last couple of decade. So infrastructure, which was actually a significant portion of the CIO’s budget is being rounded off as small change today. This is resulting in everybody trying to assimilate data about everything. There is data from systems, data from products, data from pro, and of course not to forget the data from the social media but the issue is that we have not yet mapped the speed of data assimilation with the speed of the data analysis. Our ways of work, our culture, our tendency to still stick to old models of reviews and governance create a very difficult environment of scene which is very necessary to break out into those unheaded data only business decisions.

Prosenjit Sengupta [00:03:06]:
More so I would say that it is very difficult for older organizations compared to the new Internet ready organizations like the Unicorn. So there is loads of data already started to become an nuisance, and it has started to become a  nuisance, not now, but already some years ago. There’s unnecessary information, lines cross, it is getting created in shared folders, as documents, as pictures, as videos are all getting created and we see that it is not being record. So there are versions upon versions getting created. I think there is an utopian notion that maybe all the versions will come in handy at some point in a future date, either for self reference or even for handing over to the next person in line but from my previous experience in the organization I have found that less than 10% of old stored data is ever accessed. So that’s really a very small amount of the data that we are actually storing across shared folders and making it accessible to people but now the nuisance really has turned into a  threat.

Prosenjit Sengupta [00:04:17]:
One I would say that the seemingly endless theme of data cannot all be categorized or constrained to selected groups, so they are getting shared all across. Number two, the usage of such data, again it cannot be restricted inside the organization or outside. So there are data which is in all different kinds of format, whether it is video, audio, written documents, everything and with the proliferation of data across all departments and all pieces, there can never be any hold on to whether that data is going to be used inside the organization or outside. In fact, with this tendency of the people to go social nowadays we find that there is a much higher proliferation of data to the outside world, and many times it has been done by people who are both unaware of the context as well as the repercussions of such an action. So I would say that this monetized data is now becoming a source of business weakness, embarrassment as well as sometimes also a kind of a political blunder and so I would say yes, it is no longer a nuisance, but it is turning very fast into a real threat.

Sanjog Aul [00:05:28]:
So that is a great explanation, Prosenjit, about how this earlier was thought as benign is becoming a threat. One is that it is becoming one and you and I can analyze it but are the organizations and their stakeholders really heeding the advice and doing something about it or we are going to see another decade of people saying oh I don’t have the time to deal with this dark data and they would try to work on the more new and shiny things or some things which are very obvious threats and this is going to come and bite them with time.

Prosenjit Sengupta [00:06:05]:
I would say that even though there is an understanding within the organizations, there’s a lot of data which is lying unused, and there might be a potential threat. However, I see that the organizations are not taking active steps the way the budgets are getting sanctioned. So let us say infrastructure upgrade or the new areas like digitalization. In fact, I am also, let us say one of the culprits, being the head of digital and having a mandate to digitally orient the organization. My budget shall move tuned towards digitally automating the organization rather than securing the data which is lying all across. However, I think that it is very important that we need to think about these angles as much as possible because if not sooner, but later we are going to come across a situation when all this data from really come back and start biting us in the wrong way.

Prosenjit Sengupta [00:07:07]:
So I think it is important it is urgent but unfortunately the organizations are not given in name good solid money for these ventures.

Sanjog Aul [00:07:18]:
So let’s talk about the personally identifiable data and other type of data which may be covered under some sort of a regulation or a mandate and that type of data is sitting in the pile for the data we are collecting and perhaps not using that. So we are giving that that term of dark data which is going to have exposure but ignorance is not bliss when it comes to regulatory and compliance related things. So are people not even looking at such things where they could pay heavy penalties and just trying to say if some nobody has seen it, that means it is not under the purview of compliance and that’s why they are letting it sit as is.

Prosenjit Sengupta [00:08:03]:
Yes, to a large extent that is the reason. Convincing the management to spend on data management is exactly like convincing them to spend money on disaster recovery setups or let us say information security system. The success of these platforms is actually evident when nothing happens or let us say when, basically if there is a negative impact or a threat or a hack that happens into the organization and there is a lot of data or lot of reputation losses that happens, that is the time when the management wakes up. However, the success of these platforms is not really evident on a day to day basis. In the normal running scenarios these trends are not appreciated and repeatedly the dissertants maneuver toward the probability ratios toward the percentages of them happening.

Prosenjit Sengupta [00:08:54]:
So what I would say is a good way of learning such debate is to prepare oneself with a notional monitoring office which accompany such hypothetical cases. In case let us say there is a hack or a breach on top. It is also prudent to be well versed with instances that had happened across industries and also the globe. Also, I would say that increasingly the families were not following good governance practises, doing businesses are also multiplying. Hence it might be a good idea to link the need of data protection or data management with the general risks and controls and the segregation of duties and other likewise statutory norms and principles to govern the business environment.

Sanjog Aul [00:09:40]:
So what you’ve given is the recommendation, well, this is what we should do, but is that happening and if not, to what degree is it happening and what’s missing?

Prosenjit Sengupta [00:09:50]:
So let me say that today, any organization which is using good ERP for doing their operational transactions and storing the data is in a much better position than the companies who are not using a defined ERP system because when we are using ERPs, we are collecting the data within the defined systems of record and then there is a very high ability after capturing the data to categorize and also to softly sensorize it before storing the data.

Sanjog Aul [00:10:26]:
So if you are to look at the next element that we should look at is the sensitive information we spoke about the compliance and regulatory related data which may be sitting there and we have not bothered in many cases to go after it but then there are certain other things which have a direct impact on our top line and bottom line, which could be the kind of business practices and workflows which are very unique to us, which give us a competitive advantage, or our system related data that we have created, some analysis we have done on what our customers like to buy, etc or there could be some partnership agreements that we have created which have given us the competitive advantage that I could just go on and on the type of things we’re talking here but even could be sitting in this dark data, the pool of dark data that we have. What are we doing knowing that we could be sitting on it and this could cause us a whole lot damage or if you uncover that and further mine it, you could further enhance your competitive advantage. So it’s not just about losses, it’s also about gains that are just sitting there waiting to be mined. What are we doing to get them out from that dark data and actually make some use of them.

Prosenjit Sengupta [00:11:47]:
Okay, let me first start talking about the dark data and how the data which is lying all across and is not being used. So what I would say is I fully agree with your question. The points that are mentioned, there’s a lot of important and sensitive data which is lying all across and many of these data is actually comprising of data which relates to computational advantages as well as the business practices, partnership agreements and many other things. My answer to that will again relate to what I was talking to you about in the previous questions related to the regulation and the financial liability part because the surestart way of ensuring that the sensitive business information and documents do not land at the wrong place. If so, wrap them up under a GRC follow segregation of duties hierarchy. Hence all the hard documents which are pertaining to these business sources or even the marketing unique salary propositions, the supplier and customer contracts all should be converted into electric form and they should be embedded under the related transactional and operational system. Once this is done then the assets can be restricted. For example print outs of all strategic documents  or business review trends should be completely avoided.

Prosenjit Sengupta [00:13:08]:
India plays a much more secure way of this to do it online and do online review with the business intelligent dashboards and other systems similarly, rather than circulating hard copies of document or even I would say going to the extent of emailing soft copies of documents, it is always a must preferred approach to use online scene share mechanisms where multiple people from multiple geographies can come together, they can read, they can contribute and then they can see an active part instead of sending documents and emails and other content all across to multiple people and multiple fields. These are some of the ways in which we are trying to actively tone down or reduce the dark data concepts in our organizations and pretty much we have also put in a restrictive practice not to share documents openly through all and sundry. Having said that, as I mentioned that today there is a data is getting collected and it becomes very important to start analyzing them. Unfortunately it is also before we start analyzing them and really start using the data to our advantage, we need to start segregating and categorizing them and that is one area where I would say that is an area most of the organizations are not doing, let me say optimum job. Then it starts with the data aggregation and segregation. It starts off then for a couple of months, but over the years again it’s looser and people go back to their old approaches of creating documents, sharing document lots of documents in the software manner and then lying all across and getting shared at all different places.

Prosenjit Sengupta [00:14:59]:
I would say that using these systems again going back to the ERPs and other such systems have got a very strong hierarchy of aggregation of DOC as well as widths and controls, embedded into them is very important for organizations to embrace so that these competitive advantages and the documents are safeguarded.

Sanjog Aul [00:15:23]:
What you are sharing, of course is a best practice. So on the ground one is of course we can throughout the show look at the opportunities that are there which are waiting and I know we are talking about the perils which we are already facing, what has been tried and what has worked in this regard. So you said we should do this and suggested some best practices. When we tried to accommodate those or tried to implement those, what were the challenges or what are the challenges we are facing because of which we are not truly making the desired level of headway in this situation.

Prosenjit Sengupta [00:16:02]:
I talked about a couple of areas of best practices. Number one, I talked about using the chief intelligence system for normal governance and review. The reason why many of the organizations go again and again to the documents and the forensic reports and let us say there is government starts before. There are a lot of tweaks that happen during the presentation. People are still comfortable in painting the data in their own formats rather than using the base data of the analysis and algorithm. That might be done for suiting one’s purposes, of course but it is also I think because of the long standing culture in the companies people must see more your suit looking at PowerPoint presentation and charts which have been created by people rather than online systems and having reviews around that but that is one area where as I mentioned best practice might be the use of BI system but people are going back into the presentation.

Prosenjit Sengupta [00:17:09]:
Second is I mentioned that it is very important to start sharing document through common share mechanisms within zephynes like theme share and other such things. However, again we have become so used to sending teams, sending documents, printing document, sending hard copies to multiple people. Even in meeting it becomes customary for example for people to bring four or five copies of the same presentations and the document and circulating it to the people. That there is basically not only sheer wastage of resources, but there is a very high probability that those same documents are going to be your and pass on into the outer world. I think what is stopping us is our the way we have been working so far. What is stopping us is our initial of maybe switching the data that we want to show and represent or discuss and I think that is where most of these processes or most of these  best practices are not really taking shape. Then it comes down to the reality.

Sanjog Aul [00:18:18]:
So if I were to ask you about the different places where we have attempted such things and you mentioned about the BI but people keep going back are there any strategies that have truly worked to get over this because a year from now, if I wanted to talk about Dark data, I hope I would not have to ask questions of this type where we’re saying why are we still stuck? Do you see silver lining in the cloud or we are going to be doomed a year from now as well?

Prosenjit Sengupta [00:18:48]:
I wouldn’t say we’ll be doomed a year from now of course, but the way just to give you examples, the bigger the companies, the bigger the possibility of them getting hacked. I have worked in Tata Motors in the past, I’ve worked in Volkswagen in the past and I’ve seen multiple cases where hacking attempts are made on a daily basis for this conflict and it is very important that we start securing our data. Of course we have got very good firewalls and all those things are there to prevent any hacking from the outside world but what I am really talking about, or rather I would like to talk about in this dark data is data which is getting produced internally or getting selected and it is in the hands of the employee, even in the hands of the various stakeholders like the customers, the suppliers and everybody else but this is getting indiscriminately fast all around. Not by intention of course, but also because of a lack of the knowledge just how severely the reputation damage or the competitive advantage might be lost

Prosenjit Sengupta [00:19:58]:
and so I would say that even though at the soft management there is an understanding and an awareness that there is a data that used to be pilot tapes were tested only the end maybe I will come back is to have these systems document the operations and the data rather than it is getting created in a fairly fast form, for example document, excel sheet, presentation. Four years since we are very familiar now over the last 10 to 15, 20 years we cannot imagine a world we cannot imagine a day in office without  working on these but unfortunately these are the very sources through which data is getting percolated everywhere. So I would say that organizations are moving forward will not say that everybody is doing but yes, I think we all need to become more and more aware of not using such documents. At least use them  current letter and use systems more. That would be the only way out of this that we got today. .

Sanjog Aul [00:20:58]:
Now anything which we want to change requires some sort of ownership and dark data is coming from all over the place or maybe it’s sitting all over the place and we say that it is not IT’s job to own the data. It should be embedded in the business or the ownership or stewardship should be with the business. Who is going to rally these folks or who is going to assign or have them recognize the responsibility so they at least start taking ownership and then eventually we’ll make this all possible or discovery and making use of dark data and then if you do it, that means the dark data is no longer dark data but to make that happen, you have to have first someone take ownership and someone to lead the charts to. Say folks, let’s get together to tackle this problem. Where are we with that? What is the kind of organizational structure that we need to have and the ownership responsibility before we can really tackle this dark data problem effectively? So let’s talk more about this when we come back. Please stay tuned listeners.

Sanjog Aul [00:23:16]:
Welcome back, Prosenjit we did speak a whole lot about the different challenges why people don’t tackle dark data effectively. Let’s get into the solutioning on how do we crack this problem and anytime when you have such a thing which is pervasive throughout the organization and you got people junior and senior equally clueless in many cases or not willing to lead the charge. How do you get people to come together? Who becomes the Pied Piper who assigns or makes people understand the responsibility and help them take ownership? How should someone work on this?

Prosenjit Sengupta [00:23:53]:
This does not have a straight answer. The question also evokes a lot of debate. My forefront view is that the owner is neither the IT department nor the user department by themselves. It requires attendant to be played. The IT teams need to put the technology stacks together to ensure that the data is being selected securely. IT is being analyzed inside the business systems and IT is built Crafts X Engineering sold the IT came also ensure that systems and mechanisms are in place to place IT in the GRC and the federalization of the UC conflict requirement. The fray of the user department comes in using these systems that have been laid out. The user department including the managers of each department need to ensure that adequate screening of the system and the usage or the labeling needs at faster are done for this model, the entire training is done properly. The use of the performance also needs to ensure that the proper, the segregation and the hierarchies are being followed and all the employees who are using the system are  catering and they are adhering to these hierarchies as far as possible this the solution to this does not slide one department or the other and hence it is the responsibility of the management, the apex management body of any company to seek the issue.

Prosenjit Sengupta [00:25:20]:
To give you an example of how we are catering and how we are handling in our current organization. So this entire topic has been segrated and crashed out and the ownership of DISH has been a selective entire FX management scene, including myself. However, we need to have one, let us say one mesh to hang and that is the person and I have been. Since I am responsible for the data and I am also responsible for the usage of the data and analysis, the need for this role has been made. As for this role, however, with the most we have done we have created a rasci matrix, take responsibilities and the ownerships and the accountability that is properly defined and as I mentioned, the responsibility of the technology team is going to be to provide the necessary system and the necessary hierarchies within the system and the role of the user default are going to be that they are going to be followed in a diligent manner. Of course we will not know whether all the founder systems are the soft or not. So what we have also agreed is that from time to time here and have multiple training sessions for the end user

Prosenjit Sengupta [00:26:41]:
as well as we will have I think letters in the internal audit snare role in seeing whether the rules and the regulations have been followed as a defined policy. So what we are hoping is that with this kind of a mixture of responsibility and ownership make sure we making a good headway into the  future.

Sanjog Aul [00:27:03]:
So basically based on what your response is, if I were to consider this as a playbook, a playbook for dealing with dark data, you are recommending that we go ahead and build a steering committee of sorts which has got representatives from different departments. We first deliberate on seeing what’s the value of what type of data which he should uncover in mind, build a business case around it. Then we get our respective staff brought up to speed on what we are up to, build some processes, get the managers involved, do some training and that’s when we will start seeing some progress in this space. So it’s not like a hundred percent clear playbook, but at least these are first few steps you’re recommending one should take on to tackle dark data type problems. Am I correct?

Prosenjit Sengupta [00:27:57]:
You are absolutely right and this entire topic has been learned from the top and there has to be guiding policies and frameworks that needs to be defined and also we should not forget that the technology and their systems are going to play a very important role. For before even we delve into this topic, it is very important that they do and manifest all the systems that we are having. So we should analyze all the gaps in those systems, we should analyze how we can make them more secure and only after that you should put up those policies and go about the screening of the people. Exactly what are the pitfalls of not doing it and that manner can take it forward  step by step.

Sanjog Aul [00:28:45]:
So you of course, as part of the playbook, did mention that indeed the whole agenda or the whole initiative should be driven by the top and we did talk earlier in the show that the people at the top, if they start looking at this as a regular governance or compliance, not a compliance, but a more a governance or an overhead, then they are not the most thrilled people. So how do you get them at the top to even agree to sit in the meetings, work with you, champion it and of course this is required funding. Whether you do analysis or you do further mining, all of that takes funding. How do you get them to agree in the first place to even get this started? What have you seen being successful?

Prosenjit Sengupta [00:29:29]:
Again, no easy answers here because the top management should also have the CFO questioning every single cent of the dollar. However, I think what helps are instances where there have been such data needs to release into fox. There are lots and lots of instances and stories that amounts. I think it is very important that we need to understand that we need to talk about the data, but all the data across all the systems and the department that are getting connected, it needs to be  painted in the proper way. It needs to be, the management needs to be shown the kind of data which is getting collected. There has to be specifications that needs to be discussed. For example, from one particular department in a particular month or in a particular year, the amount of data that has got connected, the amount of space it is taking up, the amount of analysis which has been done or could not be done because the data has not been categorized properly to show basically by, there are lots of tools, for example, that show that the photos of the various systems, for example, if I give you an example of that we are using, rather I have used in the past, there are multiple instances when we have found that the maker and the checker are the same person. So these are some of the hard facts that can be uncovered by doing an analysis of this distance and the analysis of the data that is getting collected

Prosenjit Sengupta [00:31:07]:
and once the data is presented in the proper way and showcase that these are the reputational or these are the  competitive risks that the company facing because of all this data that is designed is getting collected and lying that is unused in areas that can be accessed by anybody and everybody and it can go into the outside world. I think the management works taking officer also what I would like to see is nowadays there is a lot of stuff which is also being put by let us say the auditors and the government agencies etc. Who got the bill sets when you talk about these to be done the way the data needs to be protected. There is already a huge discussion and debate about the DDCR in Europe. So all this has also allowed think in the mind of the money. So I think there are sockets would have been extremely difficult to navigate  maybe let us say a decade earlier, it may not be such a big task to convince the self management what it needs is information. What it needs is the analysis of all the data that is getting connected where they have kept, how they are being used or not being used, blind items etc and I think that should be a good enough DA keeps on the discussions in the management and to real sensitive site when this area.

Sanjog Aul [00:32:36]:
So I have this one last question I would ask you because of course this is a topic we can talk all day trying to go into different nooks and corners but eventually if the responsibilities at the top and the leaders who may end up listening maybe CIOs or other business leaders listening to this. What would be your advice or message to them because it is hurting them. What kind of a shift in their leadership mindset should we bring so that this dark data problem starts getting resolved and we don’t have to some extent there can always be dark data no matter what you say but at least we are not turning blind to the obvious profitability potential as well as the risks that it creates. What should be that shift in the leader’s mindset?

Prosenjit Sengupta [00:33:29]:
I think a couple of areas, a couple of points here. One is the management team success focusing on data. You know, let us say the recording and the analysis of the data because you see data is now getting selected from the. As I mentioned, whether it is products or things or files, social media, multiple slaves, it’s all getting connected. I think the management start let us say changing the mindsets to start doing the number one is to start doing the reviews on systems that is very important.  It brings about two or three facts. One is that the management does not believe what people are saying in a way but they say that I am going to believe zip suffix only when I see that they have formed the system not something that has been created by a person manually. So once people that is not so fat that data is getting analyzed automatically using machines and systems and CISO are going to and managers are going to use that data without human or manual intervention. I think is your first step to ensure that redundant document and data is not getting created.

Prosenjit Sengupta [00:34:46]:
That is number one. Number two if I think people should now start the management start getting you soak the fast is getting phases manually. One let us say once we automate the system once we automate the processes once we try to robotize all the manual and mundane stuff. So we already have quite a lot of technologies available nowadays for example the robotic process automation systems et cetera that can very easily turn their day to day manual work into automations. So more and more we start using these headphones and starts relying on data and the states. I think slowly the entire mindset organization will change. Not only the management but even the organization. The people who are under the day to day fact will change deeply over a period of time

Prosenjit Sengupta [00:35:41]:
and this is a challenging assignment. I fully understand and appreciate also I know that the organization have got sales in the end of office so not the for freedom slight barriers and discipline mainly to what is light what is your they will have to be spoke about the right and wrong behavior and they will have to be sold that what are the pitfalls and the outcomes of long behavior. I think that is how slowly this entire mindset or the culture of the company is going to change to become more and more data analysis sensitive rather than depending on different creating data for years of need. So I think that is these are the ways that you just purely over a period of time the dark data is going to go out of the system.

Sanjog Aul [00:36:31]:
On behalf of the show and our listeners, thanks so much Prosenjit for sharing your thoughts and insights about how organizations can get the most out of what’s sitting in the crevices which is the dark data, we’re talking and handle the related perils. Thanks so much.

Prosenjit Sengupta [00:36:48]:
Thank you very much and it has been a pleasure for me also to be on the show.

Sanjog Aul [00:36:52]:
Thanks again Prosenjit and I hope listeners you got some nuggets out of this please like us on Facebook, search for CTN and be sure to follow us on Twitter and LinkedIn and please download this podcast from over 23 channels that we are on and please rate us so that more people learn about us and benefit. Thank you again for listening to this segment on CTN. This is Sanjog Aul, your talk show host. Till next week, take care and God bless.