Try talking to an IT Leader about how to measure value of IT for an organization. Usually, the answer is long winded and full of abstraction. Now is the time to do the real math. What measures can be used to quantify the value of IT in form of hard numbers and dollars?

Contributor

• • Douglas Hubbard, Inventor, Applied Information Economics (AIE)

Transcript

Sanjog Aul [00:00:00]:
Good morning folks and welcome to the show.

Sanjog Aul [00:00:01]:
It needs careful planning and flawless execution to make technology work for us but how many times do we succeed? On this show we invite business leaders and subject matter experts with extensive experience in technology management. The intent here is to learn from their experience and discuss better ways to manage technology. To learn more about the show, please visit talkshow.avvald.com that is talkshow A-V-V-V-L.com Today’s topic is The True Measure Of IT Value and our guest for today’s show is Douglas Hubbard. Doug is the inventor of applied information economics that is also known as AIE. His methodology has earned him critical praise from the Darknet Group, GIGA Information Group and Forrester Research. Doug is an internationally recognized expert in the field of IT value and is a popular speaker at numerous conferences. His published articles are in Information Week, PIO Enterprise and the DBMS magazines.

Sanjog Aul [00:00:58]:
He was formerly with Coperson Library and has over 18 years experience in IT management, including 10 years experience specifically leading organizations to use this AIE method. His other professional experience includes managing large IT and software projects in insurance manufacturing, nuclear power, banking and pharmaceuticals. Good morning Doug. Welcome to the show.

Douglas Hubbard [00:01:21]:
Morning Sanjog, thanks for having me.

Sanjog Aul [00:01:23]:
Great to have you. Now, how is life treating you? I am sure you must be very busy.

Douglas Hubbard [00:01:29]:
I am pretty busy. I definitely see more business coming out of the federal side and so I’ve grown in both the commercial areas and as well as the federal areas but the federal government has really put a lot of emphasis on measuring IT values. So yes, we’re very busy.

Sanjog Aul [00:01:46]:
There has been a lot of dollars spent in the past where there was no accountability in a way and now everybody’s looking at it. Now that economy took a downturn and I guess commercially and government sector are both getting. We’re kind of waking up.

Douglas Hubbard [00:01:59]:
I’ve noted that actually I suppose my business is slightly countercyclical that our business, if anything make a increase as the belts get tighter, want to analyze their budgets a little bit more closely.

Sanjog Aul [00:02:13]:
That’s true, so when we talk about IT value in general, people literally start looking from 30,000ft view and they don’t have mostly a clue about what exactly are we talking about and then when they say you know what, I cannot measure it because we are mixing the things which are obviously not quantifiable, the things we thought which there was a workaround you can quantify and then there are certain things which are very directly quantifiable. When they do a mishmatch then of course they’ll get all confused. So if you were to kind of inventory the different areas and kind of categorize them in areas where either it is directly quantifiable versus it can be done with a workaround and with some specific structure, and there are things which cannot be quantified. So please share your thoughts.

Douglas Hubbard [00:02:55]:
I think the short answer to that is everything is quantifiable. Here’s some recent examples that I suppose some people would dismiss as intangibles or unquantifiable. We had to measure the economic impact of improved public health from safer drinking water, from an EPA information. We had to measure the monetary value of better fuel core casting for the battlefield for the Marine Corps and these are things that I suppose a lot of people might have dismissed as intangibles but usually that just means they may not be familiar with the variety of economic modeling tools out there to measure things exactly like that. So probably one thing that’s worth mentioning is we think of measurement in the very scientific, empirical, scientific way

Douglas Hubbard [00:03:44]:
and that is a series of observations that reduce uncertainty about a quantity. So all we do is we make observations and we reduce our uncertainty from our previous state of uncertainty and that’s all we really mean by measurements, so once you start looking at measurements that way, when everything’s measurable, we can always make observations that reduced our uncertainty about something, regardless of what our current level of uncertainty is.

Sanjog Aul [00:04:12]:
So when you said that everything is measurable, the way we could potentially challenge is that what is the accuracy of that measurement that you could achieve? Especially when we’re talking about uncertainty, when we’re talking about things which are like moving targets.

Douglas Hubbard [00:04:26]:
Actually, everything is measurable is actually the title of the first article I ever wrote for CIO and I described there’s only three reasons why anybody ever thought something was immeasurable and why they’re all illusion. We think of accuracy, accuracy itself is a pretty arbitrary term. When we take measurement from an economist or statistics point of view, what we think is we reduce our uncertainty until further uncertainty reduction is no longer economically justified. Now, when we reach that point, we still have uncertainty. It’s just not cost effective to attempt to reduce it anymore, so whether or not you call that point the point of accuracy or not, is arbitrary.

Douglas Hubbard [00:05:09]:
We simply ask, will additional observations cost less than the value of the uncertainty reduction we get from this? That’s the important thing.

Sanjog Aul [00:05:19]:
So basically, in simpler terms, you are basically saying that there has to be a breakeven point where the cost of getting the level of accuracy in measuring IT value is to be recognized before you would go way too much into it.

Douglas Hubbard [00:05:32]:
Exactly, I mean, the big difference here, and I think that a lot of IT departments could really adopt methods where they do probabilistic modeling. That’s the big difference. Most IT departments, if they’re doing any kind of cost benefit analysis at all, are doing really a traditional accounting kind of business case. You have a series of exact numbers. None of them, exactly, you think about things like productivity improvements and revenue enhancements, even the cost of this system or the duration of the implementation.

Douglas Hubbard [00:06:03]:
When I ask my clients if they know any of those exactly. The answer is no, they know none of them exactly. That’s why we do probabilistic modeling, so to us, all of those are ranges and probabilities. That represents our uncertainty about those things, and that’s a more realistic representation of the value of IT in general and in fact, the only way to measure risk, you have to start thinking about uncertainty like an actuary

Douglas Hubbard [00:06:27]:
and once you start doing that, you have all the tools you need to start measuring the big problem in IT, which is the risk of IT.

Sanjog Aul [00:06:35]:
So can you safely say that you can measure that what level of uncertainty there is at least, or you cannot even measure that because then it becomes very fuzzy that you think that, okay, we can reduce the amount of uncertainty in a given IT value measurement factor shy but if we cannot even say that this is the. You get the accuracy of the uncertainty that you’re measuring. I’m trying to reach it like a tongue twister here

Sanjog Aul [00:07:02]:
but the whole point I’m trying to make it is that at some point of time we have to put a stick in the ground. We have to isolate few things and say, this is it, and these are the baseline based on which you’re working but if you have a very fuzzy foundation, then everything else would be fuzzy for everybody there.

Douglas Hubbard [00:07:16]:
Exactly, here’s the approach we take and these are all methods and tools that are well developed and mature in other industries. In fact, almost all the methods that we use were born outside of IT and even before IT came along and that’s where we find most of our answers is we look to actuarial science and econometrics and managerial economics in general, and management science, et cetera but here’s what we do when we try to model those things. Number one, we allibrate people, we teach them how to assess their own uncertainty quantitatively.

Douglas Hubbard [00:07:49]:
It turns out this is a skill that can be taught. Now we go out and we ask a bunch of IT professionals, they’re confident about different forecasts that they make and let’s say they’re 80% confident some forecast and 90% confident in other forecasts. When you go back historically and look at their track record, it turns out whenever they say they’re 90% confident, they’re really only right about 60% of the time and when they say they’re 100% confident, they’re right about 67% of the time. Statistically we call it extremely overconfident but that is something we can opt as through training, so we train people to take about a half a day to get them to be very good at assessing odds.

Douglas Hubbard [00:08:30]:
Now if somebody wants to do a Google search on a phrase calibration of probability, they’ll find about 30 years of research in this field. They started decision psychologists started noticing years ago that some professionals were better at assessing odds than other professions. Now why would that be if there wasn’t a skill involved? Guess which professions do best things like bookies, if you call that a profession but yeah, bookies actually are quite good at assessing odds and I suppose more disturbingly, doctors are the worst but IT professionals rank right up there with Harvard MBAs as being statistically overconfident. You’re about as overconfident at Harvard MBAs, which is pretty overconfident, not as bad as doctors though,

Douglas Hubbard [00:09:16]:
but fortunately that problem can be solved with training. People can be trained to accept odds on things. So if you can build a model based on calibrated estimate, what you have is a model of your current state of knowledge about that investment. So any IT investment, some big new things, some three letter acronym that is the next big thing around the corner, you still have some state of uncertainty about it. Whenever I ask people for their ranges for uncertain quantities like the productivity improvements or the duration of implementation, the range is never plus or minus infinity. They always have some logical balance on the range. They know it can’t exceed or be less than the sum quantity and they put those ranges in there

Douglas Hubbard [00:10:03]:
and then we ask the question, what’s the value of a digital measurement and what we’ll find out is when we actually compute the value of a digital measurement, that we end up measuring completely different things. I wrote another article in CIO magazine called the IT Measurement Inversion. So the formula for the value of information is 50 years old, comes from game theory, and it’s widely used in a variety of industries and almost never in IT periodically. Ironically, information technology professionals are generally unaware of the fact that they’re even our equation for modeling the value of information and we use them on every project that we’re on, so that’s probably one good lesson to learn is once we can start modeling IT investments, just like other risky investments are modeled in other industries, insurance is better off. Then we have a wide variety of very powerful tools for assessing these uncertainties and that’s probably the biggest lesson someone can take away from that.

Sanjog Aul [00:11:05]:
Let’s take a quick break listeners. We’ll be right back after these messages and let’s talk about the fact that most of the IT value measurement is rather reactive versus proactive. Well, of course we talk about probability and et cetera, but the fact is that IT leaders are challenged with identifying what is going to be the actual value of an investment and definitely as business needs change in this batch based environment, the variables changed and their original estimates or the original evaluation or valuation of IT investment would change drastically and when they come back, the benchmarking still remains the same in front of the management and they come back and say whatever IT value that you said you’re going to deliver that is not there and they could get fired or scorched, their hand gets slapped. So how do you deal with that situation? How could you be more proactive, involved, but stay tuned and we’ll be right back after these messages and discuss this topic.

Sanjog Aul [00:13:32]:
Co. What stock Should I buy?

Sanjog Aul [00:15:09]:
Welcome back to the show folks. For listeners who just tuned in, today’s topic is a True Mmeasure Of IT Value and our guest for today’s show is Douglas Hubbard. Doug is the inventor of applied information economics that is also known as AIE and its methodology has earned him critical praise from the Gartner Group, Giga Information Group and Forrester Research. Before the break we touched a point where the IT value the whole conversation of IT value is mostly very reactive. Then people go into a post mortem and find out what exactly happened but at the same time IT leaders are also challenged that when they bring a proposal to the table, they have to already justify the ROI and tell exactly what of value will be created. Now they the management takes a benchmark on whatever is provided to them but as the business needs change the IT value based on the factors based on which the IT value was initially even estimated, that goes out of the window because of the fact that it’s a moving target and when the time comes for a revaluation by management about what IT delivered, since there would definitely be a deviation since their benchmark had not changed.

Sanjog Aul [00:16:15]:
So, you know, people get penalized, IT leaders get penalized for that. So how can it leaders be helped by making this process a little more manageable, a little more predictable.

Douglas Hubbard [00:16:27]:
I think that the solution is already premature and pretty well proven methodologically in other areas of business outside of IT and government outside of IT. If you look at insurance companies or exploratory oil drilling or manufacturing processes, or even nuclear powered, you’ll find that what they do when they’re forecasting uncertain investments, where there’s they have to look out several years and try to figure out what would happen, is they build something called a money problem simulation. This is a probabilistic model. They put ranges on the variables that they don’t know for certain, which as we stated is just about all of them and they wanted simulations. They generate thousands of scenarios and it gives you the probability of different outcome. So we know when we start out that nothing is going to be exactly equal to our stated mean. The stated averages in each one of these variables we’ve already modeled in the fact that there is uncertainty and there’s going to be a variance.

Douglas Hubbard [00:17:27]:
So the big thing here for people is probably to look at IT, as a risk return analysis, because that’s how everybody else looks at it. When you look at comparative uses of capital, you’re looking at a risk return analysis. In 1990, something called modern portfolio theory won the Nobel Prize in economics and it really is the basis for all modern portfolio optimization method but in order to use that, you have to be able to state not just the return on an investment, but the variance, the expected variance on that return and if we could translate the IT investments into that kind of language, we get all these very powerful tools to use.

Douglas Hubbard [00:18:07]:
So we don’t have to say here’s my forecast and I expect that forecast to be exactly right five years from now. All we have to do is to say here’s my ranges, my 90% confidence interval and 90% of my forecast have fallen within my 90% confidence interval. That’s a relevant measure of how good we are at forecasting, for example, meteorologists are actually pretty good, when you look at all the times they said there was a 95% chance of precipitation, they were light 95% of the time. When they say there’s an 80% chance of sun shine, they’re light 80% of the time. So the trick is not to be right exactly on the dot every time, but to be realistic about how much uncertainty you have.

Douglas Hubbard [00:18:51]:
and when you look at your track record? Does your track record match what your stated uncertainty was? Are you right 90% of the time when you say you’re 90% confident, et cetera.

Sanjog Aul [00:19:02]:
Given that the IT leadership is of course not total novices, they have to definitely go through these exercise multiple times but they must have come back and devised a mechanism where the chief can. They can set expectations that things would change but as a management, a CEO or a CFO would expect that they are used to the other Monte Carlo or other kind of methods. They would expect the IT leadership to be adopting the same thing. However, in reality it seldom happens and is there a specific reason why it’s not done, because it’s in the whole framework is not fine tuned enough for those people to feel confident about using that framework?

Douglas Hubbard [00:19:39]:
I don’t. I think the that framework and methodology is so widely used in so many other industries and government problems that would be hard to say that it’s not mature in that respect. It may be as simple as it’s just a cultural issue when that will evolve over time. When insurance started out, when people first started selling insurance policies, there was no such thing as actuaries. Actuaries evolved out of necessity later on and maybe we’re finally getting to that point with IT, IT is probably the riskiest investment most organizations will make that receives no quantitative risk analysis. We’re evolving to a point where we probably can’t really put up with that too much long we’re going to have to do quantitative risk analysis just like organizations already do on other capital investments and IT just needs to adopt it. So it’s just a matter of the industry being a relatively new industry catching up in that respect.

Sanjog Aul [00:20:38]:
You mentioned about risk based analysis, basically you quantify risk and then it’s corresponding with benefits is kind of measured and seen and reworked the net result. On the other hand, there’s also one more thing which needs to be looked at is the opportunity cost. Especially when we are talking about an area where where you want multiple initiatives in place and not every initiative is approved. So then one of the factors should be that what’s the opportunity cost that you’re carrying with each initiative individually?

Douglas Hubbard [00:21:04]:
Yes, absolutely, ultimately your opportunity costs are the fully loaded cost of your capital and labor and other resources you’re allocating to us. In fact, opportunity costs are naturally dealt with in the modern portfolio theory methods that I already talked about. It simply asks the question if you’ve got a set of resources and a number of investments that you could make, what’s the optimal combination of those? If you choose the optimal combination, then you’re already by definition minimizing opportunity cost for these sorts of things. You’re allocating resources in the best possible way, but that means you have to optimize the whole investment portfolio.

Sanjog Aul [00:21:47]:
One of a very obvious example and which is actually getting a lot of exposure is the area of outsourcing where the question about what is a good balance between their on site resources or in house resources works as outsourced resources. is a constant challenge that every company which is trying to get into it or is already into it is constantly battling with but there doesn’t seem to be any specific answer that they are able to come up with or they don’t have already a port that they could look at to say, okay, this is a good break even, or you can take good balance approaches.

Douglas Hubbard [00:22:19]:
I would say that coming up with one ratio is saying every company should have a 200,000 square foot office.

Sanjog Aul [00:22:26]:
It’s not that.

Douglas Hubbard [00:22:27]:
I think there are obviously different answers for different companies but if we can start to quantify the risk of outsourcing as well as the benefit of outsourcing and in sourcing, including all those things we typically leave off the table because someone doesn’t know how to measure it. What we should do is figure out how to measure those things and include them in our analysis. Otherwise we’re always going to, we’re going to continue to make the wrong answers, make the wrong decisions on each of these questions and these are big important dilemmas for most organizations. The outsourcing dilemmas, if I look at an insurance company, for example, I use them a lot as examples because everyone knows what actuaries do.

Douglas Hubbard [00:23:07]:
They’ll look at an insurance company. They’re probably not using the same sophisticated analysis method on the outsourcing question as they do when they analyze new insurance products coming out and the fact is, there’s no reason they shouldn’t. They’re very similar issues. In fact, you’re trying to optimize some allocation of resources through different investment and some have higher risk, some have but higher returns. So that’s why you outsource. It may be slightly higher risk with outsourcing or a lot higher risk, but also with a lower cost, you get a higher return for those sorts of efforts.

Douglas Hubbard [00:23:42]:
So it’s a trade off and again, that’s what more modern portfolio theory is all about, is trading off risk in return for a variety of investments and the outsourcing example is just one good example of how that could be applied.

Sanjog Aul [00:23:56]:
So now when we talk about research and there are different methodologies, you have one and there are other established methodologies out there. What is being done in terms of the research conducted at how these methodologies are working out and also when we go and do some primary and secondary research, since IT measures of IT value is a chronic topic lately and of course always what have been some of the common areas which stand out where what are the specific problems that you look at, what are the typical solutions that you use, what works, what doesn’t? That’s something that we should look at.

Douglas Hubbard [00:24:34]:
Absolutely, in fact, I would say here is probably the most useful tidbit I can give to people on the show right now, which is most people, if they go out and develop their own portfolio prioritization method, I would say 9 out of 10 people will generate some sort of subjective weighted scoring method. You’re familiar with this, people coming up with different sorts of subjective weighted scores, this is actually pretty thoroughly researched methodology within decision psychology and it turns out that the subjective weighted scoring method that most organizations develop for IT prioritization, probably don’t improve decisions at all. In fact, they often don’t even change the decision, much less improve them. In fact, we find out that the decisions might actually be worse except for the fact that people routinely override the output anyway. So 70% of the time, approximately the weighted score agrees with what you would have done anyway

Douglas Hubbard [00:25:34]:
and then 30% of the time when it doesn’t agree, override it.

Sanjog Aul [00:25:38]:
Let’s take a quick break, listeners. We’ll be right back after these messages. This is a very interesting topic and I’m pretty sure listeners will be very intently listening to all the numbers but let’s take a few minutes to listen to these messages. We’ll come back, so please stay tuned.

Sanjog Aul [00:29:10]:
Welcome back to the show and for listeners who just tuned in, today’s topic is A True Measure Of IT Value and our guest affiliate show is Douglas Hubbard. Doug is the inventor of applied information economics, also known as AIE. His methodology has earned him critical praise from the Gartner Group, Giga Information Group and the Forrester Research. Before the break we were discussing about the areas of research or in fact a lot of research which has been conducted in the area of measuring IT value and different methodologies have been tested out and proven and basically falling flat on their face and different organizations have been checked out within that respect. So what are the some of the lessons learned and or what are some of the conclusions that have been drawn from the studies and Doug, you were basically sharing your thoughts regarding that. So let’s pick up the press and move.

Douglas Hubbard [00:29:57]:
What I was just noticing is that when I feel that they have to reinvent the wheel themselves. Typically they’ll come up with something like a simple weighted scoring method and as I mentioned before the break, that’s probably the one method that’s been most widely measured to be pretty ineffectual. The most important thing for it to remember here is that many of the problems that they’re dealing with are already solved in other areas of business and government. We don’t have to reinvent how to measure risk. That’s a mature industry. We can borrow very heavily from that

Douglas Hubbard [00:30:27]:
and we don’t have to reinvent how to optimize a portfolio of investment. Again, these are Nobel prize winning methodologies that have been around for a while. Even though modern portfolio theory won a Nobel Prize in 1990, it was developed back in the 50s. So these are things that have been developed for decades and sometimes longer. So we don’t even have to reinvent the way for measuring the value of information because that formula has been around since the 50s in game theory. So we really have a lot of very powerful tools if we can start thinking about IT as a risk return decision and look outside of IT for the method.

Sanjog Aul [00:31:06]:
So when we talk about IT leaders, they actually go out and try to use different methodologies. Some either a consultant brings in or they come to know about that in a seminar and of course either it could be hassle and or maybe it’s a methodical approach brought in by a third party. Either way they see the results are not as they expect and this kind of frustrating experience for them. So is the method, of course we say that methodologies are pretty mature and they have been tried and tested. So does the IT world suffer from lack of knowledge of how to implement it properly?

Douglas Hubbard [00:31:41]:
No, I think it’s lack of knowledge that there is other tools in other areas of the business. I once ran into a director of IT in an insurance company who said, Doug, the problem with IT is it’s risky and there’s no way to measure risk and I said, what are you talking about? You work for an insurance company, of course there’s a way to measure risk. This is sort of even in insurance companies, IT is usually kind of oblivious to the quantitative methods for measuring the value of risky, uncertain investment. That’s really what the problem is with IT. That’s a pretty risky investment. There’s a lot of uncertainty to that and until we can actually start to borrow heavily on all of the quantitative methods for evaluating uncertainty, it’s probably going to continue to have problems I would say the best way for evaluating a methodology is to ask a first question.

Douglas Hubbard [00:32:29]:
What’s the track record that actually improves decision beyond my intuition, because that’s what you’re always comparing a methodology to. Is it better than your expert intuition, already as an executive. Executives have some pretty good rules and their intuition is usually better than just random. So their expertise, their experience does add value to the decision making process. So if you’re looking at the methodology, you got to do better than that already. There have to be shown to be an improvement on that. Now when we look outside of IT, there’s really quite a lot of methodologies that are shown to be improvements on intuition. Unfortunately, in IT we tend to keep reinventing the wheel and we more often than not come up with the methodology that we already know. Don’t improve on intuition like the subjective weighted scoring method, et cetera.

Sanjog Aul [00:33:25]:
During the prior show I had a discussion we had few very experienced folks from IT and we would talk about IT portfolio management and all product project management and things like that and my comment was that we are not yet as mature in the way we budget the project and when we come back from it, it’s usually an overage in terms of time or money and our quality suffers and they kind of big time put back on the fact saying that no, it is almost a science now in IT for us to be able to budget pretty accurately on when it will be delivered and what is the cost going to be. Now all that was more on the tactical side and or you can say it’s on a tangent when compared to when you’re trying to measure IT value. Those people as IT leaders are claiming that yes, your budgeting and your risk management and your project management is all in place, then what stops them from being able to for them to be able to measure IT value as accurately as we would all like to be.

Douglas Hubbard [00:34:21]:
Again, remember when we talk about accuracy, we mean the economically optimal amount of uncertainty. There’s still quite a lot of uncertainty left when we’re done measuring, when we’re past the point of adding any value with additional measurements. So the accuracy is whatever we’re left with. When we’re done with all the economically justified measurements, that’s what we’re left with. I would say that, I’m sorry to your initial question.

Sanjog Aul [00:34:47]:
Basically what I was saying is that, on one end IT leadership says that they have made up science of the whole IT estimation process but then when it comes down to actually talking about hard dollars of value delivery, they are more instance based versus the bigger picture.

Douglas Hubbard [00:35:02]:
On the first part of that, your comment there is that even for those who feel that the cost estimating side is down to a science, that is that the biggest explanatory factor in the variance of IT project costs from the original estimate are external events that IT doesn’t have any control over. So unless you’re actually modeling the probability of those events coming in, you’re missing a big source of variance in your cost. I think it’s a little bit naive to say that we estimate all of our projects within plus or minus 10% when you know that the thing that actually interrupts IT projects are things outside of your control. More often than not, it could be a change in CEOs, a change in CIOs, a merger, a spin off, your company could go bankrupt, et cetera. All there’s unfortunate things that do happen and they happen with a higher frequency than most people would probably want to admit in business cases. In fact, there is a probability that when you start an IT project, it will never finish and it will in fact be canceled and there’s a way to work out the probability on this.

Douglas Hubbard [00:36:08]:
If you have a software development project, for example, that’s longer than two years in duration, the cancellation of that project exceeds the default rate of the worst rated junk bond. Yet you almost never see anti cancellation on a business case. In fact, that’s a real factor that you have to take into account for long duration projects is the fact that it won’t finish at all. So I think it’s underestimating risk and uncertainty could say that there’s a high degree of accuracy on estimating project costs when you have uncertainties even about whether or not you can finish. Often these are things that are outside of IT control.

Douglas Hubbard [00:36:46]:
On the benefit side, again, I think the answer is doing probabilistic modeling, acknowledging your uncertainty and saying here’s how much my uncertainty is going out and making measurements only those that are economically justified, reducing the uncertainty a little bit and ultimately having to make the risk return decision. Is this a good investment or not , on a risk return basis Risk is actually a bigger source of variance on the value of IT than the return is. We’re leaving out half the equation or more when we don’t talk about risk like an actuary does.

Sanjog Aul [00:37:20]:
So at one end, while we answer the question about it’s not about actually getting an accurate measure all the time, it’s reducing the degree of uncertainty in any of the measurements that you would like to get on IT value.

Douglas Hubbard [00:37:35]:
The economically justifiable amount of uncertainty reduction. That’s what you’re going for.

Sanjog Aul [00:37:39]:
Okay. That’s a very scientifically and very condensed form of, you know, thought that you brought out and which is very valuable to the IT leaders. Now when we come back to the fact that everything is, well, a science, but then they also are supposed to be artists now, that is they have to do more with less and in fact there is a new paradigm which CEOs and CIO both are trying to for, is that every year I will reduce your budget growth in terms of percentages, but I want you to increase the value that your increase in budget creates incremental value. So we are saying more with less and then when you do that, I’m pretty sure IT leader will not try to cut corners to achieve that and they will try to come up with some creative means. Now when they do that, those creative means are beyond comprehension till the time they all got adopted given the situation

Sanjog Aul [00:38:33]:
and then again when we start talking about measuring an IT value on a given project or initiative and looking at the bigger picture of overall IT portfolio. This art, if you will, can disrupt a very stability of a framework that has been utilized to measure IT value. So how do you deal with that because there’s a mandate from the management, at the same time they also want to get a reduced degree of uncertainty, as you said. How do you do that at the same time eating the cake and having it through?

Douglas Hubbard [00:39:02]:
I would suppose it’s a bigger obstacle to that level of creativity is the traditional accounting version of doing your business thesis where you have to come up with a lot of exact numbers and you don’t really quantify your uncertainty about each of those numbers, you just state the number and that kind of forces a framework. I don’t know if that necessarily is directly related to creativity or not, but I would say that would probably style risk taking a little bit more. If you can quantify your risk and include that in your portfolio optimization, you can make a lot called trade offs in risk. You don’t always have to avoid it. There’s times when you can accept it and you’ll be rational about it by trading off risk to one plate versus another in your portfolio but you can’t do that until you start measuring it. It’s interesting. I think most of the ideas for new discretionary IT investment come from the RIG,

Douglas Hubbard [00:39:57]:
that’s the random idea generator. In most organizations, people go to conferences or read an article and they get an idea and they batted around and pretty soon somebody gives it a name and it turns into an acronym and then it gets its sponsor and now it’s got some momentum and each one of those times, I think it’s a pretty fascinating life course that a lot of these ideas take. If you take them all the way back to their early dissemination, what happened to that idea? How did it get started? Chances are it is probably not the result of some really creative sitting back and thinking about the big problem kind of a process. It’s that discretionary investment idea is probably itself a reaction to immediate opportunities and needs versus a big planning sort of an issue but fortunately there are even quantitative methods that can back up the planning. We can sit back and do anticipatory development of business to really look out over the horizon on the new things around the corner.

Sanjog Aul [00:40:55]:
Let’s take a quick break listeners. We will be just be right back after these messages and going beyond the fact that we have certain things which are not known and they’re uncertain and we are trying to reduce the level of uncertainty in terms of measuring IT value, we have a tendency usually to measure the direct cost, the direct benefit, the direct risk, then coming to the fact that life is not as simple when it turns out to be when we talk about IT because there could be always a lot of indirect factors and those indirect factors are not as easily measurable because we don’t control them. So how do we account for them when we are actually trying to quantify the actual net value delivered from IT? So let’s discuss about this. Let’s explore this a little bit more when we come back from the break, so please stay tuned.

Sanjog Aul [00:44:23]:
Welcome back to the show, for listeners who have just tuned in, today’s topic is The TrueMeasure Of IT Value and our guest for today’s show is Douglas Hubbard. Doug is the inventor of applied information economics, which is also known as AIE. His methodology has earned him critical praise from the Gartner Group, Giga Information Group, and Forrester research. Before the break we touched a part where while we are trying to always reduce the uncertainty for any of the measurements that we want to have for IT value, at the same time, when we actually start measuring the real nuts and bolts of IT, we usually try to use the direct cost, the direct measures, the direct risk, and we should definitely quantify or try to at least qualify the indirect the ripple effect that could be there and what other peripheral entities which could be potentially impacting the cost and the risk and everything related to IT Now the challenge here would be that first of all is the methodology or whatever different framework that exists out there allows you to account for that and if they do, then how do they actually help you measure in reality when you are not in control or you are not even given the due insight to what those risks, cost and benefits are.

Douglas Hubbard [00:45:38]:
T he direct versus indirect questions reminds me a little bit of the hard versus soft kind of distinctions and a lot of people spend a lot of time making that fine distinction, what’s hard, what soft, what’s direct, what’s indirect, sorts of cost and benefits. Having to make that distinction of the need for it kind of goes away when you do probabilistic analysis in the first place. You quit having to say this variable is an indirect benefit and that one’s a direct benefit. You just say, what’s the benefit and what’s my uncertainty about it. Likewise, in hard and soft benefits, you don’t say that one soft, that one’s hard. You just put your range on it. Some ranges will be wider than others and some will be very narrow. If that’s the distinction between hard and soft, that’s fine, but we don’t really need to specify that distinction anymore because the important thing is the rain.

Douglas Hubbard [00:46:25]:
How much do we not know about it and there’s a lot of things, for example, if we make some investments in IT, say in the public sector, a lot of things will have to do with the behaviors of the public. Now you don’t have control over that but those uncertainties do affect your uncertainty about the value of the IT project. Maybe within the commercial side, you’re implementing a system that’s going to reduce your cost of sale. That’s a good idea but you have uncertainty about your future sales. No one’s going to say, I know exactly what our sales is going to be in the future and if you don’t know that in the future, that adds uncertainty to the value of this IT projects

Douglas Hubbard [00:47:04]:
and that’s just realistic. A lot of people might say that maybe a reduction in cost of sales is a hard savings and so they’ll put an exact dollar amount on it but the fact is they don’t know what their sales is going to be three, four, five years from now. So those, even though they’re uncertain, if we can quantify our uncertainty about each of these things, we end up not having to make this distinction between direct, indirect, hard, soft, or factors we can control or not. They’re just included in our models.

Sanjog Aul [00:47:31]:
Doesn’t your statement, in a way is using a big assumption that people who are actually making decisions or trying to, making an effort to quantify, they have the ability to be able to quantify the uncertainty.

Douglas Hubbard [00:47:46]:
Actually what we find out is that people can be trained to quantify uncertainty. Some of your listeners might be familiar with Gartner Group, and in the past Giga Group, Giga was bought by Forrester, but they used to put probabilities on the end of the statement. They would say Microsoft is going to release such and such by the end of 2005, and they’ll say 0.8 p and that represents their confidence that this event will actually occur. Actually, I think it’s a pretty ingenious journalistic tool. I would like to see it in my daily newspaper as well but when you go back and look, are they right as often as they think and I actually trained 16 giga analysts how to do that

Douglas Hubbard [00:48:25]:
and when we tracked 20 IT predictions for the 16 analysts over time, we found out that after the trainings, whenever they said they were 80% confident, they were right 80% of the time and when they said they were 90% confident, they were right 90% of those time. So we know that people can be trained to assess uncertainty and then later on, when you decide that uncertainty is worth spending money on to reduce in some particular area, there’s all sorts of powerful tools to reduce our uncertainty there. That’s when we get into empirical methods. It could be controlled experiments, random sampling methods, et cetera but we only do it in those areas that are applied where it’s necessary. In that article, the IT measurement Immersion that I wrote a few years ago for CIO Magazine, I noted that if you compute the value of measurements, you end up measuring completely different things than you otherwise would have.

Douglas Hubbard [00:49:17]:
That people spend most of their time historically measuring things that they’re already more certain about and less time measuring things that they actually have more uncertainty in but we also find out that if you measure based on information value, you measure fewer things. I’ll often put together a business case with 5200 variables in it and after calibrating people and completing our initial state of uncertainty, we’ll find out that we only need to measure two or three of those 50 to 100 variables. That’s a difficult finding, so when you actually make your measurements based on information values, you end up measuring fewer things and also, so that’s another impact of how really anybody can adopt this.

Douglas Hubbard [00:50:02]:
We swim upstream from the assumption that more quantitative methods are necessarily more time consuming or difficult and I think we find just the opposite. People spend quite a lot of time on some theoretically unsound methods that don’t improve their decision and the best methods are sometimes easier, so I do think that people can be trained to assess their uncertainty quantitatively and we can track how good they are at it. All you gotta do is keep track of your predictions and see if you’re right as often as you thought.

Sanjog Aul [00:50:32]:
On behalf of the show and our listeners, I’d like to thank you Doug, for sharing your insights and expertise about how an organization can can truly measure the value of IT in spite of all the uncertainty that’s out there and of course we are trying to inside bringing we are trying to bring uncertainty into the part of the equation and also there are so many obvious and not so obvious obstacles in doing so.

Douglas Hubbard [00:50:56]:
Thanks for your time, Sanjog.

Sanjog Aul [00:50:59]:
Thank you again. We all realize that measuring IT value is not as simple as using a mathematical formula, and it requires a painstaking analysis of the parameters involved and the results may be vastly different case by case. So every organization though must have a well defined framework which at least allows them to structure the activities, the cost, the benefits related to an IT investment and also for an ongoing investment called ongoing maintenance actually. So this enables automatic data collection over a period of time which is then available for analysis and also the framework should allow room for situations where IT leaders have to get creative to cut costs and reallocate costs to another initiative or even dispose retire an IT asset or an application. So there’s of course no silver bullet, but a structured approach to data collection, appropriate knowledge and experience of IT leaders in using the related framework and just sticking to the gun approach will deliver the desired accuracy in measuring IT value. Thank you again for listening to Managing Technology the right way this is Sanjog Aul your talk show host. Till next Friday, take care and god bless.

Download Podcast
Apple Podcast, Google Podcast, Spotify, Pandora, iHeartRadio, SoundCloud, TuneIn, and Stitcher. Find other syndication channels here or search CIO Talk Network podcast on any other app.

Explore More

• How to Build a Future Fluent Organization?
• Future-proofing Leadership Strategy and Culture
• Steps to Future-proofing Data and Analytics Environments
• The 3 Critical Skills for Future IT leadership: Solution Focus, People, and Narrative!
• The Future of IT Leadership
• The Future of Internet
• Future-proofing Leadership Strategy and Culture
• Identifying and Enabling the Skills of the Future
• Building Blocks for a Future-Ready Security Shop
• Syncing Organizational Change and Leadership Development