On the advisibility of estimating ROI

Much has been written about analyzing return on investment (ROI) in order for user experience to have influence in a corporate context. And as I detailed in "Calculating return on investment," some have argued that such analyses are and have been essential, while others have argued that dependence on ROI calculations can be and has been excessive and detrimental.

One of the examples I referenced in that blog entry was the major role estimating ROI had played at eBay for several years. Their process for creating the business case for user experience projects was described by Jeff Herman in a paper presented at CHI 2004 and again by Christian Rohrer in a presentation made in January 2007.

However, things have been changing at eBay.

You might have seen some hint of this in how President and CEO Meg Whitman recently described user experience as one of eBay's main strategic priorities.

What has been happening behind the scenes?

One of the changes made has been to how they address ROI.

eBay's Justin Miller talked about this during the session I led at CHI 2007 entitled, "Moving UX into a Position of Corporate Influence: Whose Advice Really Works?":

"At eBay, as Richard mentioned, we have changed significantly, but (not only regarding) user experience. As a company, we had much more demand than we had supply in terms of executing on projects, so we developed ROI calculations for every project that we wanted to do, whether it was user experience related or some other thing. We discussed not doing it for the user experience pieces, but we would have been the only ones not doing it and trying to make the case that we shouldn't need to do it. The fact is, the reason we are at the company -- the reason we've been hired and have grown our group significantly is to drive ROI -- to drive revenue, to drive value. So, we should be able to show that. We've had a lot of success, and we've presented and talked about it. I don't hold it against my company for saying you guys need to do it, because we were doing that for everyone.

Recently, however, we looked at those calculations -- we looked at what everyone had been presenting over the past years. After someone would present their ROI estimates, they would come back a year later and say "here is how we did..." We looked at the results and found that at least 90% of us came back and said, "we did great." But when we looked at the return that we should have gotten if every one of those projects actually delivered what they predicted, we learned that we would have 10 times the revenue of what we have today. We realized that looking at ROI on a project by project basis was not the right approach, whether it was the user experience or otherwise. We needed to be looking at the user experience and other things at a higher level.

So, now we are focused on the initiatives. We are not focused on the individual projects. What are we trying to go after? We are, for example, trying to increase conversion rate, so when a buyer looks at a listing when they come to a site, what percentage actually bid on an item? That is the kind of thing we are looking at -- at whether we able to move those metrics, not at whether a particular project moved the needle by some percentage. And that has had a huge impact and changed the morale of employees, focusing less on the details and the tactics, and focusing more on the big picture, because you can really understand that if you can generate a change at the high level, that has a big impact."

Hence, is estimating ROI advisable? Again, many respond with a blanket, "yes."

However...