Time on Site is the Denominator

A major media company recently did a major redesign of their web site. When users complained about all the problems, and there were many, the sites (re)designers said the problems were the users' fault for not being familiar with the changes, and they pointed to an increase in the time users were spending on the site as proof that the redesign was successful.

It didn't occur to them that there was another very likely reason for the increased time on the site, namely that users were spending more time on the site because the site was harder to use and harder to navigate. In fact, parts of the redesign increased the number of steps needed to perform a number of common actions, sometimes significantly.

Underlying all this is that this major media company seems to not understand what time on site represents. It's not really their fault. Time on site is a phrase so prevalent these days that it gets 138 million hits on Google.

But here's the pivotal question: does the company get value from users spending more time on the site? In fact, they do not. Think about it this way. Which of the following two customers is more valuable to a clothing store?

• Alex goes to the store, spends an hour trying on pants to find one that fits and looks good, buys the pants and leaves.
• Pat goes to the store, tries on one pair of pants and discovers that they fit and look good, so buys them and leaves five minutes later.

The answer is obviously Pat, who left a happier customer. Because they hadn't spent an extra 55 minutes trying on pants, they are far more likely to buy other things, even before they leave, and they are more likely to tell their friends about their good experience.

If the store had measured time in store, they would have thought that Alex was a happier and better customer!

Time on site is still useful to measure — but in a different way. Here are two straightforward equations:

value received by users  time on site

value received by site (e.g., ad revenue)  time on site

Notice that time on site is the denominator in both equations. The first roughly measures how well you're satisfying your users while they're on your site, while the second roughly measures income over time.

Having visitors to your brick and mortar store costs you money, in terms of employee costs, utility costs, etc., even if the visitor buys nothing. The more visitors you have, the more employees you need, even if the visitors buy nothing. The same is true of your web site. Every minute somebody spends on your web site costs you for server time. Although the absolute cost may be smaller, it is not zero. The formulas above are obvious when you think about that plus the old standard ROI, or return on investment. If you measure and worry about about the return on investment for time on site, you'll have a much better understanding of how your site is doing.

What if you have an entertainment property? My company, Puzzazz, is a puzzle technology company that has built a popular free puzzle app for iOS. You might think we should measure time in app because more time in the app must indicate happier users. Nope. Take the people who solve just the New York Times crossword in Puzzazz. Some people can solve a Sunday NYT puzzle in a few minutes; others take an hour or more. Can we draw any conclusions about a user's happiness from the time it takes them to solve the puzzle? No. Different users are not comparable to each other for many reasons and doing things like averaging their times across different puzzles to determine value is not mathematically sound. Instead, we can look at things like puzzles per session and puzzles finished divided by puzzles started. Whatever your site or product is, you should spend the time necessary to figure out what the relevant metrics are.

It's certainly true that measuring your site's value and effectiveness is far more complicated than I lay out here. But it is even more complicated than measuring a number like time on site and expecting it to tell you much by itself. Especially when you use it as the numerator instead of the denominator.

Why You Need Anecdotal Evidence

Have you ever heard the phrase "That's just anecdotal"? Usually, it's used to discount somebody's opinion about the way something should be. In UX terms, if you've observed a user, saw them have a problem, and want to respond to what you saw, you may have someone tell you it's just anecdotal -- in other words, you don't have hard, objective, statistical evidence, so forget it.

But there's a contradiction here. There's a belief that, if you gather enough anecdotal evidence, it magically becomes hard evidence. You can even see this in the latest ad campaign for Windows. Millions of people sent in their ideas, or were surveyed or studied, and their opinions -- anecdotal evidence -- is now real evidence. How does this magic work? Well, it can't. Two people who said different but similar things, or even the same things under different conditions, can't be lumped together in some statistical box.

But that doesn't mean that anecdotal evidence is worthless. Quite the contrary. It's invaluable (and Microsoft should be commended for actually listening to users). Anecdotal evidence provides you with something that hard data can't -- feelings. In a typical usability test or, these days, a web site A/B test, success is measured by whether or not somebody succeeds in a task. How about whether they frowned or smiled, tapped their fingers on their desk impatiently, hummed to themselves, or cursed at their computer?

I recommend that you gather as much anecdotal evidence as you can and let it infect your world view. Try to feel what your users feel, think like they think, and use that to design your products. And, after that, gather hard evidence on whether you were right or wrong and move forward from there. But, if you don't start with a feeling about what's the right thing to do, no amount of hard evidence will help you.

Does T-Mobile Want To Steal My Identity?

It can be hard to tell real companies from scam artists sometimes. I got a call the other day from T-Mobile about my bill. They had overcharged me by $24 and I was late paying the bill because I wanted it fixed (and, in this economy, they call the day after it's due!). The discussion of why they would possibly think I wanted text messaging turned off on my account when I switched from a BlackBerry to a MyTouch is a topic for another post in the future

The T-Mobile agent who called me asked for part of my social security number to verify that I was who I said I was. I refused. Hey, you called me! How do I know you're not a scam artist? He told me that he was from T-Mobile and I should believe him, that, if I didn't give him my social security number, he couldn't help me. All things a scam artist would say, of course. The fact remains that I had no proof he was who he said he was.

I tried to explain to the guy that T-Mobile should never, ever ask a question like that because, to the extent that people answer it, you're training them that it's OK to give your confidential information to somebody who calls you on the phone. You're enabling scam artists. Unfortunately, he just didn't get it.

The rules are simple. In the world of client-server architecture, it's known as "never trust the client". In the real world, it's "never trust somebody who calls you."

• Never, ever give confidential information to somebody who calls you, even an innocuous thing like an account number. You don't know that they are who they say they are.
• If you call somebody, never, ever ask for confidential information when you call somebody. If you need confidential information, ask them to call you back at a number which is posted prominently on your web site or which is well known (like 1-800-T-MOBILE).

Is Comcast Helping Scammers?

Comcast wants to fight scammers, but they're inadvertently going to help them.

Comcast, like all Internet service providers, is directly impacted by so-called botnets, machines that have been hijacked by viruses and other malware to serve as robots in the service of scammers. The botnets are useful to the scammers because it allows them to send spam and launch attacks from many locations instead of a single location, which makes them much harder to catch and shut down.

Comcast's idea is to inflict popup ads on their customers that appear to be compromised. which provide them with information. According the the AP article, the ad says "Comcast has detected that there may be a virus on your computer(s). For information on how to clean your computer(s), please visit the Comcast Anti-Virus Center."

There are a couple of problems with this:

• To the extent that it works, it trains people that popup ads that claim to be helping you clean your computer are legitimate. The problem is that, with this sole exception, none of them are.
• It trains people that clicking on a link in an unexpected popup ad is an ok thing to do, when it almost never is.
• It trains people that something like this can be trusted, when it's very easy to fake it.

I don't like the popup in any event, but, if they're going to do it, I think there are a couple of things they must do:

• The popup shouldn't look at all like an ad and it certainly shouldn't mimic any OS feature.
• The popup should contain no (that's zero) links in it. Just to be clear: None. Instead, the ad should say "... please visit comcast.com in your browser and click on the xyz link ..." Train people not to click on links like that and train people that the only way to know for sure that they're actually on the comcast site is to go to comcast themselves, not to trust a link.
• The popup should not have any button in it. No close button. Nothing to click on at all. Just "Close this window after you've read it." Don't train people to click buttons in unexpected popups.

And how about thinking if there's a better way to attack the whole problem, like doing something in concert with Microsoft and Apple (OS vendors), or Microsoft, Mozilla, and Google (browser vendors).

Matching User Experiences

At Thursday's UX Office Hours , a funny thing happened -- somebody came in wanting to talk about user experience. The reason I say that's funny is that, most of the time, people come in wanting to talk about their user interfaces, not their user experience. They bring in mockups, screen snapshots, prototypes, and actual products and web sites. And they want to know what to do to make it better. Almost always, I have to pop the conversation up a level, to talk about what they want to accomplish for their users, rather than how they should move pixels around. Part of what I try to do is to educate people so that, when they walk out, they're better equipped to move forward. So, what's the difference between UI and UX?

In a nutshell, you want to give your users a good user experience. A good experience means they'll be able to accomplish what they want, they'll be happy with your product, etc. One of the ways to get a good user experience is to have a good user interface. You might think that makes no sense -- how can it be that UI is only one of the ways to provide a good UX? What other ways can there possibly be? Well, here are a few:

• Provide functionality or content that your users can't find anywhere else, that they absolutely need
• Do things automatically for your users, so they don't have to see any UI
• Build a system that is fast, that, once learned, allows your users to do things faster than they can anywhere else
• Same, but replace "fast" with "better" in some context specific to your business
• Pay your users money (yes, this is real -- look at Google AdSense, Amazon Associates, or Ebay)

And what should you do? You should work to understand your users and then do those things that will give them the experience that you think will accomplish your business objectives.

Back to the guy who came in yesterday. It was a great discussion and I hope I was able to help him better understand what he needed to do. One slightly surprising thing was that his company has two very distinct classes of users and he was trying to figure out how to craft an experience that met both of their needs. Unlike a system like Ebay, where buyers and sellers are largely similar people, or Monster, where the point of the site is for job seekers and job posters to interact with each other, his two classes of users weren't similar and weren't going to be interacting with each other. I told him that he should build two completely separate UIs for these two groups, that to try to build one interface would end up serving nobody well. After he left, I thought of some great examples:

• Google provides completely different experiences for people placing ads, for people putting ads on their sites, and, of course, for people seeing ads on the net.
• Amazon Associates provides a completely different experience for associates than they do users who see associates' links.
  

While his business is different from these, the same rules apply. And the bar for the quality of the UI is different for these different classes of users -- the interface for people creating and placing ads,or creating associate links is so much less important than what the people seeing ads or an associate link get. The first group of people are making money, so they're incented and a bad UI won't stop them from using the service. But, if the ad or link UI is wrong, nobody will make any money.

To provide the best product for your users, you want to create an experience that matches them. Sometimes, that means figuring out the classes of users you have and building different experiences for them.

Users Don't Know What They Want

Years ago, I was called in to redesign the user interface for a very large system that a very large multinational company was building for a very large government agency in a very large state. Yeah, that's a lot of very large in one sentence, but I'm trying to give you an idea of scale. The company had developers in four states working on the project. They had some of the best and brightest architects working on the project, some relocated just to be able to work on the project. And the overall manager for the project was a stellar up-and-coming executive.

Yet, on this entire project, they had not one user interface designer. Not a single person with actual user experience expertise. The closest thing they had was a tester who, to his credit, had identified a number of the flaws. But, the flaws didn't really matter much. They were picky details on an interface that was fundamentally flawed -- because they had committed the cardinal sin of exposing their schema. The system they had built had an interface which was barely better than a raw database viewer. Every complexity and oddity of the underlying schema (usually necessary for good data design reasons) were plopped down right in front of the users. The client (the state) was very unhappy and on the verge of canceling the project.

Someone on the project had worked with me in the past and he suggested that they call me in to help out. Of course, I was stunned to learn everything I described above. But the real stunner came later.

You see, in addition to the hundreds of company employees, there were also four state employees working on the project. The very first thing that I did was to interview one of those people, who had been working on the project every day for two years. And I asked her questions that nobody had ever asked her!

For two years, she had been helping define what the system did. She had been spent thousands of hours answering questions and just as much time working on data relationships. They'd taken this government employee, this subject matter expert who knew nothing about computers, and made her into a schema designer! And nobody had ever asked her about her job.

Instead, they had, in essence, asked her what she wanted. She didn't know what she wanted. But, worse, she didn't know that she didn't know what she wanted. So, she answered their questions with information about the data of her job -- the details of her job. This was certainly useful information, but they could have gotten it from anybody. But the true expertise that she and her three state co-workers had not even been tapped.

In contextual inquiry, you are supposed to observe users in their "natural environment." It's sort of like going to a zoo and watching the animals. But your observation changes things and you frequently don't learn what you want to. Beyond that, you will miss things, even if you're lucky enough that the time you observe is a representative sample. And, for a typical product, there are many different target users and observing all of them is an arduous task. I think that contextual inquiry works best when you already have a product and are trying to improve it.

Fortunately, most people know a lot about not only their own work but the work of their co-workers. In the case of the state worker I interviewed, she had a pretty good idea of what everybody did, from top to bottom. The trick was in asking her about her work and the work of the agency, not her data, and certainly not how she thought the software should function.

It's Best to Iterate

Although I recently wrote that there is no best process for designing user interfaces (or, probably, anything), there are some best practices. Here's one best practice that I've boiled down to four words:

Define. Design. Refine. Iterate.

The iteration is the part that's most often missed. Far too often, designers think that they defined the whole problem at the beginning so all they need to do after that is to design it. But thinking doesn't make it so -- it's usually the case that the original definition missed some key elements. Sometimes it's just plain wrong.

It's much, much cheaper to iterate on the definition and the design before you ship something than afterwards, when your users tell you that you built the wrong thing.