Giving a (software) product a numeric usability score is a thought I had in many of my past projects. Having a final verdict in form of a %-value or a grade would give me a certain comparability between projects and some customers would prefer this kind of "certificate" over (just) an elaborate text. While for full reviews I still prefer the qualitative, informal approach, this formal summary could be applied when a full review isn't demanded.
Userfocus.co.uk has a nice example for this; guidelines can be checked (3-tier: doesn't comply, complies and kind of complies) and the XLS will promptly output a %-score for all of the 9 topics, together with a beautiful spiderweb-chart.
I gave it a try on my companys website. My impressions were mixed, it did provide some tendencies where the main problems are, but the statistical value was very dubious in my eyes. Some of the topics had only few checkpoints relevant to the website and some of the questions were highly subjective to anser. E.g. the topic "Forms & Data Entry" has a full 100% but mainly because only 6 of 23 answers were given.
What do you think? Does this quantitative approach has any value for evaluation? Could it be done right to get a meaningful and statistically valid outcome?
A single evaluation cannot be statistically significant. If you need statistical significance, then you shouldn't be doing a heuristic evaluation or expert review.
That doesn't mean that giving a numeric answer doesn't have value for a heuristic evaluation or expert review. It has value as a communication tool, which is what you say your customers are asking for. Numbers also have risks as communication tools. We assume that numbers are absolute, so something that has a score of 7 on a 10-point scale is significantly better than something that has a score of 5 on that same scale. Further, any scale that is close to a scale that is used in school grading also has risks because of the implicit meaning in those numbers (for example, a 4.0 scale or a 100-point scale has implicit meaning to an American audience).
I do not think that a numeric usability score is formal and a textual report is informal. The goal of the report, regardless of its format, is to communicate with the stakeholders about what works and what doesn't in a given design. A number is not a formal way of communicating this information to your stakeholders, and an assumption on anyone's part that a number is more formal than text is another risk of reporting a number. Further, reporting a number does not mean that you're doing quantitative work. Instead, you're assigning a qualitative meaning to a number (or range of numbers), which can introduce confusion about whether the work that you have done is actually qualitative or quantitative in nature.
If statistical significance is a requirement, then I recommend reading "Measuring the User Experience" by Thomas Tullis and William Albert. This means that you're no longer doing heuristic evaluations, but there are some relatively quick and low-cost methods to gather statistically-significant data.
I personally don't find expert reviews, or heuristic reviews, that useful as general tools. There are too many exceptions and context dependent issues.
For example in the survey you point to one of the questions is "Navigation-only pages (such as the home page) can be viewed without scrolling" that immediately has me asking:
Scrolling on what (phone? tablet? netbook? laptop?)
What about long reference pages that my experience with user testing tells me people are likely to prefer.
What is the solution? Splitting a badly designed IA so that long pages of links are pushed to many small pages of links will likely make things worse.
There are similar sort of problems with many of the questions.
If you want a number - look to things like the System Usability Scale or the Net Promoter Score that are based on what users think of the system, rather than heuristics that we hope correlate well with what the users think of the system.
I would say this is a good idea. Although in the past a five start method has worked just as well for me (with maybe half stars). It's also common to use a traffic light system to show what is good, okay but needs tweaking and bad.
In past expert reviews giving different aspects a score generates a set of results that can be viewed at a glance. For this to make more sense comparing the experience of the given product with others in the market shows who well/badly the product being reviewed is.
The criteria for an expert review should fit the problem. In the past I have broken reviews down into functionality areas and then overlapped this with overall issues like focus , consistency, content, language etc.
Giving a (software) product a numeric usability score is a thought I had in many of my past projects. Having a final verdict in form of a %-value or a grade would give me a certain comparability between projects and some customers would prefer this k
I'm wondering which one will be more valuable and have more career opportunities. Web usability or web accessibility. Please show statistics or any valid information if possible. --------------Solutions------------- Web Usability Bing says 472.000 hi
Our government website has thousands of informational pages, many online self-service task flows and 20+ content editors / stakeholder groups. In the past, periodic usability surveys have been conducted with varied success in determining actionable s
I've just set up an SMTP server on a relatively little used domain using Postfix and enabled greylisting with SQLGrey. So far it seems to be working OK, and while there's the slight irritation of delays to emails from new senders, I can see from the
I've looked at this site for the first time today. I've checked About, the FAQ and the FAQ and searched, and I don't seem to be able to find an answer on this site. These are the closest I can find: Difference between UI and UX What's the difference
I have a task to redesign a web page, the functionality is simple, but the code is robust. The problem is I think the users will give it a low score due to the blah colors and lack of presentation. I was pressed for time and just focused on the funct
As security folk, we tend to over- or under-estimate the capabilities and educatedness of our users, especially in terms of resistance to social engineering attacks. Stereotypically, this is due to us all being basement-dwelling social rejects, but m
We do a lot of web hosting management and have started to get into selling server management packages. However, we want to start monitoring servers for security related threats. Our clients may use their servers for stuff other than web hosting and s
Despite the efforts of UI and UX professionals, users sometimes get stuck, or experience bugs using software. I think we can agree that their frustration is an indicator of bad user experience. Yet, a meaningful, helpful conversation with a customer
Ivo Jansch made a comment in relation to this blog which got me thinking; We have the same problem. The backwardscompatibility issues are small, and we could easily fix our own apps. But we also do hosting for several hundreds of websites not created
This might not seem as a development question, but in essence it is. Let me explain how. Our main development focus is dynamic content pages. Some of our customers have asked us to allow them space in our servers (which they pay) for their old static
I am looking for a good solution to collaborate on Microsoft Office documents, we currently just edit directly on a Samba share but it's one big mess because sometimes people leave the office with their laptops while docs are open so swap files remai
I've just switched to Windows 7 from XP - under windows XP I had 2 shortcuts to mstsc.exe on my quick launch bar, and I want to do the same with Windows 7: "Remote desktop connection" - mstsc.exe "Console remote desktop connection" - m
Today, we're wrapping up our week-long series on marketing strategies with undercover marketing. If you missed the other posts in the series, we covered cause marketing, relationship marketing and scarcity marketing. What is Undercover Marketing? Und
I have an ADSL connection which has a /29 subnet allocated to it, giving me 6 usable IP addresses. Currently this has a cheap Netgear ADSL router attached, which has a built in switch. There are 3 servers attached, each with a public IP address. Each
I have an nForce 430 board and I want to use it's RAID 5 support with Windows 7. I can't make heads or tails about whether the latest 2011 drivers from nVidia will let me set this up. Some pages say yes and others say no. Does anyone actually have th
Update: I found that not only Windows Explorer but also other software suffers from too long path names. For example, in Firefox, when I save a webpage into a directory, I still cannot do it if its name or path name is too long. Does Firefox also rel
Two new laptops that contain an INTEL CORE I7-2920XM CPU have a Windows Experience Index score of 3.8. This laptop (a brand new Dell Precision 4600) responds very slow in various Office programs. Diagnostics has been run, and it didn't uncover any er
You may have noticed the recent BBC News item "EU bans pre-ticked website boxes to aid consumers" which was featured on the SitePoint podcast. The new legislation aims to eliminate hidden charges and costs. European websites will no longer be ab
I have trouble coming up with a system of assigning a rating to player's performance. Well, technically there is is a trivial rating system, but I don't like it because it would mean assigning negative scores, which I think most players will be disco