Who am I?

Gary Elsbernd

Sometimes I get an idea that won't leave me alone. Whether these ideas are about design patterns, technical development or implementation, archery or science fiction, the elsblog is where I put my ideas and hopefully share them with the world.

I am a passionate advocate for user centered design with more than 30 years of experience. In my current role as Principle Experience Designer for Sun Life (US) based out of the Kansas City office, I am particularly interested in usability and performance centered design in web and mobile applications. 

Measuring User Experience

In 1959 Donald Kirkpatrick introduced four levels of evaluation to the learning profession: reaction, learning, behavior, and results. These same levels can be massaged to describe the impact of XD. Thirty years later, Gloria Gery courageously informed us all that we might as well just weigh our students before and after learning, rather than use the metrics we were still using to determine the effectiveness of what we do for organizations.

Current tools and models for measuring design

Many organizations begin and end with user satisfaction measurement.  Measuring the user satisfaction is not enough. Although it is a good product metric, it doesn’t provide the information needed to the design team itself to improve upon.

Tool Description Categories/Questions Asked Scale Pros Cons

SUPR-Q

The SUPR-Q is seen as a reliable measure of a website's perceived quality, and is used broadly in many industries.

Usability

  1. The website is easy to use.
  2. It is easy to navigate within the website.

Credibility

  1. The information on the website is credible. (E-commerce variation: I feel comfortable purchasing from the website.)
  2. The information on the website is trustworthy. (E-commerce variation: I feel confident conducting business on the website.)

Appearance

  1. I find the website to be attractive.
  2. The website has a clean and simple presentation.

Loyalty

  1. I will likely return to the website in the future.
  2. How likely are you to recommend the website to a friend or colleague?

5 point Likert scale

Similar to NPS score

 

Most well known framework

Widely adopted

Ability to benchmark across other companies 

Loyalty questions don't apply to applications with captive audiences - for example, if you have a B2B portal, administrators and members have no choice as to what tool they use. Asking "How likely are you to recommend..." makes no sense.

8 questions is a lot to ask

Provides broad measure of the experience, but not specific enough to tell you what to fix

CSAT

CSAT understands the value customers are getting by measuring customer satisfaction. It’s basic but it works and is particularly helpful for assessing customer service. It’s measured on a 100 point scale.

  1. Overall, how satisfied are you with your most recent interaction with our company?
  2. Based on your most recent interaction with our company, how likely are you to purchase our products or services again?
  3. Based on your most recent interaction with our company, would you recommend our products or services to a friend or family member?
  4. If you would like to share any additional comments about your most recent interaction with our company, please enter them below.

5 point Likert scale and short answer

Similar to NPS score

Easy to interpret

Not specific to UX Design

Provides broad measure of the experience, but not specific enough to tell you what to fix

UUP

UUP is a tool for measuring the total UX rating of a product. UUP stands for Utility, Usability, and Presentation.

Survey at regular time intervals or ad hoc based on the push of significant updates.

Survey both during testing and in production.

Baseline score is set to zero, so you can see improvements or losses

  1. This system's capabilities meet my needs.
  2. This system is easy to use.
  3. This system is aesthetically pleasing.

6 point Likert scale, weighted (0 to 5) - no neutral

  • Utility worth is 3x
  • Usability is 2x
  • Presentation is 1x

Weighted to focus on priorities

 

Low lift to launch

Very high level, fewer questions reduces nuance

Provides broad measure of the experience, but not specific enough to tell you what to fix

QX Score (UserZoom)

QXscore is a standard for measuring user experience that quantifies users’ attitudes and behaviors into a single score and identifies opportunities to improve.

QXscore combines behavioral and attitudinal data, with task-level insights, captured in the UserZoom testing tool.

QX Score combines the questions from SUPR-Q with specific usability tasks.

QX Score has a 100 point scale.

Behavioral insights are more reliable than self-reported attitudes

Unknown if this can be applied outside of UserZoom.

Design maturity is hard to act on

Design maturity describes how well an organization listens to customer and user input, and how customer and user-centric its design processes are.  Design maturity results in stronger development processes that deliver more consistent products, services, and experiences that delight users, win customers, and enable true differentiation for your brand.  

The next question is, how do we evaluate design maturity?  There are many models out there to describe the continuum on which organizations exist.

  • Deloitte has 4 levels
  • Invision has 5 levels
  • Nielsen/Norman originally had 8 levels, but now has 6 levels

For the purpose of this article, let's use Nielsen Group’s current scale as an example:

UX Maturity Stage Description

1. Absent

UX is ignored or nonexistent.

2. Limited

UX work is rare, done haphazardly, and lacking importance.

3. Emergent

The UX work is functional and promising but done inconsistently and inefficiently.

4. Structured

The organization has semi-systematic UX-related methodology that is widespread, but with varying degrees of effectiveness and efficiency.

5. Integrated

UX work is comprehensive, effective, and pervasive.

6. User-driven

Dedication to UX at all levels leads to deep insights and exceptional user-centered–design outcomes.

Nielsen/Norman Group UX Maturity Model, 2021 (Source)

While design maturity scales like this are a great way to better understand where the design is at your organization and what it should aspire to, its interpretation can be subjective:

  • Executives might argue that the team is on stage 4 because it has a dedicated budget within the company.
  • A design team member might claim that it's on stage 2 (developer-centered) because design decisions are made without enough user research.
  • Different designers can also disagree about their impact and influence on Product and Engineering, and that can vary by team or individual.

This is a great discussion to have both with your team and with the big bosses. However, a design maturity scale often lacks something tangible to be measured against and to indicate if any progress has been made over time. Increasing the maturity of the design team is not something that can be rushed. It can take years.

Measurable attributes of design maturity

To measure the design maturity, we first need to define metrics that can be used by any design team, regardless of the context of work and team composition.

To do so, we can look at the basic elements of any design project. 

Design system

A style guide is an artifact of design process. A design system is a living, funded product with a roadmap & backlog, serving an ecosystem. It consists of a guideline and repository of patterns, components and assets an organization has, but it goes beyond: it means designing, planning and maintaining a consistent experience, unique to the brand and its tone and voice.

Design process

The design process is, simply, the design process used by the team. A library of methodologies and good practices to ensure a high quality standard and consistency in the work done. Not all problems can be solved with the same approach, but having a main framework can facilitate collaboration and makes other tools and methodologies more accessible.

Design vision

Design vision is the direction where the team sees the design heading. It is formed by principles, values, and even their "blue sky" concept for the business, aligned with the company's mission and goals. 

Whether or not we are actively thinking about these three elements during a project, they are the main ingredients of any design team. Every design project has an interface system, follows a process, and has a vision behind it.

Defining the maturity metrics

Within these three elements in mind, we can define points to be used as metrics for each. These points can vary a lot depending on what is important for the team and for the business, from something specific and tactical to a broader satisfaction score with that element. Here are some examples of what could be used as metrics:

Design system

    • The presence, stability, use and effectiveness of interface elements and composition, such as color, typography and iconography, fully mobile-responsive and meeting accessibility standards.
    • A clear voice and tone guide, keeping the experience consistent and on-brand.
    • The designers' satisfaction working with the current pattern library.

Design process

    • A consistent high-level process, from discovery and research to QA and launch tasks.
    • Clarity for the team of all design steps and knowledge of the tools and methods available.
    • Opportunity (space and time) to study and experiment different methods and cross-team collaboration.

Design vision

    • Clear business goals and a clear user market. Every designer should understand the company's mission and high-level roadmap.
    • Design principles and vision that the team shares and is confident to use in their projects.
    • Design initiatives and projects, aligned to its vision, and not necessarily directly linked to a company's project or product.
    • Design literacy in the organization, and how deep and wide is the impact and influence of user-centered design.

 

Setting up a process to measure it

Regardless of the metric points decided for each element, the team will need a rating scale (o to 10? Five stars? Emoji faces?) and specific weights for each point selected.

Defining what and how to measure should be a process owned by the team. Every design team has a different and unique context that needs to be taken in consideration. Any formula has the risk of being too generic. Discussing the metrics and the approach is a great starting point for any evaluation discussion and also makes the team feel like they are owning their self evaluation mechanism.

If the design team concludes that the current state of any of these elements is not satisfactory, they have a clear direction on what to work on and a good argument to ask for more resources for the team if needed.

One can argue that it’s impossible to do well in any of these three areas without doing well the others. It is true, but their level and quality can vary a lot. And that’s exactly why we need to measure it.

Even if the all the scores are initially low, doing this exercise with the team is the best way to check that everyone is on the same page and to measure the progress of the team over time.

On the other hand, if the team seems to have everything under control, there is always new challenges in each of these three elements. Whether because patterns need to be updated, the team has to onboard new members, or a process needs to be reviewed for a new customer channel that is coming up, for example.

The reality is that the design industry is extremely dynamic and the context can switch quickly and it's the same for the design team. Nothing will be perfect — everything can be perfected.

Whether the team is centralized or distributed, with these three elements as talking points in the agenda, it is easier for the designers to assess how they, individually, perceive where they are to each one of them, starting the conversation about what needs to be done and where to invest more.