In 1959 Donald Kirkpatrick introduced four levels of evaluation to the learning profession: reaction, learning, behavior, and results. These same levels can be massaged to describe the impact of XD. Thirty years later, Gloria Gery courageously informed us all that we might as well just weigh our students before and after learning, rather than use the metrics we were still using to determine the effectiveness of what we do for organizations.
Current tools and models for measuring design
Many organizations begin and end with user satisfaction measurement. Measuring the user satisfaction is not enough. Although it is a good product metric, it doesn’t provide the information needed to the design team itself to improve upon.
| Tool | Description | Categories/Questions Asked | Scale | Pros | Cons |
|
SUPR-Q |
The SUPR-Q is seen as a reliable measure of a website's perceived quality, and is used broadly in many industries. |
Usability
Credibility
Appearance
Loyalty
|
5 point Likert scale Similar to NPS score
|
Most well known framework Widely adopted Ability to benchmark across other companies |
Loyalty questions don't apply to applications with captive audiences - for example, if you have a B2B portal, administrators and members have no choice as to what tool they use. Asking "How likely are you to recommend..." makes no sense. 8 questions is a lot to ask Provides broad measure of the experience, but not specific enough to tell you what to fix |
|
CSAT |
CSAT understands the value customers are getting by measuring customer satisfaction. It’s basic but it works and is particularly helpful for assessing customer service. It’s measured on a 100 point scale. |
|
5 point Likert scale and short answer Similar to NPS score |
Easy to interpret Not specific to UX Design |
Provides broad measure of the experience, but not specific enough to tell you what to fix |
|
UUP |
UUP is a tool for measuring the total UX rating of a product. UUP stands for Utility, Usability, and Presentation. |
Survey at regular time intervals or ad hoc based on the push of significant updates. Survey both during testing and in production. |
Baseline score is set to zero, so you can see improvements or losses
6 point Likert scale, weighted (0 to 5) - no neutral
Weighted to focus on priorities
|
Low lift to launch Very high level, fewer questions reduces nuance |
Provides broad measure of the experience, but not specific enough to tell you what to fix |
|
QX Score (UserZoom) |
QXscore is a standard for measuring user experience that quantifies users’ attitudes and behaviors into a single score and identifies opportunities to improve. |
QXscore combines behavioral and attitudinal data, with task-level insights, captured in the UserZoom testing tool. QX Score combines the questions from SUPR-Q with specific usability tasks. |
QX Score has a 100 point scale. |
Behavioral insights are more reliable than self-reported attitudes |
Unknown if this can be applied outside of UserZoom. |
Design maturity is hard to act on
Design maturity describes how well an organization listens to customer and user input, and how customer and user-centric its design processes are. Design maturity results in stronger development processes that deliver more consistent products, services, and experiences that delight users, win customers, and enable true differentiation for your brand.
The next question is, how do we evaluate design maturity? There are many models out there to describe the continuum on which organizations exist.
- Deloitte has 4 levels
- Invision has 5 levels
- Nielsen/Norman originally had 8 levels, but now has 6 levels
For the purpose of this article, let's use Nielsen Group’s current scale as an example:
| UX Maturity Stage | Description |
|
1. Absent |
UX is ignored or nonexistent. |
|
2. Limited |
UX work is rare, done haphazardly, and lacking importance. |
|
3. Emergent |
The UX work is functional and promising but done inconsistently and inefficiently. |
|
4. Structured |
The organization has semi-systematic UX-related methodology that is widespread, but with varying degrees of effectiveness and efficiency. |
|
5. Integrated |
UX work is comprehensive, effective, and pervasive. |
|
6. User-driven |
Dedication to UX at all levels leads to deep insights and exceptional user-centered–design outcomes. |
While design maturity scales like this are a great way to better understand where the design is at your organization and what it should aspire to, its interpretation can be subjective:
- Executives might argue that the team is on stage 4 because it has a dedicated budget within the company.
- A design team member might claim that it's on stage 2 (developer-centered) because design decisions are made without enough user research.
- Different designers can also disagree about their impact and influence on Product and Engineering, and that can vary by team or individual.
This is a great discussion to have both with your team and with the big bosses. However, a design maturity scale often lacks something tangible to be measured against and to indicate if any progress has been made over time. Increasing the maturity of the design team is not something that can be rushed. It can take years.
Measurable attributes of design maturity
To measure the design maturity, we first need to define metrics that can be used by any design team, regardless of the context of work and team composition.
To do so, we can look at the basic elements of any design project.
Design system
A style guide is an artifact of design process. A design system is a living, funded product with a roadmap & backlog, serving an ecosystem. It consists of a guideline and repository of patterns, components and assets an organization has, but it goes beyond: it means designing, planning and maintaining a consistent experience, unique to the brand and its tone and voice.
Design process
The design process is, simply, the design process used by the team. A library of methodologies and good practices to ensure a high quality standard and consistency in the work done. Not all problems can be solved with the same approach, but having a main framework can facilitate collaboration and makes other tools and methodologies more accessible.
Design vision
Design vision is the direction where the team sees the design heading. It is formed by principles, values, and even their "blue sky" concept for the business, aligned with the company's mission and goals.
Whether or not we are actively thinking about these three elements during a project, they are the main ingredients of any design team. Every design project has an interface system, follows a process, and has a vision behind it.
Defining the maturity metrics
Within these three elements in mind, we can define points to be used as metrics for each. These points can vary a lot depending on what is important for the team and for the business, from something specific and tactical to a broader satisfaction score with that element. Here are some examples of what could be used as metrics:
Design system
-
- The presence, stability, use and effectiveness of interface elements and composition, such as color, typography and iconography, fully mobile-responsive and meeting accessibility standards.
- A clear voice and tone guide, keeping the experience consistent and on-brand.
- The designers' satisfaction working with the current pattern library.
Design process
-
- A consistent high-level process, from discovery and research to QA and launch tasks.
- Clarity for the team of all design steps and knowledge of the tools and methods available.
- Opportunity (space and time) to study and experiment different methods and cross-team collaboration.
Design vision
-
- Clear business goals and a clear user market. Every designer should understand the company's mission and high-level roadmap.
- Design principles and vision that the team shares and is confident to use in their projects.
- Design initiatives and projects, aligned to its vision, and not necessarily directly linked to a company's project or product.
- Design literacy in the organization, and how deep and wide is the impact and influence of user-centered design.
Setting up a process to measure it
Regardless of the metric points decided for each element, the team will need a rating scale (o to 10? Five stars? Emoji faces?) and specific weights for each point selected.
Defining what and how to measure should be a process owned by the team. Every design team has a different and unique context that needs to be taken in consideration. Any formula has the risk of being too generic. Discussing the metrics and the approach is a great starting point for any evaluation discussion and also makes the team feel like they are owning their self evaluation mechanism.
If the design team concludes that the current state of any of these elements is not satisfactory, they have a clear direction on what to work on and a good argument to ask for more resources for the team if needed.
One can argue that it’s impossible to do well in any of these three areas without doing well the others. It is true, but their level and quality can vary a lot. And that’s exactly why we need to measure it.
Even if the all the scores are initially low, doing this exercise with the team is the best way to check that everyone is on the same page and to measure the progress of the team over time.
On the other hand, if the team seems to have everything under control, there is always new challenges in each of these three elements. Whether because patterns need to be updated, the team has to onboard new members, or a process needs to be reviewed for a new customer channel that is coming up, for example.
The reality is that the design industry is extremely dynamic and the context can switch quickly and it's the same for the design team. Nothing will be perfect — everything can be perfected.
Whether the team is centralized or distributed, with these three elements as talking points in the agenda, it is easier for the designers to assess how they, individually, perceive where they are to each one of them, starting the conversation about what needs to be done and where to invest more.