Usability Framework

The first step towards a successful usability evaluation of a specific human-robot interaction is understanding the concept and terminology of usability. This section provides a terminological basis and standards to refer to when investigating the usability of wearable robotics.

What is usability?

Usability is often defined as “the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.”

This definition is provided by the International Organization for Standardization (ISO), within the standards 9241-11 (2018) and 9241-210 (2019). However, there are many other definitions of usability to be found - we decided to stick with ISO. The usability methods and measures provided by the Interactive Usability Toolbox are categorized by the framework of ISO TR 16982 (2002). This framework describes method types, their application potential and fittings for different evaluation contexts.

Functionality, ease of use, comfort
- how about that?

It can be hard to wrap your head around the term usability. We understand that users and developers like to describe their usage experience with specific usability attributes.

But how is an attribute like "ease of use" defined? To what extent does it describe a system's effectiveness, efficiency, and satisfaction? Our toolbox supports 49 different attributes which were carefully studied and categorized. Learn more about each in our Glossary.

Context matters!

The usability of a system is always defined by the context of use. Usage experiences can strongly differ when testing the same design with another user, varying environments, or following different tasks.

Understanding and specifying the context of use is an essential step of user-centered design and usability evaluation. Can you assume that your device will work in dynamic, real-life situations if you only tested it in the controlled lab environment?

Glossary

To support the endeavors of more standardized terminology and definitions, we started forming a glossary that defines and classifies the terms inside the usability universe. In our glossary, you find definitons of different usability attributes. If you don't agree with these, check the "How you can help" section to find out how to help us improve the content of this website.

Help us defining usability for wearable robotics!

Participate in our 15-minute survey to find consensus on the definitions of usability attributes. These are the basis to establish a benchmark for usability evaluation in the WR community.

Usability Evaluation

Evaluating your technical solution to understand if you successfully addressed or solved the user-based problem is a key element of user-centered design. Usability evaluation allows you to compare your solution to previous iterations, other devices, user expectations and generate new requirements to enter another design iteration.

Which methods make sense?

The complex human-robot interaction that users of wearable robotics encounter have many facets which can and should be evaluated.

The usability methods and measures provided on our toolbox are categorized by the framework of ISO TR 16982 (2002), which describes method types, their application potential and fittings for different evaluation contexts. Each method has their own pros and cons. You decide which to use based on the insights you would like to get.

Who are your target users?

Understanding, evaluating, and integrating human factors is a critical challenge in the design of wearable robotics. Your context of use starts with the user, not the technology.

Who are the primary users? Are there secondary user groups? Make sure you understand your target users and involve them in the design early on. Valuable insights can be generated by evaluating your solutions with secondary user groups, like therapists, caregivers, or family members.

Define your scope

How many participants do you need for your usability study? This mostly depends on your research intentions and resources.

One can value a single user’s opinion to trigger design changes, but it is also essential to distinguish between personal preferences and global usability issues. According to Nielsen and Landauer (1993), a group of five users will be able to identify 80% of your device’s usability issues. However, your study will likely require more than 20 subjects to prove your device intervention effectiveness statistically.

Tips and Tricks

A good usability study moderator needs experience, so go ahead and try to speak and test with as many users as possible. To give you a head start, we reached out to more than 150 developers of wearable robots to ask them about the usability evaluation practice. From the data, we were able to propose a set of evaluation recommendations. This work was published by Meyer et al. (2021) in the Journal of NeuroEngineering and Rehabilitation, see the Further reading section for more information.

Find the right balance

We recommend to consciously distribute evaluation efforts between the usability dimensions effectiveness, efficiency, and satisfaction.

While this distribution will likely adapt as technology matures it is important to consider efficiency and satisfaction from the very start of a WRD project. This implies that developers should be inclined to include target users as soon as possible, also the early stages of WRD development. A simple walk-through, interviews, or focus groups with target users can shape your concept or low-fidelity prototype in the right direction from the start.

Learn to value qualitative data

In contrast to what is mainly reported in scientific literature, a usability evaluation protocol should include substantial qualitative measures.

While quantitative data allows for comparison to previous design iterations and the state of the art, qualitative data helps identifying usability issues that may not be obvious to WRD developers when only collecting and comparing numbers. Also, qualitative evaluation by nature calls for more personal interaction with the target users and their satisfaction with the device. There are two main standards that can be used to ensure the quality of the data collected with qualitative research: COREQ and SRQR.

Seek for comparability

The wearable robotics field is missing evaluation benchmarks. For benchmarks to arise, we need to start using similar, validated items that generate comparable outcomes.

How about using a usability evaluation protocol with 2/3 of reproducible, standardized measures and 1/3 of customized tools? This will likely increase data validity and generalizability of your data and could help to establish usability evaluation benchmarks.

About User-centered design

User-centered design (UCD) requires the iterative involvement of users throughout the design of technology, to focus on them and their needs. Therefore, it should be implemented from the very beginning of the project and through all the stages of development. This will ensure that your technology solves the need it aimed to aid and that it matures towards a technology that fulfills the requirements of the market and the users.

It is never too early

Before any prototype is built, assess whether there is a real need and interest by end users to develop a technology. Reach users, ask them and observe which are their real needs. When you identify the one you want to address, validate with the users if they are interested in solving it with a WRD. Is it really the best solution? Only if the advantages of a WRD are clear above other solutions and users show interest in a WRD as a solution, move on to design your device.

This early need elicitation and validation can be easily performed with methodologies like contextual interviews, user Interviews, focus groups, user observation or task analysis.

Iterative evaluation

Implementing UCD iteratively throughout the design and development of a device from beginning to end will ensure that your device’s components and features work well for the intended users and context, and thus your device’s usability. It is a way to avoid problems due to overlooking important user requirements after a device has been fully developed, that might undermine its pathway towards reaching the market and end users.

In other words, implementing UCD from the first stages of technology design saves time, money, and effort!

What about ethics?

Although the methods to implement UCD are not invasive or risky, it is always necessary to have approval from an ethics committee before reaching the users, especially if people with any kind of impairment, illness, or medical condition are involved.

When implementing UCD, researchers are treating people’s personal and sometimes medical data, as well as their perspectives, beliefs and habits. Thus, a protocol approved by an ethics committee guarantees the protection and appropriate management of such data. Users must be informed of the goal of the study and provide informed consent before participating.

Understanding the Toolbox

The Interactive Usability Toolbox (IUT) combines the knowledge shared on this wiki page. It is the output of more than five years of experience in user-centered design of wearable robotics and represents various studies published by the Rehabilitation Engineering Laboratory.

The vision

If you shouldn't compare apples and oranges, why should you do the same with usability protocols?

The Interactive Usability Toolbox is designed to help you define your context of use and find measures that fit it. Furthermore, we can tell you if other developers have used those measures in comparable studies. This way, you will use similar protocols like others, which allows you to compare your results and insights.

The backbone

Each item in our usability evaluation database (150+ methods/measures) has been carefully studied to characterize the context fit to wearable robotics applications. Some tools only apply to individuals with spinal cord injuries, others are specifically validated with children. Furthermore, the evaluation scope of each measure is described using our extensive list of usability attributes.

Use our Wizard to find all measures fitting your context of use and evaluation scope!

Based on real world data

Our database is continuously growing and improving. The data comes from literature reviews, individual studies, websites, own experiences, and dedicated surveys. Currently, our database contains 141 usability studies shared by 25 contributors around the world.

We constantly learn about more tools that can be used for wearable robotics usability evaluation. With the recorded use function, we are able to show you if and to what extent specific tools have been used for the same context of use before.

We need your help!

We don't want this website and database to be a one-way street. We invite you to become an active contributor to the database and the information displayed on the website. Start today and help us facilitate the search for and selection of usability metrics.

Become an active contributor

Our aim is to make the database, and our recommendations as representative of current practices as possible. To do that, we need more real-world data - from you!

We welcome you to sign up as a contributor and start helping us to reach our shared vision of promoting user-centered design.

Tell us about your usability evaluations

As a contributor, you can submit requests to add data to our database. This will help expand the database, and improve the 'Recorded Use' ratings of each existing evaluation item. With this, we will be able to make better, more context-specific recommendations.

Tell us which usability evaluation methods you used in the development of your wearable robot and try to be as specific as possible when explaining your context of use.

Help us to continuously improve

Let's improve this platform and its content together. We can use your help with constructive feedback and specific improvement suggestions. Let us know your usability evaluation experiences, suggestions, or if you have a great idea worth exploring by submitting open feedback in the contribution page. Thank you for your help!

Further reading

Usability evaluation as part of user-centered design is an important mindset and approach to follow in the research and development of wearable robotics. If you are interested in understanding the depths and benefits of usability testing & evaluation, we can highly recommend you a variety of further readings on the topic. Below, we list a couple of websites, books and papers that can help you further strengthen your understanding of user-centered design.

Our usability-related publications

  • J. T. Meyer, R. Gassert and O. Lambercy, "An analysis of usability evaluation practices and contexts of use in wearable robotics," J NeuroEngineering Rehabil 18, 170 (2021). https://doi.org/10.1186/s12984-021-00963-8
  • J. T. Meyer, J. Dittli, A. Stutz, O. Lambercy and R. Gassert, "A Method to Evaluate and Improve the Usability of a Robotic Hand Orthosis from the Caregiver Perspective," 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), 2020, pp. 605-610, doi: 10.1109/BioRob49111.2020.9224376
  • K. Karacan, J. T. Meyer, H. I. Bozma, R. Gassert and E. Samur, "An Environment Recognition and Parameterization System for Shared-Control of a Powered Lower-Limb Exoskeleton," 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), 2020, pp. 623-628, doi: 10.1109/BioRob49111.2020.9224407
  • J. T. Meyer, S. O. Schrade, O. Lambercy and R. Gassert, "User-centered Design and Evaluation of Physical Interfaces for an Exoskeleton for Paraplegic Users," 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR), 2019, pp. 1159-1166, doi: 10.1109/ICORR.2019.8779527

  • For more information to our work, see the About Us page.

    Websites

    Recommended books

    • Norman, Don. The design of everyday things: Revised and expanded edition. Basic books, 2013.
    • Ku, Bon, and Ellen Lupton. Health Design Thinking: Creating Products and Services for Better Health. MIT Press, 2020.
    • Rubin, Jeffrey, and Dana Chisnell. Handbook of usability testing: how to plan, design and conduct effective tests. John Wiley & Sons, 2008.
    • Albert, William, and Thomas Tullis. Measuring the user experience: collecting, analyzing, and presenting usability metrics. Newnes, 2013