Research

Ph.D. Research

Abstract

Ten guidelines for writing UPDs were developed by consulting usability practitioners through two questionnaires and a card sort. These guidelines are (briefly): be clear and avoid jargon, describe problem severity, provide backing data, describe problem causes, describe user actions, provide a solution, consider politics and diplomacy, be professional and scientific, describe your methodology, and help the reader sympathize with the user. A fourth study compared usability reports collected from 44 evaluators, both practitioners and graduate students, watching the same 10-minute UT session recording. Three judges measured problem detection for each evaluator and graded the reports for following 6 of the UPD guidelines.

There was support for existence of an evaluator effect, even when watching pre-recorded sessions, with low to moderate individual thoroughness of problem detection across all/severe problems (22%/34%), reliability of problem detection (37%/50%) and reliability of severity judgments (57% for severe ratings). Practitioners received higher grades averaged across the 6 guidelines than students did, suggesting that the guidelines may be useful for grading reports. The grades for the guidelines were not correlated with thoroughness, suggesting that the guideline grades complement measures of problem detection.

A simulation of evaluators working in groups found a 34% increase in severe problems found by adding a second evaluator. The simulation also found that thoroughness of individual evaluators would have been overestimated if the study had included a small number of evaluators. The final recommendations are to use multiple evaluators in UT, and to assess both problem detection and description when measuring evaluation effectiveness.

Electronic copy available from: http://scholar.lib.vt.edu/theses/available/etd-03222006-201913/

M.S. Research

Abstract

This study explored end-user classification of self-reported critical incidents (usability issues) using the UAF, a technique that was hoped to reduce expert interpretation of usability problems. It also explored end-user critical incident reporting from a usability session recording, rather than reporting incidents as soon as they occur, a technique that could be used in future studies to compare effectiveness of usability methods. Results indicate that users are not good at diagnosing their own critical incidents due to the level of detail required for proper classification, although observations suggest that users were able to provide usability information that would not have been captured by an expert observer. The recording technique was successful, and is recommended for future studies to further explore differences in the kind of information that can be gathered from end-users and from experts during usability studies.

Electronic copy available from: http://scholar.lib.vt.edu/theses/available/etd-11122001-143830/

Other Research Interests

Usability Diagnosis and Usability Expertise

I have found that usability diagnosis (identifying and understanding usability problems) is very similar to medical diagnosis. There is a great deal of literature about how doctors diagnose medical problems, and how medical students develop diagnostic skills through the medical education process. I am interested in relating this literature to usability to better understand how usability practitioners develop usability expertise and how they diagnose usability problems.

Usability Problem Classification

I have been working with Dr. Hartson in the CS department on the User Action Framework (UAF), a structure for the classification of usability problems, based on Don Norman's theories about stages of action. It is designed to be a user-centric classification method, focusing on true problem - how the problem interfered with the user's ability to use the site (planning, translation, physical actions, assessment, outcome) - as opposed to superficial technology-centric design issues (fonts, graphics, etc.). Use of the UAF to classify problems makes it easier to compare problems from different usability sessions, or even different user interfaces. It also encourages more thorough problem descriptions, by encouraging the usability practitioner to describe the root cause of the problem, instead of the superficial appearance.

UAF Toolkit

May 2000 - October 2000 I helped design the UAF Toolkit, a set of tools built around the UAF to allow Usability Practitioners to classify usability problems, store them in a centralized database, track their resolution, and perform other functions associated with usability studies and analyzing usability data.

Remote Usability Evaluation

Methods for performing usability evaluations where the participant is separated from the experimenter by either time and/or space. This can be particularly useful when the participant is in their own environment (i.e. at home for consumer products, on the job for business products), where they can evaluate the software in a real setting, instead of an artificial lab setting.

User-Reported Critical Incidents

A critical incident is an interaction between a user and a system that has a strong impact (either positive or negative) on the user's ability to perform a task. In the area of software interfaces, critical incidents are frequently reported by the experimenter, who observes a user interacting with software, identifies when a critical incident has occured, and makes a judgement about why the incident occured. Users can also report their own critical incidents. The advantage is that they understand what they were trying to do and how the software helped or hindered them. The disadvantage is that they need to be trained in identifying and reporting the incidents. If users can be trained in the UAF they may be able to give more complete descriptions of the problems encountered. It may also then be easier to compare problems across users.

Universal Access

Design issues associated with making software available to anyone - elderly, people with disabilities or special needs, people speaking other languages, illiterate, children, etc.