Evaluation and redesign of public health information technology: A cost-effective, reproducible, and user-informed approach

Background: Ensuring key audiences' access to high-quality information is a primary goal of health informatics. Public health practitioners often face resource constraints that limit the extent to which information technology may be evaluated. This study outlines the process of developing and implementing a cost-effective and rapidly deployed usability protocol that yields actionable results for application redesign in a short time-frame. Objective: To develop and implement a reproducible remote testing protocol for public health practitioners with limited resources to evaluate and redesign health IT tools. Methods: We developed a remote usability testing protocol and applied it to evaluate the Agency for Healthcare Research and Quality's National Resource Center website (a central repository for health IT information). Protocol development, survey implementation, analyses, and summary of results occurred within a 2 month span. First, we identified and recruited a small group of representative users. Second, to reduce logistical burdens, we developed a remote testing protocol where users could work in their natural environment and be observed by evaluators through a web-conferencing tool. Our protocol consisted of 2 main sections: (1) cognitive walk-through where users received minimal guidance as they conducted tasks using the application (while verbalizing their thoughts), and (2) system usability scale (SUS) where users provided quantifiable responses to specific questions related to system usability. Results: The cognitive walk-through yielded a series of themes related to usability issues. The SUS yielded a numeric assessment of usability, which showed whether the tool's usability was above or below average. Together, the results provided a set of actionable recommendations for redesign. Conclusion: By identifying a limited number of representative users, and conducting remote testing that utilized complimentary and established techniques, we developed actionable recommendations in a short time-frame. This protocol will be of value to practitioners faced with application redesign under significant logistical limitations.