Innovative use of multiple methods to assess health informatics system standards and interoperability

Background:Health informatics systems (HIS) are essential in collection and management of health research data. While HIS are proliferating rapidly, little is published about evaluation of system standards, functional requirements, and interoperability. We developed an innovative evaluation approach using best practices to assess 4 HIS created in different settings but supporting the same health data collection effort. Best practices included defining the evaluation criteria, objectives, and framework early on, involving stakeholders and openly communicating findings, and combining existing methods in new and interesting ways. Objective/Purpose:The approach would yield evidence for HIS functional standards and interoperability characteristics that could serve as a model for future large-scale health data collections, especially in government settings. Methods:We designed a multi-method informatics evaluation to 1) benchmark system functionality against a set of specified standards, and 2) test 4 informatics systems using real-case scenarios, leveraging collaborations among stakeholders (system funders, developers, implementers) and evaluators. Evaluators used benchmarking to review the 4 systems' documentation against relevant standards. We collected qualitative and quantitative information, using mock user roles (system administrators, managers, data collectors), about the systems' performance in practice through scenario-testing. Results:Benchmarking defined key standards for the 4 systems. Review of system/user documentation allowed evaluators to identify the strongest functional areas such as case management, and where gaps existed such as in specimen tracking. System documents and test environments made feasible a limited examination of interoperable modules across the HIS; this identified the need for further prototype testing to pinpoint system functions and modules that could be shared. Scenario-testing with real use-cases provided objective information and qualitative input from system users. Together, benchmarking and scenario-testing identified mismatches in functionality (e.g., case management) between system documentation and real use. Discussion/Conclusions:This evaluation approach identified critical functional standards and gaps in characteristics across 4 informatics systems--both open-source and proprietary--for specific remediation efforts.