Reliability of the PEONIES approach to consumer-centered to long-term care quality assurance/quality improvement

The PEONIES approach is a unique method of assessing and measuring the quality of community-based long-term services and supports, based on consumer-specific outcomes in 12 areas. It uses semi-structured interviews to identify specific outcomes of importance to consumers; determine the current status of these outcomes; identify supports and services necessary to support achievement of the desired outcomes; and assess the adequacy and acceptability of the necessary supports/services to the consumer. This detailed, consumer-specific information is used in care planning and to report quality indicators at the systems level.

Inter-rater reliability of this approach was tested during late 2007 and early 2008. Project development staff observed interviews of long-term care consumers (“frail elders,” adults with physical disabilities, and adults with developmental disabilities) conducted by skilled quality reviewers and care managers trained in the PEONIES approach. Training was conducted in a six-hour session, after which each interviewer conducted 1 or 2 interviews. A second, 8-hour training was then conducted to reinforce training in areas where there appeared to be confusion.

Interviewers and observers recorded information independently. Inter-rater reliability was assessed overall as percent agreement. Intraclass correlations were calculated to assess reliability separately for the 12 outcome areas, by type of interviewer, and by type of consumer. Detailed qualitative analysis also was done, to identify specific challenges to reliability.

This session will present findings from the reliability test, with particular emphasis given to the qualitative results and the implications for development and implementation of this type of approach to assessing quality.