Issues of assessing reliability in complex public health survey data

Issues that relate to assessing reliability in complex public health survey data are the same as in simple straightforward surveys except they become much more difficult to deal with in complex survey data. Issues relate to the match of the survey to potential respondents, nature of items on the survey (factual, descriptive vs. attitudinal, knowledge, behavioral); subscale and total scoring; item directionality for scoring; missing data; and need for matching data while maintaining anonymity across surveys. Reliability methods differ depending on the nature of the items. Reliability of factual or descriptive items can be assessed using items that should be correlated and looking for agreement among the set of items, reliability of knowledge items can be assessed using traditional test-retest or split-half methods, and reliability of attitudinal and behavioral items can be assessed using internal consistency indices. The major issue with complex surveys is they tend to include a variety of all of these types of items. Determining subscale and total scores is a complex process in these cases. The use of attitudinal and behavioral items that include items directional reversals is a major problem in surveys that requires score reflection and reduces reliability. Missing data is always a concern in these types of surveys and can result from several reasons. When multiple surveys are given to the same subjects there is a need to match data and this can be a challenge if subject anonymity is desired. Each of these issues is presented and possible solutions are proposed.