Detecting relative bias using the Bland-Altman method: Consequences of unequal imprecision

Devices to successfully detect glaucoma are important tools for treating the disease; Bland-Altman plots are often used to determine the relative bias between two instruments. Bland-Altman assumes that the two instruments have equal imprecision but this strong assumption is often unchecked. This study demonstrates the consequences when imprecisions are unequal. Using simulated intraocular pressure (IOP) data for two devices (N=1000) four models illustrating possible scenarios were constructed. In the first model both methods have equal imprecision standard deviations and a bias between the devices increasing with each increase in the true level. The second had equal imprecision and no bias. The third had unequal imprecisions and no bias. The fourth had unequal imprecision and the same bias as the first model. In the two models that held the assumption of equal imprecision, the Bland-Altman method correctly identified the bias (P<0.0001) in the first model and the lack of bias (P=0.6534) in the second. In the third model, Bland-Altman incorrectly concluded there was a bias (P<0.0001). In the fourth model, Bland-Altman failed to detect the bias (P=0.7609). In cases where two devices have the same imprecision, Bland-Altman plots are effective tools for determining relative bias. When this assumption is violated, false positives and false negatives indistinguishable from true cases can occur. Researchers must strongly justify the assumption of equal imprecision when using Bland-Altman due to the high risk of erroneous conclusions. These mistakes can greatly hinder the development of effective glaucoma detection technologies.