This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Burke, M. J. Articles by Dunlap, W. P. Search for Related Content Social Bookmarking                         What's this?

Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide

Tulane University

William P. Dunlap

Tulane University

The authors present guidelines for establishing a useful range for interrater agreement and a cutoff for acceptable interrater agreement when using Burke, Finkelstein, and Dusig’s average deviation (AD) index as well as critical values for tests of statistical significance with the AD index. Under the assumption that judges respond randomly to an item or set of items in a measure, the authors show that a criterion for acceptable interrater agreement or practical significance when using the AD index can be approximated as c/6, where c is the number of response options for a Likert-type item. The resulting values of 0.8, 1.2, 1.5, and 1.8 are discussed as standards for acceptable interrater agreement when using the AD index with 5-, 7-, 9-, and 11-point items, respectively. Using similar logic, the AD agreement index and interpretive standard are generalized to the case of a response scale that involves percentages or proportions, rather than discrete categories, or at the other extreme, the assessment of interrater agreement with respect to the rating of a single target on a dichotomous item (e.g., yes-no, agree-disagree, true-false item formats). Finally, the usefulness of these guidelines for judging acceptable levels of interrater agreement with respect to the metric (or units) of the original response scale is discussed.

Organizational Research Methods, Vol. 5, No. 2, 159-172 (2002)
DOI: 10.1177/1094428102005002002


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				N. de Cuyper, B. Sora, H. de Witte, A. Caballer, and J. M. Peiro Organizations' Use of Temporary Employment and a Climate of Job Insecurity among Belgian and Spanish Permanent Workers Economic and Industrial Democracy, November 1, 2009; 30(4): 564 - 591. [Abstract] [PDF]

				O. Ludtke and A. Robitzsch Assessing Within-Group Agreement: A Critical Examination of a Random-Group Resampling Approach Organizational Research Methods, July 1, 2009; 12(3): 461 - 487. [Abstract] [PDF]

				D. J. Pasisz and G. M. Hurtz Testing for Between-Group Differences in Within-Group Interrater Agreement Organizational Research Methods, July 1, 2009; 12(3): 590 - 613. [Abstract] [PDF]

				A. Zornoza, V. Orengo, and V. Penarroja Relational capital in virtual teams: the role played by trust Social Science Information, June 1, 2009; 48(2): 257 - 281. [Abstract] [PDF]

				S. Sonnentag and J. Volmer Individual-Level Predictors of Task-Related Teamwork Processes: The Role of Expertise and Self-Efficacy in Team Meetings Group Organization Management, February 1, 2009; 34(1): 37 - 66. [Abstract] [PDF]

				A. Cohen, E. Doveh, and I. Nahum-Shani Testing Agreement for Multi-Item Scales With the IndicesrWG(J) and AD M(J) Organizational Research Methods, January 1, 2009; 12(1): 148 - 164. [Abstract] [PDF]

				D. A. Newman and H.-P. Sin How Do Missing Data Bias Estimates of Within-Group Agreement? Sensitivity of SD WG, CVWG, rWG(J), rWG(J) * , and ICC to Systematic Nonresponse Organizational Research Methods, January 1, 2009; 12(1): 113 - 147. [Abstract] [PDF]

				J. M. LeBreton and J. L. Senter Answers to 20 Questions About Interrater Reliability and Interrater Agreement Organizational Research Methods, October 1, 2008; 11(4): 815 - 852. [Abstract] [PDF]

				G. C. Migliaccio, G. E. Gibson Jr., and J. T. O'Connor Changing Project Delivery Strategy: An Implementation Framework Public Works Management Policy, January 1, 2008; 12(3): 483 - 502. [Abstract] [PDF]

				Q. M. Roberson, M. C. Sturman, and T. L. Simons Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes Organizational Research Methods, October 1, 2007; 10(4): 564 - 588. [Abstract] [PDF]

				A. W. Meade and L. T. Eby Using Indices of Group Agreement in Multilevel Construct Validation Organizational Research Methods, January 1, 2007; 10(1): 75 - 96. [Abstract] [PDF]

				A Kouvonen, M Kivimaki, M Elovainio, J Pentti, A Linna, M Virtanen, and J Vahtera Effort/reward imbalance and sedentary lifestyle: an observational study in a large occupational cohort Occup. Environ. Med., June 1, 2006; 63(6): 422 - 427. [Abstract] [Full Text] [PDF]

				R. D. Brown and N. M. A. Hauenstein Interrater Agreement Reconsidered: An Alternative to the rwg Indices Organizational Research Methods, April 1, 2005; 8(2): 165 - 184. [Abstract] [PDF]

				K. Smith-Crowe and M. J. Burke Interpreting the Statistical Significance of Observed AD Interrater Agreement Values: Correction to Burke and Dunlap (2002) Organizational Research Methods, January 1, 2003; 6(1): 129 - 131. [Abstract] [PDF]