This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Burke, M. J. Articles by Dusig, M. S. Search for Related Content Social Bookmarking                         What's this?

On Average Deviation Indices for Estimating Interrater Agreement

Tulane University

Lisa M. Finkelstein

Northern Illinois University

Michelle S. Dusig

Tulane University

In this investigation, the authors report the results of two studies designed to investigate the efficacy of two proposed indices of interrater agreement based on average deviations from the mean and from the median (ADM and ADMd, respectively). Using survey response data collected from 6,549 sales employees in 119 stores of a national retail company, Study 1 compared the results of six interrater agreement indices across four types of Likert-type response scales (i.e., 5-, 6-, 7-, and 11-point scales). The results indicated that the AD indices were highly correlated with an index of proportional agreement and with within-group interrater agreement indices. Study 2, based on survey data collected from 4,158 sales employees in 109 other stores of this company, constructively replicated Study 1 and examined the consistency of interrater agreement decisions across six indices with respect to a priori decision rules. Study 2 results also supported the use of AD indices. Practical issues concerning the use of AD indices for estimating interrater agreement and future research directions are discussed.

Organizational Research Methods, Vol. 2, No. 1, 49-68 (1999)
DOI: 10.1177/109442819921004


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				N. de Cuyper, B. Sora, H. de Witte, A. Caballer, and J. M. Peiro Organizations' Use of Temporary Employment and a Climate of Job Insecurity among Belgian and Spanish Permanent Workers Economic and Industrial Democracy, November 1, 2009; 30(4): 564 - 591. [Abstract] [PDF]

				X. Dumay Origins and Consequences of Schools' Organizational Culture for Student Achievement Educational Administration Quarterly, October 1, 2009; 45(4): 523 - 555. [Abstract] [PDF]

				O. Ludtke and A. Robitzsch Assessing Within-Group Agreement: A Critical Examination of a Random-Group Resampling Approach Organizational Research Methods, July 1, 2009; 12(3): 461 - 487. [Abstract] [PDF]

				A. Cohen, E. Doveh, and I. Nahum-Shani Testing Agreement for Multi-Item Scales With the IndicesrWG(J) and AD M(J) Organizational Research Methods, January 1, 2009; 12(1): 148 - 164. [Abstract] [PDF]

				D. A. Newman and H.-P. Sin How Do Missing Data Bias Estimates of Within-Group Agreement? Sensitivity of SD WG, CVWG, rWG(J), rWG(J) * , and ICC to Systematic Nonresponse Organizational Research Methods, January 1, 2009; 12(1): 113 - 147. [Abstract] [PDF]

				J. M. LeBreton and J. L. Senter Answers to 20 Questions About Interrater Reliability and Interrater Agreement Organizational Research Methods, October 1, 2008; 11(4): 815 - 852. [Abstract] [PDF]

				Q. M. Roberson, M. C. Sturman, and T. L. Simons Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes Organizational Research Methods, October 1, 2007; 10(4): 564 - 588. [Abstract] [PDF]

				A. W. Meade and L. T. Eby Using Indices of Group Agreement in Multilevel Construct Validation Organizational Research Methods, January 1, 2007; 10(1): 75 - 96. [Abstract] [PDF]

				R. D. Brown and N. M. A. Hauenstein Interrater Agreement Reconsidered: An Alternative to the rwg Indices Organizational Research Methods, April 1, 2005; 8(2): 165 - 184. [Abstract] [PDF]

				M. J. Burke and W. P. Dunlap Estimating Interrater Agreement with the Average Deviation Index: A User's Guide Organizational Research Methods, April 1, 2002; 5(2): 159 - 172. [Abstract] [PDF]