This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Web of Science (15) Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Brown, R. D. Articles by Hauenstein, N. M. A. Search for Related Content Social Bookmarking                         What's this?

Interrater Agreement Reconsidered: An Alternative to the rwg Indices

Western Kentucky University

Neil M. A. Hauenstein

Virginia Polytechnic Institute and State University

For continuous constructs, the most frequently used index of interrater agreement (r wg(1))can be problematic. Typically, rwg(1) is estimated with the assumption that a uniform distribution represents no agreement. The authors review the limitations of this uniform nullr ^wg(1) index and discuss alternative methods for measuring interrater agreement. A new interrater agreement statistic,a ^wg(1),is proposed. The authors derive thea ^wg(1)statistic and demonstrate thatawg(1) is an analogue to Cohen’s kappa, an interrater agreement index for nominal data. A comparison is made between agreement estimates based on the uniformr ^wg(1)and a wg(1), and issues such as minimum sample size and practical significance levels are discussed. The authors close with recommendations regarding the use ofr ^wg(1)/rwg(J) when a uniform null is assumed,r ^wg(1)/rwg(J) indices that do not assume a uniform null,awg(1) / a ^wg(J)indices, and generalizability estimates of interrater agreement.

Key Words: interrater agreement • interrater reliability • kappa

Organizational Research Methods, Vol. 8, No. 2, 165-184 (2005)
DOI: 10.1177/1094428105275376


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				R. Fischer Where Is Culture in Cross Cultural Research?: An Outline of a Multilevel Research Process for Measuring Culture as a Shared Meaning System International Journal of Cross Cultural Management, April 1, 2009; 9(1): 25 - 49. [Abstract] [PDF]

				R. Fischer, M. C. Ferreira, E. Assmar, P. Redford, C. Harb, S. Glazer, B.-S. Cheng, D.-Y. Jiang, C. C. Wong, N. Kumar, et al. Individualism-collectivism as Descriptive Norms: Development of a Subjective Norm Approach to Culture Measurement Journal of Cross-Cultural Psychology, March 1, 2009; 40(2): 187 - 213. [Abstract] [PDF]

				A. Cohen, E. Doveh, and I. Nahum-Shani Testing Agreement for Multi-Item Scales With the IndicesrWG(J) and AD M(J) Organizational Research Methods, January 1, 2009; 12(1): 148 - 164. [Abstract] [PDF]

				D. A. Newman and H.-P. Sin How Do Missing Data Bias Estimates of Within-Group Agreement? Sensitivity of SD WG, CVWG, rWG(J), rWG(J) * , and ICC to Systematic Nonresponse Organizational Research Methods, January 1, 2009; 12(1): 113 - 147. [Abstract] [PDF]

				J. M. LeBreton and J. L. Senter Answers to 20 Questions About Interrater Reliability and Interrater Agreement Organizational Research Methods, October 1, 2008; 11(4): 815 - 852. [Abstract] [PDF]

				Q. M. Roberson, M. C. Sturman, and T. L. Simons Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes Organizational Research Methods, October 1, 2007; 10(4): 564 - 588. [Abstract] [PDF]

				R. Fischer, P. B. Smith, B. Richey, M. C. Ferreira, E. M. L. Assmar, J. Maes, and S. Stumpf How Do Organizations Allocate Rewards? The Predictive Validity of National Values, Economic and Organizational Factors Across Six Nations Journal of Cross-Cultural Psychology, January 1, 2007; 38(1): 3 - 18. [Abstract] [PDF]

				C. E. Lance, M. M. Butts, and L. C. Michels The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say? Organizational Research Methods, April 1, 2006; 9(2): 202 - 220. [Abstract] [PDF]