Organizational Research Methods

 

Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Register here to gain access to SAGE's 500+ Journals Online

Click here to sign up for SAGE Journal Email Alerts today!

Sign In to gain access to subscriptions and/or personal tools.
This Article
Right arrow Full Text (PDF)
Right arrow References
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via ISI Web of Science (6)
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Brown, R. D.
Right arrow Articles by Hauenstein, N. M. A.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?
Organizational Research Methods, Vol. 8, No. 2, 165-184 (2005)
DOI: 10.1177/1094428105275376

Interrater Agreement Reconsidered: An Alternative to the rwg Indices

Reagan D. Brown

Western Kentucky University

Neil M. A. Hauenstein

Virginia Polytechnic Institute and State University

For continuous constructs, the most frequently used index of interrater agreement (r wg(1))can be problematic. Typically, rwg(1) is estimated with the assumption that a uniform distribution represents no agreement. The authors review the limitations of this uniform nullr wg(1) index and discuss alternative methods for measuring interrater agreement. A new interrater agreement statistic,a wg(1),is proposed. The authors derive thea wg(1)statistic and demonstrate thatawg(1) is an analogue to Cohen’s kappa, an interrater agreement index for nominal data. A comparison is made between agreement estimates based on the uniformr wg(1)and a wg(1), and issues such as minimum sample size and practical significance levels are discussed. The authors close with recommendations regarding the use ofr wg(1)/rwg(J) when a uniform null is assumed,r wg(1)/rwg(J) indices that do not assume a uniform null,awg(1) / a wg(J)indices, and generalizability estimates of interrater agreement.

Key Words: interrater agreement • interrater reliability • kappa


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?


This article has been cited by other articles:


Home page
 Organizational Research MethodsHome page
J. M. LeBreton and J. L. Senter
Answers to 20 Questions About Interrater Reliability and Interrater Agreement
Organizational Research Methods, October 1, 2008; 11(4): 815 - 852.
[Abstract] [PDF]

Home page
 Organizational Research MethodsHome page
Q. M. Roberson, M. C. Sturman, and T. L. Simons
Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes
Organizational Research Methods, October 1, 2007; 10(4): 564 - 588.
[Abstract] [PDF]

Home page
 Journal of Cross-Cultural PsychologyHome page
R. Fischer, P. B. Smith, B. Richey, M. C. Ferreira, E. M. L. Assmar, J. Maes, and S. Stumpf
How Do Organizations Allocate Rewards? The Predictive Validity of National Values, Economic and Organizational Factors Across Six Nations
Journal of Cross-Cultural Psychology, January 1, 2007; 38(1): 3 - 18.
[Abstract] [PDF]

Home page
 Organizational Research MethodsHome page
C. E. Lance, M. M. Butts, and L. C. Michels
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Organizational Research Methods, April 1, 2006; 9(2): 202 - 220.
[Abstract] [PDF]