Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

CiteULike is a free service for managing and discovering scholarly references - click here to get started.

Sign In to gain access to subscriptions and/or personal tools.
Organizational Research Methods
This Article
Right arrow Full Text (PDF)
Right arrow All Versions of this Article:
1094428108317406v1
12/3/461    most recent
Right arrow References
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Lüdtke, O.
Right arrow Articles by Robitzsch, A.
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?

Assessing Within-Group Agreement

A Critical Examination of a Random-Group Resampling Approach

Oliver Lüdtke

Max Planck Institute for Human Development, Berlin, luedtke{at}mpib-berlin.mpg.de

Alexander Robitzsch

Institute for Educational Progress, Berlin

The measure of within-group agreement most frequently encountered in organizational psychology is the rWG index. The rWG index is determined by comparing the observed group variance among raters with an expected random variance. The most critical issue in calculating the rWG is the choice of an appropriate random distribution that would be expected to follow from raters making their ratings at random. A data-driven approach that uses random-group resampling (RGR) procedures to determine the expected random variance has been proposed. In the present study, the application of the RGR procedure will be illustrated with reference to students' ratings of their mathematics instruction and critically compared with a recently proposed simulation-based approach. It will be shown mathematically that the probability of obtaining statistically significant within-group agreement when applying the RGR procedure strongly depends on the intraclass correlation as well as on the group sizes. Finally, implications for applying the RGR procedure to assess within-group agreement in multilevel data will be discussed.

Key Words: within-group agreement • intraclass correlation • levels of analysis • multilevel analysis

This version was published on July 1, 2009

Organizational Research Methods, Vol. 12, No. 3, 461-487 (2009)
DOI: 10.1177/1094428108317406
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati   Add to Twitter Twitter    What's this?