| Sign In to gain access to subscriptions and/or personal tools. |
Assessing Within-Group AgreementA Critical Examination of a Random-Group Resampling ApproachMax Planck Institute for Human Development, Berlin, luedtke{at}mpib-berlin.mpg.de
Institute for Educational Progress, Berlin The measure of within-group agreement most frequently encountered in organizational psychology is the rWG index. The rWG index is determined by comparing the observed group variance among raters with an expected random variance. The most critical issue in calculating the rWG is the choice of an appropriate random distribution that would be expected to follow from raters making their ratings at random. A data-driven approach that uses random-group resampling (RGR) procedures to determine the expected random variance has been proposed. In the present study, the application of the RGR procedure will be illustrated with reference to students' ratings of their mathematics instruction and critically compared with a recently proposed simulation-based approach. It will be shown mathematically that the probability of obtaining statistically significant within-group agreement when applying the RGR procedure strongly depends on the intraclass correlation as well as on the group sizes. Finally, implications for applying the RGR procedure to assess within-group agreement in multilevel data will be discussed.
Key Words: within-group agreement • intraclass correlation • levels of analysis • multilevel analysis
This version was published on July
1, 2009 Organizational Research Methods, Vol. 12, No. 3,
461-487 (2009) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
|
 |
|
Sign In to gain access to subscriptions and/or personal tools.
