This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Roberson, Q. M. Articles by Simons, T. L. Search for Related Content Social Bookmarking                         What's this?

Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes

Cornell University

Michael C. Sturman

Cornell University

Tony L. Simons

Cornell University

Within the context of climate strength, this simulation study examines the validity of various dispersion indexes for detecting meaningful relationships between variability in group member perceptions and outcome variables. We used the simulation to model both individual-and group-level phenomena, vary appropriate population characteristics, and test the proclivity of standard and average deviation, interrater agreement indexes (rwg, r*wg, awg), and coefficient of variation (both normed and unnormed) for Type I and Type II errors. The results show that the coefficient of variation was less likely to detect interaction effects although it outperformed other measures when detecting level effects. Standard deviation was shown to be inferior to other indexes when no level effect is present although it may be an effective measure of dispersion when modeling strength or interaction effects. The implications for future research, in which dispersion is a critical component of the theoretical model, are discussed.

Key Words: multilevel • climate strength • dispersion • agreement

References

• Allison, P.D. (1978). Measures of inequality . American Sociological Review, 43, 865-880 .[CrossRef][Web of Science]
• Bedeian, A.G., & Mossholder, K.W. (2000). On the use of the coefficient of variation as a measure of diversity . Organizational Research Methods, 3, 285-297 .[Abstract]
• Bliese, P.D. (1998). Group size, ICC values, and group-level correlations: A simulation . Organizational Research Methods, 1, 355-373 .[Abstract]
• Bliese, P.D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. Kozlowski (Eds.), Multilevel theory, research and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass .
• Brown, R.D., & Hauenstein, N.M.A. (2005). Interrater agreement reconsidered: An alternative to the rwg indices . Organizational Research Methods, 8, 165-184 .[Abstract]
• Burke, M.J., & Dunlap, W.P. (2002). Estimating interrater agreement with the average deviation index: A user's guide . Organizational Research Methods, 5, 159-172 .[Abstract]
• Burke, M.J., Finkelstein, L.M., & Dusig, M.S. (1999). On average deviation indices for estimating interrater agreement . Organizational Research Methods, 2, 49-68 .[Abstract]
• Castro, S.L. (2002). Data analytic methods for the analysis of multilevel questions: A comparison of intraclass correlation coefficients, rwg(j), hierarchical linear modeling, within- and between-analysis, and random group resampling . Leadership Quarterly, 13, 69-93 .[CrossRef][Web of Science]
• Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models . Journal of Applied Psychology, 83, 234-246 .[CrossRef][Web of Science]
• Cohen, A., Dovey, E., & Eick, U. (2001). Statistical properties of the rwg(j) index of agreement . Psychological Methods, 6, 297-310 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
• Cohen, J. (1960). A coefficient of agreement for nominal scales . Educational and Psychological Measurement, 20, 37-46 .[Medline] [Order article via Infotrieve]
• Cohen, J. (1992). A power primer . Psychological Bulletin, 112, 155-159 .[CrossRef][Medline] [Order article via Infotrieve]
• Colquitt, J.A., Noe, R.A., & Jackson, C.L. (2002). Justice in teams: Antecedents and Consequences of Procedural Justice Climate . Personnel Psychology, 58, 83-109 .
• Colquitt, J.A., Zapata-Phelan, C., & Roberson, Q. (2005). Justice in teams: A review of fairness effects in collective contexts . In J. J. Martocchio (Ed.), Literature review and agenda for future research . Research in personnel and human resource management (Vol. 24, pp. 53-94 ). Oxford , UK: Elsevier .[CrossRef]
• Conway, L.G., III, & Schaller, M. (1998). Methods for the measurement of consensual beliefs within groups . Group Dynamics: Theory, Research and Practice, 2, 241-252 .[CrossRef]
• Gonzalez-Roma, V., Peiro, J.M., & Tordera, N. (2002). An examination of the antecedents and moderator influences of climate strength . Journal of Applied Psychology, 87, 465-473 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
• James, L.R., Demaree, R.G., & Wolf, G. (1984). Estimating within group interrater reliability with and without response bias . Journal of Applied Psychology, 69, 85-98 .[CrossRef][Web of Science]
• James, L.R., Demaree, R.G., & Wolf, G. (1993). rwg: an assessment of within-group agreement . Journal of Applied Psychology, 78, 306-309 .[CrossRef][Web of Science]
• Klein, K.J., & Kozlowski, S.W. (2000). From micro to meso: Critical steps in conceptualizing and conducting multilevel research . Organizational Research Methods, 3, 211-236 .[Abstract]
• Kozlowski, S.W.J., & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus . Journal of Applied Psychology, 77, 161-167 .[CrossRef][Web of Science]
• Kozlowski, S.W.J., & Klein, K.J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal and emergent processes. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research and methods in organizations: Foundations, extensions, and new directions (pp. 3-90). San Francisco: Jossey-Bass .
• LeBreton, J.M., James, L.R., & Lindell, M.K. (2005). Recent issues regarding rwg, r*wg, rwg(j), and r*wg(j) . Organizational Research Methods, 8, 128-138 .[Abstract]
• Lindell, M.K., & Brandt, C.J. (2000). Climate quality and climate consensus as mediators of the relationship between organizational antecedents and outcomes . Journal of Applied Psychology, 85, 331-348 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
• Lindell, M.K., Brandt, C.J., & Whitney, D.J. (1999). A revised index of interrater agreement for multi-item ratings of a single target . Applied Psychological Measurement, 23, 127-135 .[Abstract/Free Full Text]
• Martin, J.D., & Gray, L.N. (1971). Measurement of relative variation: Sociological examples . American Sociological Review, 36, 496-502 .[CrossRef][Web of Science]
• Microsoft Corporation. (1998). Microsoft visual basic 6.0 [Computer software]. Redmond, WA: Author .
• Mischel, W. (1973). Toward a cognitive social learning reconceptualization of personality . Psychological Review, 80, 252-283 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
• Morgeson, F.P., & Hoffmann, D.A. (1999). The structure and function of collective constructs: Implications for multilevel research and theory development . Academy of Management Review, 8, 547-558 .[CrossRef]
• Nunnally, J., & Bernstein, I. (1994). Psychometric Theory (3rd edition). New York: McGraw Hill .
• Schmidt, F.L., & Hunter, J.E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated . Journal of Applied Psychology, 74, 368-370 .[CrossRef][Web of Science]
• Schneider, B., & Reichers, A.E. (1983). On the etiology of climates . Personnel Psychology, 36, 19-39 .[CrossRef][Web of Science]
• Schneider, B., Salvaggio, A., & Subirats, M. (2002). Climate strength: A new direction for climate research . Journal of Applied Psychology, 87, 220-229 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
• Schriesheim, C.A., Cogliser, C.C., & Neider, L.L. (1995). Is it "trustworthy"? A multiple levels-of-analysis reexamination of an Ohio State leadership study, with implications for future research . Leadership Quarterly, 6, 111-145 .[CrossRef]
• Smithson, M. (1982). On relative dispersion: A new solution for some old problems . Quality and Quantity, 16, 261-271 .
• Stigler, S.M. (1973). Studies in the history of probability and statistics XXXII: Laplace, Fisher and the discovery of the concept of sufficiency . Biometrika, 60, 439-445 .[Abstract/Free Full Text]
• Sturman, M.C. (2004). DataSim [Computer software]. Retrieved November 15, 2005, from http://www.people.cornell.edu/pages/mcs5/Pages/DataSimPage.htm

This version was published on October 1, 2007

Organizational Research Methods, Vol. 10, No. 4, 564-588 (2007)
DOI: 10.1177/1094428106294746


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Roberson, Q. M. Articles by Simons, T. L. Search for Related Content Social Bookmarking                         What's this?