Organizational Research Methods

 

Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Register here to gain access to SAGE's 500+ Journals Online

Click here to sign up for SAGE Journal Email Alerts today!

Sign In to gain access to subscriptions and/or personal tools.
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Roberson, Q. M.
Right arrow Articles by Simons, T. L.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?
This version was published on October 1, 2007
Organizational Research Methods, Vol. 10, No. 4, 564-588 (2007)
DOI: 10.1177/1094428106294746

Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes

Quinetta M. Roberson

Cornell University

Michael C. Sturman

Cornell University

Tony L. Simons

Cornell University

Within the context of climate strength, this simulation study examines the validity of various dispersion indexes for detecting meaningful relationships between variability in group member perceptions and outcome variables. We used the simulation to model both individual-and group-level phenomena, vary appropriate population characteristics, and test the proclivity of standard and average deviation, interrater agreement indexes (rwg, r*wg, awg), and coefficient of variation (both normed and unnormed) for Type I and Type II errors. The results show that the coefficient of variation was less likely to detect interaction effects although it outperformed other measures when detecting level effects. Standard deviation was shown to be inferior to other indexes when no level effect is present although it may be an effective measure of dispersion when modeling strength or interaction effects. The implications for future research, in which dispersion is a critical component of the theoretical model, are discussed.

Key Words: multilevel • climate strength • dispersion • agreement

References

  • Allison, P.D. (1978). Measures of inequality . American Sociological Review, 43, 865-880 .[CrossRef][ISI]
  • Bedeian, A.G., & Mossholder, K.W. (2000). On the use of the coefficient of variation as a measure of diversity . Organizational Research Methods, 3, 285-297 .[Abstract]
  • Bliese, P.D. (1998). Group size, ICC values, and group-level correlations: A simulation . Organizational Research Methods, 1, 355-373 .[Abstract]
  • Bliese, P.D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. Kozlowski (Eds.), Multilevel theory, research and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass .
  • Brown, R.D., & Hauenstein, N.M.A. (2005). Interrater agreement reconsidered: An alternative to the rwg indices . Organizational Research Methods, 8, 165-184 .[Abstract]
  • Burke, M.J., & Dunlap, W.P. (2002). Estimating interrater agreement with the average deviation index: A user's guide . Organizational Research Methods, 5, 159-172 .[Abstract]
  • Burke, M.J., Finkelstein, L.M., & Dusig, M.S. (1999). On average deviation indices for estimating interrater agreement . Organizational Research Methods, 2, 49-68 .[Abstract]
  • Castro, S.L. (2002). Data analytic methods for the analysis of multilevel questions: A comparison of intraclass correlation coefficients, rwg(j), hierarchical linear modeling, within- and between-analysis, and random group resampling . Leadership Quarterly, 13, 69-93 .[CrossRef][ISI]
  • Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models . Journal of Applied Psychology, 83, 234-246 .[CrossRef][ISI]
  • Cohen, A., Dovey, E., & Eick, U. (2001). Statistical properties of the rwg(j) index of agreement . Psychological Methods, 6, 297-310 .[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Cohen, J. (1960). A coefficient of agreement for nominal scales . Educational and Psychological Measurement, 20, 37-46 .[Medline] [Order article via Infotrieve]
  • Cohen, J. (1992). A power primer . Psychological Bulletin, 112, 155-159 .[CrossRef][ISI]
  • Colquitt, J.A., Noe, R.A., & Jackson, C.L. (2002). Justice in teams: Antecedents and Consequences of Procedural Justice Climate . Personnel Psychology, 58, 83-109 .
  • Colquitt, J.A., Zapata-Phelan, C., & Roberson, Q. (2005). Justice in teams: A review of fairness effects in collective contexts . In J. J. Martocchio (Ed.), Literature review and agenda for future research . Research in personnel and human resource management (Vol. 24, pp. 53-94 ). Oxford , UK: Elsevier .[CrossRef]
  • Conway, L.G., III, & Schaller, M. (1998). Methods for the measurement of consensual beliefs within groups . Group Dynamics: Theory, Research and Practice, 2, 241-252 .[CrossRef]
  • Gonzalez-Roma, V., Peiro, J.M., & Tordera, N. (2002). An examination of the antecedents and moderator influences of climate strength . Journal of Applied Psychology, 87, 465-473 .[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • James, L.R., Demaree, R.G., & Wolf, G. (1984). Estimating within group interrater reliability with and without response bias . Journal of Applied Psychology, 69, 85-98 .[CrossRef][ISI]
  • James, L.R., Demaree, R.G., & Wolf, G. (1993). rwg: an assessment of within-group agreement . Journal of Applied Psychology, 78, 306-309 .[CrossRef][ISI]
  • Klein, K.J., & Kozlowski, S.W. (2000). From micro to meso: Critical steps in conceptualizing and conducting multilevel research . Organizational Research Methods, 3, 211-236 .[Abstract]
  • Kozlowski, S.W.J., & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus . Journal of Applied Psychology, 77, 161-167 .[CrossRef][ISI]
  • Kozlowski, S.W.J., & Klein, K.J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal and emergent processes. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research and methods in organizations: Foundations, extensions, and new directions (pp. 3-90). San Francisco: Jossey-Bass .
  • LeBreton, J.M., James, L.R., & Lindell, M.K. (2005). Recent issues regarding rwg, r*wg, rwg(j), and r*wg(j) . Organizational Research Methods, 8, 128-138 .[Abstract]
  • Lindell, M.K., & Brandt, C.J. (2000). Climate quality and climate consensus as mediators of the relationship between organizational antecedents and outcomes . Journal of Applied Psychology, 85, 331-348 .[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Lindell, M.K., Brandt, C.J., & Whitney, D.J. (1999). A revised index of interrater agreement for multi-item ratings of a single target . Applied Psychological Measurement, 23, 127-135 .[Abstract/Free Full Text]
  • Martin, J.D., & Gray, L.N. (1971). Measurement of relative variation: Sociological examples . American Sociological Review, 36, 496-502 .[CrossRef][ISI]
  • Microsoft Corporation. (1998). Microsoft visual basic 6.0 [Computer software]. Redmond, WA: Author .
  • Mischel, W. (1973). Toward a cognitive social learning reconceptualization of personality . Psychological Review, 80, 252-283 .[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Morgeson, F.P., & Hoffmann, D.A. (1999). The structure and function of collective constructs: Implications for multilevel research and theory development . Academy of Management Review, 8, 547-558 .[CrossRef]
  • Nunnally, J., & Bernstein, I. (1994). Psychometric Theory (3rd edition). New York: McGraw Hill .
  • Schmidt, F.L., & Hunter, J.E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated . Journal of Applied Psychology, 74, 368-370 .[CrossRef][ISI]
  • Schneider, B., & Reichers, A.E. (1983). On the etiology of climates . Personnel Psychology, 36, 19-39 .[CrossRef][ISI]
  • Schneider, B., Salvaggio, A., & Subirats, M. (2002). Climate strength: A new direction for climate research . Journal of Applied Psychology, 87, 220-229 .[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Schriesheim, C.A., Cogliser, C.C., & Neider, L.L. (1995). Is it "trustworthy"? A multiple levels-of-analysis reexamination of an Ohio State leadership study, with implications for future research . Leadership Quarterly, 6, 111-145 .[CrossRef]
  • Smithson, M. (1982). On relative dispersion: A new solution for some old problems . Quality and Quantity, 16, 261-271 .
  • Stigler, S.M. (1973). Studies in the history of probability and statistics XXXII: Laplace, Fisher and the discovery of the concept of sufficiency . Biometrika, 60, 439-445 .[Abstract/Free Full Text]
  • Sturman, M.C. (2004). DataSim [Computer software]. Retrieved November 15, 2005, from http://www.people.cornell.edu/pages/mcs5/Pages/DataSimPage.htm

Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?



This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Roberson, Q. M.
Right arrow Articles by Simons, T. L.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?