This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Web of Science (2) Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Beal, D. J. Articles by Dawson, J. F. Search for Related Content Social Bookmarking                         What's this?

On the Use of Likert-Type Scales in Multilevel Data

Influence on Aggregate Variables

Rice University, dbeal{at}rice.edu

Jeremy F. Dawson

Aston University

In multilevel analyses, problems may arise when using Likert-type scales at the lowest level of analysis. Specifically, increases in variance should lead to greater censoring for the groups whose true scores fall at either end of the distribution. The current study used simulation methods to examine the influence of single-item Likert-type scale usage on ICC(1), ICC(2), and group-level correlations. Results revealed substantial underestimation of ICC(1) when using Likert-type scales with common response formats (e.g., 5 points). ICC(2) and group-level correlations were also underestimated, but to a lesser extent. Finally, the magnitude of underestimation was driven in large part to an interaction between Likert-type scale usage and the amounts of within- and between-group variance.

Key Words: multilevel • aggregation • Likert-type scale • intraclass correlation • Monte Carlo

References

• Bliese, P.D. (1998). Group size, ICC values, and group-level correlations: A simulation. Organizational Research Methods, 1, 355-373.[Abstract]
• Bliese, P.D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analyses. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349—381). San Francisco: Jossey-Bass.
• Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.[CrossRef][Web of Science]
• Chen, G., Bliese, P.D., & Mathieu, J.E. (2005). Conceptual framework and statistical procedures for delineating and testing multilevel theories of homology. Organizational Research Methods, 8, 375-409.[Abstract]
• Cicchetti, D.V., Showalter, D., & Tyrer, P.J. (1985). The effect of number of rating scale categories on levels of interrater reliability: A Monte Carlo investigation. Journal of Applied Psychology, 9, 31-36.
• Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-253.[Free Full Text]
• Gangestad, S., & Snyder, M. (2000). Self-monitoring: Appraisal and reappraisal. Psychological Bulletin, 126, 530-555.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
• George, J.M., & James, L.R. (1993). Personality, affect, and behavior in groups revisited: Comment on aggregation, levels of analysis, and a recent application of within and between analysis. Journal of Applied Psychology, 78, 798-804.[CrossRef][Web of Science]
• Glass, G.V., Peckham, P.D., & Sanders, J.R. (1972). Consequences of failure to meet assumptions underlying the fixed effects analysis of variances and covariances. Review of Educational Research, 42, 237-288.[Free Full Text]
• Gupta, A.K. (1952). Estimation of the mean and standard deviation of a normal population from a censored sample. Biometrika, 39, 260-273.[Free Full Text]
• Hofmann, S.A., Griffin, M.A., & Gavin, M.B. (2000). The application of hierarchical linear modeling to organizational research. In K. J. Klein & S. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 467-511). San Francisco: Jossey-Bass.
• Hox, J.J. (2002). Multilevel analysis. Mahwah, NJ: Lawrence Erlbaum.
• James, L.R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67, 219-229.[CrossRef][Web of Science]
• Jones, A.P., & James, L.R. (1979). Psychological climate: Dimensions and relationships of individual and aggregated work environment perceptions. Organizational Behavior and Human Performance, 23, 201-250.[CrossRef][Web of Science]
• Komorita, S.S., & Graham, W.K. (1965). Number of scale points and the reliability of scales. Educational & Psychological Measurement, 4, 987-995.
• Kotz, S., Balakrishnan, N., & Johnson, N.L. (2000). Continuous multivariate distributions. New York: Wiley-Interscience.
• Krieg, E.F. (1999). Biases induced by course measurement scales. Educational & Psychological Measurement, 59, 749-766.[CrossRef]
• Landy, F.J., & Farr, J.L. (1980). Performance rating. Psychological Bulletin, 87, 72-107.[CrossRef][Web of Science]
• LeBreton, J.M., James, L.R., & Lindell, M.K. (2005). Recent issues regarding rwg, r*wg, rwg(j), and r*wg(j). Organizational Research Methods, 8, 128-138.[Abstract]
• Maddala, G.S. (1983). Limited-dependent and qualitative variables in econometrics. Cambridge, UK: Cambridge University Press.
• Micceri, T. (1989). The Unicorn, the normal curve and other improbable creatures. Psychological Bulletin, 105, 156-166.[CrossRef][Web of Science]
• Muthén, B. (1989). Tobit factor analysis. British Journal of Mathematical and Statistical Psychology, 42, 241-250.
• Pearson, K., & Lee, A. (1908). On the generalized probable error in the multiple normal correlation. Biometrika, 6, 59-68.[Free Full Text]
• Rasmussen, J.L. (1989). Analysis of Likert-scale data: A reinterpretation of Gregoire and Driver. Psychological Bulletin, 105, 167-170.[CrossRef][Web of Science]
• Raudenbush, S.W., & Bryk, A.S. (2001). Hierarchical linear models (2nd ed.). Thousand Oaks, CA: Sage.
• Russell, C.J., & Bobko, P. (1992). Moderated regression analysis and Likert scales: Too coarse for comfort. Journal of Applied Psychology, 77, 336-342.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
• Scheffé, H. (1959). The analysis of variance. New York: John Wiley.
• Shrout, P., & Fleiss, J. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]

This version was published on October 1, 2007

Organizational Research Methods, Vol. 10, No. 4, 657-672 (2007)
DOI: 10.1177/1094428106295492


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Web of Science (2) Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Beal, D. J. Articles by Dawson, J. F. Search for Related Content Social Bookmarking                         What's this?