Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Click here to sign up for SAGE Journal Email Alerts today!

Sign In to gain access to subscriptions and/or personal tools.
Organizational Research Methods
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via Web of Science (3)
Right arrow Citing Articles via Google Scholar
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Bobko, P.
Right arrow Articles by Buster, M. A.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?

The Usefulness of Unit Weights in Creating Composite Scores

A Literature Review, Application to Content Validity, and Meta-Analysis

Philip Bobko

Gettysburg College, pbobko{at}gettysburg.edu

Philip L. Roth

Clemson University

Maury A. Buster

Alabama State Personnel Department

Combining/weighting subscores into an aggregate score involves issues that apply to many fields in the organizational sciences (e.g., weighting predictors in selection, weighting multiple performance appraisal indicators, overall evaluation of organizations). The weights that are used in practice can be different (differential weights) or equal (unit weights). Relevant literature across multiple disciplines and multiple decades is reviewed. The literature indicates that unit weights have substantial predictive validity when compared with regression weights, but there is a lack of data on how other differential weighting strategies (e.g., weights generated by subject matter experts) compare to unit weights. In response, a primary and a meta-analytic study are provided here. The recent literature also contains some potential criticisms of unit weights in regard to personnel selection and content validation—and those statements are evaluated. The data and findings indicate that unit weights can be a highly appropriate approach for weighting under many circumstances.

Key Words: unit weights • selection • composites • content validity

References

  • *An asterisk denotes that data from these studies were used in our meta-analysis.
  • *Aamodt, M., & Kimbrough, W. (1985). Comparison of four methods for weighting multiple predictors. Educational and Psychological Measurement, 45, 477-482.[Web of Science]
  • *Arthur, W., Doverspike, D., & Barrett, G. (1996). Development of a job analysis-based procedure for weighting and combining content-related tests into a single test battery score. Personnel Psychology, 49, 971-985.[CrossRef][Web of Science]
  • Biddle, D. (2005). Adverse impact and test validation: A practitioner's guide to valid and defensible employment testing. Burlington, VT: Gower.
  • Bobko, P. (2001). Correlation and regression (Rev. ed.). Thousand Oaks, CA: Sage.
  • Bobko, P., Roth, P., & Bobko, C. (2001). Correcting the effect size of d for range restriction and unreliability. Organizational Research Methods, 4, 46-61.[Medline] [Order article via Infotrieve]
  • Bobko, P., & Schemmer, M. (1980). A note on standardized regression estimators. Psychological Bulletin, 88, 233-236.[CrossRef][Web of Science]
  • Boreham, N., Russell, C., & Wastell, D. (1988). Can past academic criteria predict students at risk of future failure? Medical Education, 22, 94-98.[Web of Science][Medline] [Order article via Infotrieve]
  • Cascio, W. (1987). Costing human resources: The financial impact of behavior in organizations (2nd ed.). Boston: PWS-Kent.
  • Claudy, J. (1972). A comparison of five variable weighting procedures. Educational and Psychological Measurement, 32, 311-322.[CrossRef][Web of Science]
  • Clements, D., & Reid, R. (1994). Analytical MS/OR tools applied to a plant closure. Interfaces, 24, 1-12.[Medline] [Order article via Infotrieve]
  • Cohen, J. (1990). Things I have learned (so far). American Psychologist, 12, 1304-1312.
  • Cohen, J., West, S., Aiken, L., & Cohen, P. (2002). Applied multivariate regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
  • Dawes, R.M. (1974). The robust beauty of improper linear modes in decision making. American Psychologist, 34, 571-582.[CrossRef]
  • Dawes, R.M., & Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81, 95-106.[CrossRef][Web of Science]
  • Dorans, N., & Drasgrow, F. (1978). Alternative weighting schemes for linear prediction. Organizational Behavior and Human Performance, 21, 316-345.[CrossRef][Web of Science]
  • Doverspike, D., Winter, J., Healy, M., & Barrett, G. (1996). Simulations as a method of illustrating the impact of differential weights on personnel selection outcomes. Human Performance, 9, 259-273.[CrossRef][Web of Science]
  • Einhorn, H., & Hogarth, R. (1975). Unit weighting schemes for decision making. Organizational Behavior and Human Performance, 13, 171-192.[CrossRef][Web of Science]
  • *Fralicz, R., & Raju, N. (1982). A comparison of five methods for combining multiple criteria into a single composite. Educational and Psychological Measurement, 42, 823-827.[Abstract]
  • Gatewood, R., & Feild, H. (2001). Human resource selection (5th ed.). New York: Harcourt.
  • *Goldstein, H., Rumison, K., & Yusko, K. (2002, April). The impact of test composite score weighting procedures on Black-White subgroup differences. Paper presented at the annual meeting of the Society for Industrial & Organizational Psychology, Toronto, Canada.
  • Goldstein, H., Yusko, K., Braverman, E., Smith, D., & Chung, B. (1998). The role of cognitive ability in the subgroup differences and incremental validity of assessment center exercises. Personnel Psychology, 51, 357-374.[CrossRef][Web of Science]
  • Goldstein, I., Zedeck, S., & Schneider, B. (1993). An exploration of the job analysis-content validity process. In N. Schmitt & W. Borman (Eds.), Personnel selection in organizations (pp. 1-34). San Francisco: Jossey-Bass.
  • Grant, D., & Bray, D. (1970). Validation of employment tests for telephone company installation and repair occupations. Journal of Applied Psychology, 54, 7-14.[CrossRef][Web of Science]
  • Guion, R. (1991). Personnel assessment, selection, and placement. In M. Dunnette & L. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 2, pp. 327-397). Palo Alto, CA: Consulting Psychologists Press.
  • Hausknecht, J., Day, D., & Thomas, S. (2004). Applicant reactions to selection procedures: An updated model and meta-analysis. Personnel Psychology, 57, 639-683.[CrossRef][Web of Science]
  • Janz, T., Hellervik, L., & Gilmore, D. (1986). Behavior description interviewing: New, accurate, cost effective. Boston: Allyn & Bacon.
  • Laughlin, J. (1978). Comment on "estimating coefficients in linear models: It don't make no nevermind." Psychological Bulletin, 85, 247-253.[CrossRef][Web of Science]
  • Lawshe, C. (1969). Statistical theory and practice in applied psychology. Personnel Psychology, 22, 117-123.[CrossRef][Web of Science]
  • McCall, M., & Bobko, P. (1990). Research methods and discovery in industrial/organizational psychology. In M. Dunnette & L. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 1, pp. 381-418). Palo Alto, CA: Consulting Psychologists Press.
  • Meehl, P. (1965). Seer over sign: The first good example. Journal of Experimental Research in Personality, 1, 27-32.[Web of Science]
  • *O'Leary, R., O'Leary, B., & MacLane, C. (2004, April). Weighting content valid composites: A cautionary note on unit weights. Paper presented at the nineteenth annual meeting of the Society for Industrial & Organizational Psychology, Chicago.
  • Raju, N., Bilgic, R., Edwards, J., & Fleer, P. (1997). Methodology review: Estimation of population validity and cross validity, and the use of equal weights in prediction. Applied Psychological Measurement, 21, 291-305.[Abstract/Free Full Text]
  • Raju, N., Bilgic, R., Edwards, J., & Fleer, P. (1999). Accuracy of population validity and cross-validity estimation: An empirical comparison of formula-based, traditional empirical, and equal weights procedures. Applied Psychological Measurement, 23, 99-115.[Abstract/Free Full Text]
  • Ree, M., Carretta, T., & Earles, J. (1998). In top-down decisions, weighting variables does not matter: A consequence of Wilk's theorem. Organizational Research Methods, 1, 407-420.[Abstract]
  • Reynolds v. Alabama Department of Transportation, No. 85-665 (Memorandum Opinion & Order) (M. D. Ala. December 19, 2003).
  • Rosenberg, N., Skordahl, D., & Kaplan, H. (1961a, May). Validation of fixed-wing aptitude battery against success in Army flight training. Human Factors Research Branch Technical Research Note 112, Adjutant General's Office (Army), Washington, DC.
  • Rosenberg, N., Skordahl, D., & Kaplan, H. (May, 1961b). Validation of fixed-wing aptitude battery against success in ROTC flight training. Human Factors Research Branch Technical Research Note 111; Army Project 2L95-60-001, Adjutant General's Office (Army), Washington, DC.
  • Roth, P., & Switzer, F. (2002). Outliers and influential cases: Handling those discordant contaminated maverick rogues. In S. Rogelberg (Ed.), Handbook of methods in industrial and organizational psychology (pp. 297-309). Malden, MA: Blackwell.
  • Schmidt, F. (1971). The relative efficiency of regression and simple unit weight predictor weights in applied differential psychology. Educational and Psychological Measurement, 31, 699-714.[CrossRef][Web of Science]
  • Schmidt, F., Hunter, J., & Urry, V. (1976). Statistical power in criterion-related validity studies. Journal of Applied Psychology, 61, 473-485.[CrossRef][Web of Science]
  • Schmidt, F., & Le, H. (2004). Software for the Hunter-Schmidt meta-analysis methods. Iowa City: University of Iowa, Department of Management & Organization.
  • Seashore, H. (1957). Watch your weights. Test Service Bulletin, 52, 2-5.
  • Stillwell, W., Barron, F., & Edwards, W. (1983). Evaluating credit applications: A validation of multiattribute utility weight elicitation techniques. Organizational Behavior and Human Performance, 32, 87-108.[CrossRef][Web of Science]
  • Trattner, M. (1963). Comparison of three methods for assembling aptitude test batteries. Personnel Psychology, 16, 221-232.[CrossRef][Web of Science]
  • Wainer, H. (1976). Estimating coefficients in linear models: It don't make no nevermind. Psychological Bulletin, 83, 213-217.[CrossRef][Web of Science]
  • Wallace, M., & Schwab, D. (1976). A cross-validated comparison of five models used to predict graduate admissions committee decisions. Journal of Applied Psychology, 61, 559-563.[CrossRef][Web of Science]
  • Wang, M., & Stanley, J. (1970). Differential weighting: A review of methods and empirical studies. Review of Educational Research, 40, 663-705.[Free Full Text]
  • Werts, C., & Linn, R. (1970). Path analysis: Psychological examples. Psychological Bulletin, 74, 193-212.[CrossRef][Web of Science]
  • Wilks, S. (1938). Weighting systems for linear functions of correlated variables when there is no dependent variable. Psychometrika, 3, 23-40.[CrossRef]

This version was published on October 1, 2007

Organizational Research Methods, Vol. 10, No. 4, 689-709 (2007)
DOI: 10.1177/1094428106294734
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati   Add to Twitter Twitter    What's this?



This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via Web of Science (3)
Right arrow Citing Articles via Google Scholar
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Bobko, P.
Right arrow Articles by Buster, M. A.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?