Organizational Research Methods recent issues

Scale Coarseness as a Methodological Artifact: Correcting Correlation Coefficients Attenuated From Using Coarse Scales

Aguinis, H., Pierce, C. A., Culpepper, S. A. — Wed, 23 Sep 2009 09:23:46 PDT

Scale coarseness is a pervasive yet ignored methodological artifact that attenuates observed correlation coefficients in relation to population coefficients. The authors describe how to disattenuate correlations that are biased by scale coarseness in primary-level as well as meta-analytic studies and derive the sampling error variance for the corrected correlation. Results of two Monte Carlo simulations reveal that the correction procedure is accurate and show the extent to which coarseness biases the correlation coefficient under various conditions (i.e., value of the population correlation, number of item scale points, and number of scale items). The authors also offer a Web-based computer program that disattenuates correlations at the primary-study level and computes the sampling error variance as well as confidence intervals for the corrected correlation. Using this program, which implements the correction in primary-level studies, and incorporating the suggested correction in meta-analytic reviews will lead to more accurate estimates of construct-level correlation coefficients.

The Neglected Science and Art of Quasi-Experimentation: Why-to, When-to, and How-to Advice for Organizational Researchers

Grant, A. M., Wall, T. D. — Wed, 23 Sep 2009 09:23:46 PDT

Although quasi-experiments can facilitate causal inferences by combining good internal validity with high external validity, organizational scholars underutilize them. In this article, the authors aim to encourage the novel use of quasi-experimentation by identifying five of its key benefits: (a) strengthening causal inference when random assignment and controlled manipulation are not possible or ethical; (b) building better theories of time and temporal progression; (c) minimizing ethical dilemmas of harm, inequity, paternalism, and deception; (d) facilitating collaboration with practitioners; and (e) using context to explain conflicting findings. We offer advice and illustrative examples to guide future research, and provide recommendations for gaining access to organizations to open doors for collaborating on quasi-experiments.

Decimal Dust, Significant Digits, and the Search for Stars

Bedeian, A. G., Sturman, M. C., Streiner, D. L. — Wed, 23 Sep 2009 09:23:46 PDT

The practice of rounding statistical results to two decimal places is one of a large number of heuristics followed in the social sciences. In evaluating this heuristic, the authors conducted simulations to investigate the precision of simple correlations. They considered a true correlation of .15 and ran simulations in which the sample sizes were 60, 100, 200, 500, 1,000, 10,000, and 100,000. They then looked at the digits in the correlations’ first, second, and third decimal places to determine their reproducibility. They conclude that when n < 500, the habit of reporting a result to two decimal places seems unwarranted, and it never makes sense to report the third digit after the decimal place unless one has a sample size larger than 100,000. Similar results were found with rhos of .30, .50, and .70. The results offer an important qualification to what is otherwise a misleading practice.

Testing Multilevel Mediation Using Hierarchical Linear Models: Problems and Solutions

Zhang, Z., Zyphur, M. J., Preacher, K. J. — Wed, 23 Sep 2009 09:23:46 PDT

Testing multilevel mediation using hierarchical linear modeling (HLM) has gained tremendous popularity in recent years. However, potential confounding in multilevel mediation effect estimates can arise in these models when within-group effects differ from between-group effects. This study summarizes three types of HLM-based multilevel mediation models, and then explains that in two types of these models confounding can be produced and erroneous conclusions may be derived when using popularly recommended procedures. A Monte Carlo simulation study illustrates that these procedures can underestimate or overestimate true mediation effects. Recommendations are provided for appropriately testing multilevel mediation and for differentiating within-group versus between-group effects in multilevel settings.

A Proposal for Operationalizing the Pace and Scope of Organizational Change in Management Studies

Street, C. T., Gallupe, R. B. — Wed, 23 Sep 2009 09:23:46 PDT

Organizational change is an important construct for management theorists, yet organizational research is being hampered by inconsistent and incompatible operationalizations of the construct. This article presents a proposal for improving clarity about how the types and characteristics of organizational change can be operationalized and measured. In particular, the scope and pace of organizational change are examined and a common approach is developed to measure the impacts of these two factors on patterns of organizational change.

Test Bias, Differential Prediction, and a Revised Approach for Determining the Suitability of a Predictor in a Selection Context

Meade, A. W., Fetzer, M. — Wed, 23 Sep 2009 09:23:46 PDT

The most commonly used and accepted model of assessing bias in a selection context is that proposed by Cleary in which predictor-criterion regression lines are tested for both slope and intercept equality. With this approach, any difference in intercepts or slopes is considered an indication of bias. We argue that differing regression lines intercepts is indicative of differential prediction but not test bias. We describe several fundamentally different potential causes of differences in groups’ regression line intercepts, many of which are unrelated to test properties. We argue that differential prediction because of such sources should not preclude the use of the test in selection contexts. We propose a new procedure to potentially identify the source of regression line differences and illustrate this framework using a job incumbent sample.

A Tale of Three Perspectives: Examining Post Hoc Statistical Techniques for Detection and Correction of Common Method Variance

Richardson, H. A., Simmering, M. J., Sturman, M. C. — Wed, 23 Sep 2009 09:23:46 PDT

Many researchers who use same-source data face concerns about common method variance (CMV). Although post hoc statistical detection and correction techniques for CMV have been proposed, there is a lack of empirical evidence regarding their efficacy. Because of disagreement among scholars regarding the likelihood and nature of CMV in self-report data, the current study evaluates three post hoc strategies and the strategy of doing nothing within three sets of assumptions about CMV: that CMV does not exist, that CMV exists and has equal effects across constructs, and that CMV exists and has unequal effects across constructs. The implications of using each strategy within each of the three assumptions are examined empirically using 691,200 simulated data sets varying factors such as the amount of true variance and the amount and nature of CMV modeled. Based on analyses of these data, potential benefits and likely risks of using the different techniques are detailed.

Book Review: Creswell, J., & Plano Clark, V. (2007). Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage

Yu, C. H. — Wed, 23 Sep 2009 09:23:46 PDT

Book Review: Van de Ven, A. (2007). Engaged scholarship: A guide for organizational and social research. New York: Oxford University Press

Adelman, M., Spivack, A. — Wed, 23 Sep 2009 09:23:46 PDT

Book Review: Dunn-Rankin, P., Knezek, G. A., Wallace, S., & Zhang, S. (2004). Scaling methods (2nd ed.). Mahwah, NJ: Lawrence Erlbaum

Hurley-Hanson, A. E. — Wed, 23 Sep 2009 09:23:47 PDT

Ad Hoc Reviewers From July 1, 2007, Through June 1, 2009

Wed, 23 Sep 2009 09:23:47 PDT

Using Artificial Neural Networks to Model Nonlinearity: The Case of the Job Satisfaction--Job Performance Relationship

Somers, M. J., Casal, J. C. — Fri, 29 May 2009 10:48:48 PDT

Neural networks are advanced pattern recognition algorithms capable of extracting complex, nonlinear relationships among variables. This study examines those capabilities by modeling nonlinearities in the job satisfaction—job performance relationship with multilayer perceptron and radial basis function neural networks. A framework for studying nonlinear relationships with neural networks is offered. It is implemented using the job satisfaction—job performance relationship with results indicative of pervasive patterns of nonlinearity.

The Accuracy of Significance Tests for Slope Variance Components in Multilevel Random Coefficient Models

LaHuis, D. M., Ferguson, M. W. — Fri, 29 May 2009 10:48:48 PDT

This study examines the behavior of three tests for significant slope variance in multilevel random coefficient (MRC) models: the Hierarchical Linear Modeling chi-square test, the likelihood ratio test (LRT), and the corrected LRT. Monte Carlo simulations are conducted varying the numbers of groups, group size, and effect size. Results suggest that neither the number of groups nor group size influenced Type I errors. Group size has a stronger effect on power compared with the number of groups. The one-tailed LRT demonstrates the best balance between power and Type I errors. Recommendations for conducting MRC analyses are presented.

Text Mining in Qualitative Research: Application of an Unsupervised Learning Method

Janasik, N., Honkela, T., Bruun, H. — Fri, 29 May 2009 10:48:48 PDT

The article provides an introduction to and a demonstration of the self-organizing map (SOM) method for organizational researchers interested in the use of qualitative data. The SOM is a versatile quantitative method very commonly used across many disciplines to analyze large data sets. The outcome of the SOM analysis is a map in which entities are positioned according to similarity. The authors' argument is that text mining using the SOM is particularly effective in improving inference quality within qualitative research. SOM creates multiple well-grounded perspectives on the data and thus improves the quality of the concepts and categories used in the analysis.

Assessing Within-Group Agreement: A Critical Examination of a Random-Group Resampling Approach

Ludtke, O., Robitzsch, A. — Fri, 29 May 2009 10:48:48 PDT

The measure of within-group agreement most frequently encountered in organizational psychology is the r_WG index. The r_WG index is determined by comparing the observed group variance among raters with an expected random variance. The most critical issue in calculating the r_WG is the choice of an appropriate random distribution that would be expected to follow from raters making their ratings at random. A data-driven approach that uses random-group resampling (RGR) procedures to determine the expected random variance has been proposed. In the present study, the application of the RGR procedure will be illustrated with reference to students' ratings of their mathematics instruction and critically compared with a recently proposed simulation-based approach. It will be shown mathematically that the probability of obtaining statistically significant within-group agreement when applying the RGR procedure strongly depends on the intraclass correlation as well as on the group sizes. Finally, implications for applying the RGR procedure to assess within-group agreement in multilevel data will be discussed.

The Design of Member Review: Showing What to Organization Members and Why

Locke, K., Ramakrishna Velamuri, S. — Fri, 29 May 2009 10:48:48 PDT

Noting various forces prompting qualitative researchers to incorporate some form of member review into their studies, this article aims to help researchers anticipate and develop their own considered strategies for designing and executing this process. Drawing on existing discussions of member review in the sociological and anthropological literature, the article develops a framework that suggests different ways in which member reviews might be designed and executed, it outlines the types of challenges researchers may anticipate during execution of the designs and highlights the positive and negative influences that creating the opportunity for such challenges can have on the research. A dissertation-based case study illustrates how challenges to the research arising from execution of a particular member review design unfolded in practice and forms the basis for considering how researchers might respond when research participants take exception to what we write.

Detecting Interaction Effects in Moderated Multiple Regression With Continuous Variables Power and Sample Size Considerations

Shieh, G. — Fri, 29 May 2009 10:48:48 PDT

In view of the long-recognized difficulties in detecting interactions among continuous variables in moderated multiple regression analysis, this article aims to address the problem by providing feasible solutions to power calculation and sample size determination for significance test of moderating effects. The proposed approach incorporates the essential factors of strength of moderator effect, magnitude of error variation, and distributional property of predictor and moderator variables into a unified framework. Accordingly, careful consideration across different plausible and practical configurations of the prescribed factors is an important aspect of power and sample size computations in planning moderated multiple regression research. The performance of the suggested procedure and an alternative simplified method is illustrated with detailed numerical studies. The simulation results demonstrate that an acceptable degree of accuracy can be obtained using the recommended method in assessing moderated relationships.

Examining Question and Context Effects in Organization Survey Data Using Item Response Theory

Rivers, D. C., Meade, A. W., Lou Fuller, W. — Fri, 29 May 2009 10:48:48 PDT

Organizational researchers routinely use attitudinal surveys to track organizational development and identify areas for intervention. However, seemingly trivial changes to the survey instrument, such as question wording or question order, can introduce measurement artifacts leading to differences in observed responses that are not due to actual employee attitudinal change. Traditional methods for assessing the presence of artifacts because of survey changes require additional survey administration using multiple survey forms and random assignment. However, the item response theory method illustrated in this study eliminates the need for additional data collection, offers a more rigorous design, and requires fewer organizational resources.

Designing Experiments That Generalize

Highhouse, S. — Fri, 29 May 2009 10:48:49 PDT

Organizational research has relied too heavily on methods characterized by passive observation, likely because there is a widespread belief that experimental research has limited generalizability. However, this is often because researchers (and reviewers or editors) misunderstand the nature of generalizability and what it requires. This article reiterates the importance of experimental research for understanding organizational phenomena and separates the legitimate concerns about experimental generalizability from the irrelevant ones. Whereas most criticisms of experiments focus on sample characteristics and mundane realism (i.e., superficial resemblance to the real world), more attention needs to be paid to the degree to which the treatment manipulation is valid, representative, and strong.

The Case Study as Disciplinary Convention: Evidence From International Business Journals

Piekkari, R., Welch, C., Paavilainen, E. — Fri, 29 May 2009 10:48:49 PDT

This article explores case study practices within a specific management discipline, that of international business. The authors contrast the case study debate in the general methodological literature to how this method is practiced within this particular scientific community. They review 135 case study—based articles published in four international business journals from 1995 to 2005 and 22 from 1975 to 1994, finding the disciplinary convention in these journals to be exploratory, interview-based multiple case studies, drawing on positivistic assumptions and cross-sectional designs. Alternative perspectives on the case study that the authors identify in the methodological literature have had little impact on this field. Even the most commonly cited methodological literature is not consistently followed. Given these limitations of the disciplinary convention, the authors argue for greater methodological pluralism in conducting case studies and provide suggestions for researchers seeking to adopt alternative case study traditions.

Testing for Between-Group Differences in Within-Group Interrater Agreement

Pasisz, D. J., Hurtz, G. M. — Fri, 29 May 2009 10:48:49 PDT

Users of r_wg may find situations where they want to compare the levels of within-group agreement across two or more groups. F tests are presented that compare two values of r_wg, rwg(J), or r*_wg(J), and recommendations are made for comparing pairwise group values for three or more groups with a control for alpha inflation. More robust and flexible methods based on O'Brien's test are then recommended when sufficient raw data are available. The meaning of findings that agreement statistics are either the same or different across groups is discussed in light of simultaneous comparisons of mean differences. Examples are shown to demonstrate the robust methods using employee responses to questions regarding climate perceptions of organizational support for employee development activities. Other scenarios where the F test may prove useful are discussed, and directions for future research are provided.

Book Review: Corbin, J., & Strauss, A. (2008). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (3rd ed.). Thousand Oaks, CA: Sage

Service, R. W. — Fri, 29 May 2009 10:48:49 PDT

Book Review: Elsbach, K. D. (Ed.). (2005). Qualitative Organizational Research: Best Papers from the Davis Conference on Qualitative Research. Greenwich, CT: Information Age

Ford, L. R. — Fri, 29 May 2009 10:48:49 PDT

Scientific Achievement and Editorial Board Membership

Bedeian, A. G., Van Fleet, D. D., Hyman, H. H. — Tue, 24 Feb 2009 10:58:15 PST

A cornerstone of the scientific ethos is that editorial board members should be selected based on their scholarly achievements, as demonstrated by publications in peer-reviewed journals and evidence that their work is of value to others in their disciplines. To discern if this reasoning is applied in practice, this study examined the scholarly records of the editorial boards of premier peer-reviewed journals sponsored by the leading professional associations in management and six related disciplines.

Begin the Journey With the End in Mind

Klimoski, R. — Tue, 24 Feb 2009 10:58:15 PST

This commentary critically examines the issues raised in the article by Bedeian, Van Fleet, and Hyman from a functional perspective, one that might be used in decisions about who should be a member of a prototypic work group. It takes the position that, like a work group, a journal editorial board exists for a purpose, and that purpose (or function) should play a major role in determining the criteria to be used when staffing the board's membership. When contemporary thinking about the staffing of work groups is applied, the current practices of the journals of interest are found to be appropriate and defensible.

Editorial Judgments, Quality Scholarship, and the Academy of Management's Journals

Hitt, M. A. — Tue, 24 Feb 2009 10:58:15 PST

Bedeian, Van Fleet, and Hyman's article offers a critical examination of the scholarly achievements of editorial review board members for several scholarly journals in related disciplines, including the Academy of Management's primary scholarly journals, the Academy of Management Review and the Academy of Management Journal (AMJ). The author finds their critique to be interesting and thorough in many ways. However, this commentary examines several issues related to their critique, including the methodology, criteria for and process of selecting editorial review board members, the editor's role in the evaluation of work submitted to journals, and the performance of scholarly journals. Consideration of these issues allows a broader and more complete view of scholarly publishing, leading to some different conclusions.

Successful Authors and Effective Reviewers: Balancing Supply and Demand in the Organizational Sciences

Tsui, A. S., Hollenbeck, J. R. — Tue, 24 Feb 2009 10:58:16 PST

Bedeian, Van Fleet, and Hyman offer some data that cast doubt on the qualifications of the reviewers of the flagship journals in our field. Using two alternative measures of authoring success, we found the reviewers of Academy of Management Journal are as a group more accomplished scholars than the authors are as a group. However, we suggest that a more relevant conversation should be about addressing the large gap between the demand for effective reviewers and the supply of individuals who are both successful authors and effective reviewers. We encourage a two-pronged approach involving reducing the demand for reviews on one hand and increasing the supply of effective reviewers on the other hand. We offer several procedural and structural alternatives to initiate a conversation that might help to move us toward some norms to encourage and reward effective reviewing in our field.

``Circle the Wagons and Defend the Faith'': Slicing and Dicing the Data

Bedeian, A. G., Van Fleet, D. D., Hyman, H. H. — Tue, 24 Feb 2009 10:58:16 PST

Commentators expressed contrasting views on the authors' examination of scientific achievement and editorial board membership in the management discipline. In response, the authors address key points with which they disagree and hold fast to their admonitory conclusions, neither compromising nor retreating from the recounting of base facts. If the authors' conclusions have prompted a measure of cognitive dissonance, they hope that any associated discomfort will lead to action on the part of all of the discipline's journals and their sponsoring organizations.

Investigating Faking Using a Multilevel Logistic Regression Approach to Measuring Person Fit

LaHuis, D. M., Copeland, D. — Tue, 24 Feb 2009 10:58:16 PST

This article describes how a multilevel logistic regression (MLR) approach to assessing person fit can be used to test hypotheses concerning faking on personality assessments. Item difficulty and person trait estimates obtained from a two-parameter logistic item response theory model are used to predict the probability of endorsing an item in a MLR equation. The regression slope for item difficulty reflects the extent to which the probability of endorsement decreases as item difficulty increases. Less negative slopes may indicate faking, and slope variance may be modeled with person-level variables using MLR. Two examples are presented. Example 1 models faking on a personality assessment with dichotomous items. Example 2 extends the approach to scales using polytomous items.

Estimating Statistical Power With Incomplete Data

Davey, A., Savla, J. — Tue, 24 Feb 2009 10:58:16 PST

Software developments increasingly facilitate inclusion of incomplete data, but relatively little research has examined the effects of incomplete data on statistical power. Seven steps needed to conduct power analyses with incomplete data for a variety of commonly tested hypotheses are illustrated, focusing on significance tests of individual parameters. The example extends a growth curve model simulation presented by Curran and Muthén (1999) to the incomplete data situation. How to estimate statistical power for a range of sample sizes from a single model, as well as how to calculate the sample size required to obtain a desired value of statistical power, is demonstrated. Effects of data being missing completely at random (MCAR) or missing at random (MAR) across a range from 0% (complete data) to 95% missing data are considered. SAS and LISREL syntax are provided in this paper with syntax for other software available from the authors.

Estimating Statistical Power and Required Sample Sizes for Organizational Research Using Multilevel Modeling

Scherbaum, C. A., Ferreter, J. M. — Tue, 24 Feb 2009 10:58:16 PST

The use of multilevel modeling to investigate organizational phenomena is rapidly increasing. Unfortunately, little advice is readily available for organizational researchers attempting to determine statistical power when using multilevel models or when determining sample sizes for each level that will maximize statistical power. This article presents an introduction to statistical power in multilevel models. The unique factors influencing power in multilevel models and calculations for estimating power for simple fixed effects, variance components, and cross-level interactions are presented. The results of simulation studies and the existing general rules of thumb are discussed, and the available power analysis software is reviewed.

Composing Group-Level Constructs From Individual-Level Survey Data

van Mierlo, H., Vermunt, J. K., Rutte, C. G. — Tue, 24 Feb 2009 10:58:16 PST

Group-level constructs are often derived from individual-level data. This procedure requires a composition model, specifying how the lower level data can be combined to compose the higher level construct. Two common composition methods are direct consensus composition, where items refer to the individual, and referent-shift consensus composition, where items refer to the group. The use and selection of composition methods is subject to a number of problems, calling for more systematic work on the empirical properties of and distinction between constructs composed by different methods. To facilitate and encourage such work, the authors present a methodological framework for addressing the distinction between and the baseline psychometric quality of composed group constructs, illustrated by an empirical example in the group job-design domain. The framework primarily represents a developmental tool with applications in multilevel theory building and scale construction, but also in meta-analysis or secondary analysis, and more general, the validation of group constructs.

Book Review: Brewer, E. W., Achilles, C. M., Fuhriman, J. R., & Hollingsworth, C. (2001). Finding Funding: Grant Writing From Start to Finish, Including Project Management and Internet Use (4th ed.). Thousand Oaks, CA: Corwin Press

Wallace, J. C. — Tue, 24 Feb 2009 10:58:16 PST

Book Review: Locke, L. F., Spirduso, W. W., & Silverman, S. J. (2007). Proposals that work: A guide for planning dissertations and grant proposals (5th ed.). Thousand Oaks, CA: Sage

Tejeda, M. J. — Tue, 24 Feb 2009 10:58:16 PST

Organizational Research Methods Yearly Update

Vandenberg, R. — Wed, 17 Dec 2008 16:47:39 PST

Introducing the Latent Congruence Model for Improving the Assessment of Similarity, Agreement, and Fit in Organizational Research

Cheung, G. W. — Wed, 17 Dec 2008 16:47:39 PST

A structural equation modeling—based latent congruence model (LCM) is developed for studying congruence in organizational research. Numerical examples are used to demonstrate that the LCM offers many advantages over the current approaches (difference scores, profile similarity indices, and polynomial regression) to studying congruence. The LCM can (a) control for measurement errors by specifying the congruence of latent variables, (b) examine measurement equivalence across the component measures, (c) examine the antecedents and consequences of both the mean (absolute level) and difference (congruence) of two component measures simultaneously, (d) study both congruence and components by decomposing the congruence measures into component measures, and (e) examine complex congruence models that include multiple congruence measures as antecedents and/or consequences.

Latent Variable Modeling in Congruence Research: Current Problems and Future Directions

Edwards, J. R. — Wed, 17 Dec 2008 16:47:39 PST

During the past decade, the use of polynomial regression has become increasingly prevalent in congruence research. One drawback of polynomial regression is that it relies on the assumption that variables are measured without error. This assumption is relaxed by structural equation modeling with latent variables. One application of structural equation modeling to congruence research is the latent congruence model (LCM). Although the LCM takes measurement error into account and allows tests of measurement equivalence, it is framed around the mean and algebraic difference of the components of congruence (e.g., the person and organization), which creates various interpretational problems. This article discusses problems with the LCM and shows how these problems are resolved by a linear structural equation model that uses the components of congruence as predictors and outcomes. Extensions of the linear model to quadratic equations used in polynomial regression analysis are discussed.

A Multiple-Perspective Approach to Data Analysis in Congruence Research

Cheung, G. W. — Wed, 17 Dec 2008 16:47:39 PST

Despite the popularity of the congruence construct (similarity, fit, and agreement) in organizational theories, the operationalization of congruence and the appropriate methods to analyze it have concerned many researchers. A structural equation modeling-based latent congruence model (LCM) that operationalizes congruence as the mean and difference of the component measures has recently been introduced. The LCM provides a simple analytical framework for examining the measurement equivalence of the component measures and for conducting congruence analysis and component analysis. The objective of this note is to highlight the similarities and differences between the LCM and the polynomial regression (PR) approach to studying congruence. The major difference is that the LCM considers congruence and its components as distinctive constructs, and therefore they can be used to answer different research questions. The determination of which constructs and analytical approach to use should be based on the theory and research hypotheses that answer the research question. Indeed, because LCM provides a simple framework for both congruence analysis and component analysis, researchers are encouraged to answer their research questions from both perspectives.

First Decade of Organizational Research Methods: Trends in Design, Measurement, and Data-Analysis Topics

Aguinis, H., Pierce, C. A., Bosco, F. A., Muslin, I. S. — Wed, 17 Dec 2008 16:47:39 PST

The authors conducted a content analysis of the 193 articles published in the first 10 volumes (1998 to 2007) of Organizational Research Methods (ORM). The most popular quantitative topics are surveys, temporal issues, and electronic/Web research (research design); validity, reliability, and level of analysis of the dependent variable (measurement); and multiple regression/correlation, structural equation modeling, and multilevel research (data analysis). The most popular qualitative topics are interpretive, policy capturing, and action research (research design); surveys and reliability (measurement); and interpretive, policy capturing, and content analysis (data analysis). The authors found upward trends in the attention devoted to surveys and electronic/Web research, interpretive, and action research (research design); level of analysis of the dependent variable and validity (measurement); and multilevel research (data analysis). Implications for training doctoral students, retooling researchers, future research on methodology, the advancement of the organizational sciences, and the extent to which ORM is fulfilling its mission are discussed.

How Do Missing Data Bias Estimates of Within-Group Agreement? Sensitivity of SD WG, CVWG, rWG(J), rWG(J) * , and ICC to Systematic Nonresponse

Newman, D. A., Sin, H.-P. — Wed, 17 Dec 2008 16:47:39 PST

In multilevel theory testing, estimation of group-level properties (i.e., consensus and diversity) is often complicated by missing data. Researchers are left to draw inferences about group constructs (e.g., organizational climate and climate strength) from the responses of only a subset of group members. This study analyzes the biasing impact of random and non-random missingness patterns on within-group agreement and reliability (standard deviation, coefficient of variation, r_WG(J), r^*_WG(J), AD_M, a_WG , and intraclass correlation) across a range of response rates, numbers of items, and systematic missing data mechanisms. Results demonstrate biases up to 20% over- or underestimation for common response rates found in organizational research. Correction formulae are presented, which enable assessment of the sensitivity of multilevel results to survey nonresponse.

Testing Agreement for Multi-Item Scales With the IndicesrWG(J) and AD M(J)

Cohen, A., Doveh, E., Nahum-Shani, I. — Wed, 17 Dec 2008 16:47:39 PST

The most popular index of agreement has been r_WG(J); more recently, the AD_M(J) index also has been used. This study addresses two problems: first, how to test the statistical significance of r_WG(J) and AD_M(J) and, second, how to infer from the indices that were evaluated for each group about the agreement of the ensemble of groups. The authors extend the inference based on either r_WG(J) or AD_M(J) by focusing on multiple-item scales and on the whole ensemble of groups. Their method is based on simulations, as was done by Dunlap, Burke, and Smith-Crowe (2003) and by Cohen, Doveh, and Eick (2001). The tests are illustrated on the data of Bliese, Halverson, and Schriesheim (2002) pertaining to a sample of 2,042 U.S Army soldiers in 49 U.S. Army companies. Software for our procedures is available both as a SAS code and in the Multilevel Modeling in R package (Bliese, 2006).

The Multifaceted Nature of Measurement Artifacts and Its Implications for Estimating Construct-Level Relationships

Le, H., Schmidt, F. L., Putka, D. J. — Wed, 17 Dec 2008 16:47:39 PST

Measurement artifacts, including measurement errors and scale-specific factors, distort observed correlations between measures of psychological and organizational constructs. The authors discuss two alternative procedures, one using the generalized coefficient of equivalence and stability (GCES) and one based on structural equation modeling, to correct for the biasing effect of measurement artifacts in order to estimate construct-level relationships. Assumptions underlying the procedures are discussed and the degrees of biases resulting from violating the assumptions are examined by means of Monte Carlo simulation. They then propose an approach using cumulative knowledge in the literature about properties of measures of a construct to estimate the GCES. That approach can allow researchers to estimate relationships between constructs in most research situations. The authors apply the approach to estimate the GCES for overall job satisfaction, an important organizational construct.

Book Review: Wilson, M. (2004). Constructing Measures: An Item Response Modeling Approach. Mahwah, NJ: Lawrence Erlbaum

Schmitt, N. — Wed, 17 Dec 2008 16:47:39 PST

Book Review: Stevens, J. P. (2007). Intermediate Statistics: A Modern Approach (3rd ed.). Mahwah, NJ: Lawrence Erlbaum

Baugh, S. G. — Wed, 17 Dec 2008 16:47:39 PST