In all of our research, we pay particular attention to the appropriateness of various methods and practices and what can be gained (or lost) from choosing one alternative over another. For instance, social psychologists often seek to shed light on the basic process underlying an effect by testing for mediation. A simple three-variable mediation model can be analyzed either with a typical regression approach, or with structural equation modeling (SEM) using latent variables. Which is the better option? This is a more complex and consequential question than researchers often realize.

On the one hand, statisticians often recommend an SEM approach because it tends to produce more accurate estimates. (Regression tends to give inaccurate estimates that are often too small, because they have been attenuated by measurement error.) On the other hand, our own work has shown that a regression approach tends to produce more precise estimates, with smaller standard errors and therefore increased power to detect an effect in the first place (see Ledgerwood & Shrout, 2011).

This means that SEM estimates are correctly centered (i.e., accurate) but also widely scattered (imprecise, and therefore less likely to be significant).

If each study is like a dart thrown at a dartboard, an SEM approach will get the darts to converge on the bulls-eye—the real strength of the relation between variables in the population—but there will be a lot of variability in where exactly each individual dart ends up.

 

 

In contrast, regression estimates are incorrectly centered (inaccurate, and often far too small) but tightly clustered (precise, and therefore more likely to be significant). Because there’s less variability in where the darts end up, the results of a single study are more likely to be significant—so this approach is good at telling us if there is an effect in the first place. But because the darts are nowhere near the bulls-eye, we’ll get the wrong idea about how big or small that effect actually is.

Is there a happy medium? Researchers can maximize both accuracy and precision by investing in reliable measures and by planning mediation studies with adequate power. But when highly reliable measures aren’t feasible, a two-step strategy for testing and estimating the indirect effect in a mediation model may be the best approach. We recommend using observed variables to test the indirect effect for significance (e.g., using regression and a bootstrapped SE), and then estimating the path coefficient for the indirect effect using latent variables in SEM.

See: Ledgerwood, A., & Shrout, P. E. (2011). The tradeoff between accuracy and precision in latent variable models of mediation processes. Journal of Personality and Social Psychology, 101, 1174-1188.

Other work on methods and practices:

Ledgerwood, A. (in press). New developments in research methods. Chapter to appear in R. Baumeister & E. Finkel (Eds.), Advanced Social Psychology. Oxford University Press.

Wang, Y. A., Sparks, J., Gonzales, J., Hess, Y. D., & Ledgerwood, A. (2017). Using independent covariates in experimental designs: Quantifying the trade-off between power boost and Type I error inflation. Journal of Experimental Social Psychology, 72, 118-124.

Ledgerwood, A. (2016). Introduction to the special section on improving research practices: Thinking deeply across the research cycle. Perspectives on Psychological Science, 11, 661-663.

Ledgerwood, A. (2016). The Start-Local Approach. Talk Presented at the Training Preconference of the 2016 Annual Convention of the Society for Personality and Social Psychology.
[full talk available here]

Ledgerwood, A. (2015). Practical and Painless: Easy Strategies to Transition Your Lab. Talk Presented at a Symposium on Best Practices at the 2015 Annual Convention of the Society for Personality and Social Psychology.  [slides]

Ledgerwood, A. (2014). Introduction to the special section on advancing our methods and practices. Perspectives on Psychological Science, 9, 275-277.

Ledgerwood, A. (2014). Introduction to the special section on moving toward a cumulative science: Maximizing what our research can tell us. Perspectives on Psychological Science, 9, 610-611.

Ledgerwood, A., & Sherman, J. W. (2012). Short, sweet, and problematic? The rise of the short report in psychological science. Perspectives on Psychological Science, 7, 60-66.