Psychological Methods

Self-report scales are widely used in psychology to compare means in latent constructs across groups, experimental conditions, or time points. However, for these comparisons to be meaningful and unbiased, the scales must demonstrate measurement invariance (MI) across compared time points or (experimental) groups. MI testing determines whether the latent constructs are measured equivalently across groups or time, which is essential for meaningful comparisons. We conducted a systematic review of 426 psychology articles with openly available data, to (a) examine common practices in conducting and reporting of MI testing, (b) assess whether we could reproduce the reported MI results, and (c) conduct MI tests for the comparisons that enabled sufficiently powerful MI testing...

Network psychometrics leverages pairwise Markov random fields to depict conditional dependencies among a set of psychological variables as undirected edge-weighted graphs. Researchers often intend to compare such psychometric networks across subpopulations, and recent methodological advances provide invariance tests of differences in subpopulation networks. What remains missing, though, is an analogue to an effect size measure that quantifies differences in psychometric networks. We address this gap by complementing recent advances for investigating whether psychometric networks differ with an intuitive similarity measure quantifying the extent to which networks differ...

For over three decades, methodologists have cautioned against the use of cross-sectional mediation analyses because they yield biased parameter estimates. Yet, cross-sectional mediation models persist in practice and sometimes represent the only analytic option. We propose a sensitivity analysis procedure to encourage a more principled use of cross-sectional mediation analysis, drawing inspiration from Gollob and Reichardt (1987, 1991). The procedure is based on the two-wave longitudinal mediation model and uses phantom variables for the baseline data...

While a great deal of thought, planning, and money goes into the design of multisite randomized control trials (RCTs) that are used to evaluate the effectiveness of interventions in fields like education and psychology, relatively little thought is often paid to the measurement choices made in such evaluations. In this study, we conduct a series of simulation studies that consider a wide range of options for producing scores from multiple administration of assessments in the context of multisite RCTs. The scoring models considered range from the simple (sum scores) to highly complex (multilevel two-tier item response theory [IRT] models with latent regression)...

In observational data, understanding the causal link when estimating the causal effect of an independent variable ( x ) on a dependent variable ( y ) often requires researchers to identify the role of a third variable in the x → y relationship. Mediation, confounding, and colliding are three key third-variable effects that yield different theoretical and methodological implications for drawing causal conclusions. Commonly used covariance-based statistical methods, such as linear regression and structural equation modeling, cannot distinguish these effects in practice, however...

Factor mixture modeling (FMM) incorporates both continuous latent variables and categorical latent variables in a single analytic model clustering items and observations simultaneously. After two decades since the introduction of FMM to psychological and behavioral science research, it is an opportune time to review FMM applications to understand how these applications are utilized in real-world research. We conducted a systematic review of 76 FMM applications. We developed a comprehensive coding scheme based on the current methodological literature of FMM and evaluated common usages and practices of FMM...

The availability of smart devices has made it possible to collect intensive longitudinal data (ILD) from individuals, providing a unique opportunity to study the complex dynamics of psychological systems. Existing time-series methods often have limitations, such as assuming linear interactions or having restricted forms, leading to difficulties in capturing the complex nature of these systems. To address this issue, we introduce fitlandr, a method with implementation as an R package that integrates nonparametric estimation of the drift-diffusion function and stability landscape...

The planning of adequately powered research designs increasingly goes beyond determining a suitable sample size. More challenging scenarios demand simultaneous tuning of multiple design parameter dimensions and can only be addressed using Monte Carlo simulation if no analytical approach is available. In addition, cost considerations, for example, in terms of monetary costs, are a relevant target for optimization. In this context, optimal design parameters can imply a desired level of power at minimum cost or maximum power at a cost threshold...

Polynomial regression is an old and commonly discussed modeling technique, though recommendations for its usage are widely variable. Here, we make the case that polynomial regression with second- and third-order terms should be part of every applied practitioners standard model-building toolbox, and should be taught to new students of the subject as the default technique to model nonlinearity. We argue that polynomial regression is superior to nonparametric alternatives for nonstatisticians due to its ease of interpretation, flexibility, and its nonreliance on sophisticated mathematics, like knots and kernel smoothing...

This study proposes a Bayesian approach to testing informative hypotheses in confirmatory factor analysis (CFA) models. The informative hypothesis, which is formulated by the constrained loadings, can directly represent researchers' theories or expectations about the tau equivalence in reliability analysis, item-level discriminant validity, and relative importance of indicators. Support for the informative hypothesis is quantified by the Bayes factor. We present the adjusted fractional Bayes factor of which the prior distribution is specified using a part of the data and adjusted according to the hypotheses under evaluation...

Meta- d'/d' has become the quasi-gold standard to quantify metacognitive efficiency because meta- d'/d' was developed to control for discrimination performance, discrimination criteria, and confidence criteria even without the assumption of a specific generative model underlying confidence judgments. Using simulations, we demonstrate that meta- d'/d' is not free from assumptions about confidence models: Only when we simulated data using a generative model of confidence according to which the evidence underlying confidence judgments is sampled independently from the evidence utilized in the choice process from a truncated Gaussian distribution, meta- d'/d' was unaffected by discrimination performance, discrimination task criteria, and confidence criteria...

Intervention studies in psychology often focus on identifying mechanisms that explain change over time. Cross-lagged panel models (CLPMs) are well suited to study mechanisms, but there is a controversy regarding the importance of detrending-defined here as separating longer-term time trends from cross-lagged effects-when modeling these change processes. The aim of this study was to present and test the arguments for and against detrending CLPMs in the presence of an intervention effect. We conducted Monte Carlo simulations to examine the impact of trends on estimates of cross-lagged effects from several longitudinal structural equation models...

Repeated measure data design has been used extensively in a wide range of fields, such as brain aging or developmental psychology, to answer important research questions exploring relationships between trajectory of change and external variables. In many cases, such data may be collected from multiple study cohorts and harmonized, with the intention of gaining higher statistical power and enhanced external validity. When psychological constructs are measured using survey scales, a fundamental psychometric challenge for data harmonization is to create commensurate measures for the constructs of interest across studies...

Data sets with missing observations are common in psychology research. One typically analyzes such data by applying statistical methods that rely on the assumption that the missing observations are missing at random (MAR). This assumption greatly simplifies analysis but is unverifiable from the data at hand, and assuming it incorrectly may lead to bias. Thus we often wish to conduct sensitivity analyses to judge whether conclusions are robust to departures from MAR-that is, whether key findings would hold up even if MAR does not in fact hold...

Psychological data are often clustered within organizational units, which violates the independence assumption in standard regression models. Clustered errors, multilevel models, and fixed-effects models all address this issue, but in different ways. Disciplinary preferences for approaching clustered data are strong, which can restrict questions researchers ask because certain approaches are better equipped to handle particular types of questions. Resources comparing approaches to facilitate broader understanding of clustered data approaches exist for economists, political scientists, and biostatisticians...

We introduce equivalence testing procedures for linear regression analyses. Such tests can be very useful for confirming the lack of a meaningful association between a continuous outcome and a continuous or binary predictor. Specifically, we propose an equivalence test for unstandardized regression coefficients and an equivalence test for semipartial correlation coefficients. We review how to define valid hypotheses, how to calculate p values, and how these tests compare to an alternative Bayesian approach with applications to examples in the literature...

The causal inference methods of potential outcomes (POs), directed acyclic graphs (DAGs), and structural equation models (SEMs) have contributed much to our understanding of causal effects. Yet the teaching and application of these methods (especially POs and DAGs) have nearly always regarded treatment as binary even when the magnitude of treatment can differ greatly. The two most common types of binary treatments are those from randomized experiments and those that are categorized versions of continuous treatments...

The fulfillment of measurement invariance/equivalence is considered a prerequisite for meaningfully proceeding with substantive cross-group comparisons. In the multiple-group confirmatory factor analysis approach, one model identification issue has unfortunately received little attention: the specification of a referent variable in the test of measurement invariance. A multiple-indicator multiple-cause (MIMIC) model with moderated effects (i.e., MIMIC-interaction modeling; Woods & Grimm, 2011) and a moderated nonlinear factor analysis (MNLFA; Bauer, 2017) model for detecting uniform and nonuniform measurement inequivalences in tandem were proposed to identify credible referent variables...

Estimating power for multilevel models is complex because there are many moving parts, several sources of variation to consider, and unique sample sizes at Level 1 and Level 2. Monte Carlo computer simulation is a flexible tool that has received considerable attention in the literature. However, much of the work to date has focused on very simple models with one predictor at each level and one cross-level interaction effect, and approaches that do not share this limitation require users to specify a large set of population parameters...

Partially clustered designs are widely used in psychological research, especially in randomized controlled trials that examine the effectiveness of prevention or intervention strategies. In a partially clustered trial, individuals are clustered into intervention groups in one or more study arms, for the purpose of intervention delivery, whereas individuals in other arms (e.g., the waitlist control arm) are unclustered. Missing data are almost inevitable in partially clustered trials and could pose a major challenge in drawing valid research conclusions...