Psychological Methods

The comparison of idiographic network structures to determine the presence of heterogeneity is a challenging endeavor in many applied settings. Previously, researchers eyeballed idiographic networks, computed correlations, and used techniques that make use of the multilevel structure of the data (e.g., group iterative multiple model estimation and multilevel vector autoregressive) to investigate individual differences. However, these methods do not allow for testing the (in)equality of idiographic network structures directly...

Human decision behavior entails a graded awareness of its certainty, known as a feeling of confidence. Until now, considerable interest has been paid to behavioral and computational dissociations of decision and confidence, which has raised an urgent need for measurement frameworks that can quantify the efficiency of confidence rating relative to decision accuracy (metacognitive efficiency). As a unique addition to such frameworks, we have developed a new signal detection theory paradigm utilizing the generalized Gaussian distribution (GGSDT)...

Latent moderated structural equation (LMS) is one of the most common techniques for estimating interaction effects involving latent variables (i.e., XWITH command in Mplus). However, empirical applications of LMS often overlook that this estimation technique assumes normally distributed variables and that violations of this assumption may lead to seriously biased parameter estimates. Against this backdrop, we study the robustness of LMS to different shapes and sources of nonnormality and examine whether various statistical tests can help researchers detect such distributional misspecifications...

Methods that measure the association between two intensively measured time series are of interest to researchers studying the symmetry of behaviors during social interaction. Such methods have historically focused on aggregating the amount of symmetry across all measurement occasions. However, it is rarely expected that symmetry is present at all measurement occasions. The current method, the pairwise approximate spatiotemporal symmetry (PASS) algorithm, is an approach that may be used to determine which measurement occasions in pairwise time series are indicative of symmetry and which are not...

Colliders, variables that serve as a common outcome of an independent and dependent variable, pose a major challenge in psychological research. Collider variables can induce bias in the estimation of a population relationship of interest when (a) the composition of a research sample is restricted by scores on a collider variable or (b) researchers adjust for a collider variable in their statistical analyses, as they might do for confounder variables. Both cases interfere with the accuracy and generalizability of statistical results...

Some researchers and clinicians may feel hesitant to assess sexual orientation and gender-related characteristics in youth surveys because they are unsure if youth will respond to these questions or are concerned the questions will cause discomfort or offense. This can result in missed opportunities to identify LGBTQ+ youth and address health inequities among this population. The aim of this study was to examine the prevalence and sociodemographic patterns of missingness among survey questions assessing current sexual orientation, gender identity and expression (SOGIE), and past change in sexual orientation (sexual fluidity) among a diverse sample of U...

Measurement invariance is an assumption underlying the regression of a latent variable on a background variable. It requires the measurement model parameters of the latent variable to be equal across the levels of the background variable. Item-specific violations of this assumption are referred to as differential item functioning and are ideally substantively explainable to warrant theoretically valid and meaningful results. Past research has focused on developing statistical approaches to explain differential item functioning effects in terms of item- or person-specific covariates...

Following an extensive simulation study comparing the operating characteristics of three different procedures used for establishing equivalence (the frequentist "TOST," the Bayesian "HDI-ROPE," and the Bayes factor interval null procedure), Linde et al. (2021) conclude with the recommendation that "researchers rely more on the Bayes factor interval null approach for quantifying evidence for equivalence" (p. 1). We redo the simulation study of Linde et al. (2021) in its entirety but with the different procedures calibrated to have the same predetermined maximum Type I error rate...

Individual differences are studied with a multitude of test instruments. Meta-analysis of tests is useful to understand whether individual differences in certain populations can be detected with the help of a class of tests. A method for the quantitative meta-analytical evaluation of test instruments with dichotomous items is introduced. The method assumes beta-binomially distributed test scores, an assumption that has been demonstrated to be plausible in many settings. With this assumption, the method only requires sample means and standard deviations of sum scores (or equivalently means and standard deviations of percent-correct scores), in contrast to methods that use estimates of reliability for a similar purpose...

Loftus (1978) highlighted the distinction between a theoretical concept such as memory or attention, and its observed measure such as hit rate or percent correct. If the functional relationship between the concept and its measure is nonlinear then only some interaction effects are interpretable. This is an example of the wider "problem of coordination" which pervades scientific measurement. Loftus drew on the principles of additive conjoint measurement (ACM) to discuss the consequences when the coordination function is assumed to be monotonic...

Ecological momentary assessment (EMA) involves repeated real-time sampling of respondents' current behaviors and experiences. The intensive repeated assessment imposes an increased burden on respondents, rendering EMAs vulnerable to respondent noncompliance and/or careless and insufficient effort responding (C/IER). We developed a mixture modeling approach that equips researchers with a tool for (a) gauging the degree of C/IER contamination of their EMA data and (b) studying the trajectory of C/IER across the study...

Longitudinal designs can fortify causal inquiries of a focal predictor (i.e., treatment) on an outcome. But valid causal inferences are complicated by causal feedback between confounders and treatment over time. G-estimation of a structural nested mean model (SNMM) is designed to handle the complexities beset by measured time-varying or treatment-dependent confounding in longitudinal data. But valid inference requires correctly specifying the functional form of the SNMM, such as how the effects stay constant or change over time...

The linear mixed model (LMM) and latent growth model (LGM) are frequently applied to within-subject two-group comparison studies to investigate group differences in the time effect, supposedly due to differential group treatments. Yet, research about LMM and LGM in the presence of outliers (defined as observations with a very low probability of occurrence if assumed from a given distribution) is scarce. Moreover, when such research exists, it focuses on estimation properties (bias and efficiency), neglecting inferential characteristics (e...

Multilevel models allow researchers to test hypotheses at multiple levels of analysis-for example, assessing the effects of both individual-level and school-level predictors on a target outcome. To assess these effects with the greatest clarity, researchers are well-advised to cluster mean center all Level 1 predictors and explicitly incorporate the cluster means into the model at Level 2. When an outcome of interest is continuous, this unconflated model specification serves both to increase model accuracy, by separating the level-specific effects of each predictor, and to increase model interpretability, by reframing the random intercepts as unadjusted cluster means...

Observed scores (e.g., summed scores and estimated factor scores) are assumed to reflect underlying constructs and have many uses in psychological science. Constructs are often operationalized as latent variables (LVs), which are mathematically defined by their relations with manifest variables in an LV measurement model (e.g., common factor model). We examine the performance of several types of observed scores for the purposes of (a) estimating latent scores and classifying people and (b) recovering structural relations among LVs...

A fundamental part of experimental design is to determine the sample size of a study. However, sparse information about population parameters and effect sizes before data collection renders effective sample size planning challenging. Specifically, sparse information may lead research designs to be based on inaccurate a priori assumptions, causing studies to use resources inefficiently or to produce inconclusive results. Despite its deleterious impact on sample size planning, many prominent methods for experimental design fail to adequately address the challenge of sparse a-priori information...

Bayesian linear mixed-effects models (LMMs) and Bayesian analysis of variance (ANOVA) are increasingly being used in the cognitive sciences to perform null hypothesis tests, where a null hypothesis that an effect is zero is compared with an alternative hypothesis that the effect exists and is different from zero. While software tools for Bayes factor null hypothesis tests are easily accessible, how to specify the data and the model correctly is often not clear. In Bayesian approaches, many authors use data aggregation at the by-subject level and estimate Bayes factors on aggregated data...

Self-report scales are widely used in psychology to compare means in latent constructs across groups, experimental conditions, or time points. However, for these comparisons to be meaningful and unbiased, the scales must demonstrate measurement invariance (MI) across compared time points or (experimental) groups. MI testing determines whether the latent constructs are measured equivalently across groups or time, which is essential for meaningful comparisons. We conducted a systematic review of 426 psychology articles with openly available data, to (a) examine common practices in conducting and reporting of MI testing, (b) assess whether we could reproduce the reported MI results, and (c) conduct MI tests for the comparisons that enabled sufficiently powerful MI testing...

Network psychometrics leverages pairwise Markov random fields to depict conditional dependencies among a set of psychological variables as undirected edge-weighted graphs. Researchers often intend to compare such psychometric networks across subpopulations, and recent methodological advances provide invariance tests of differences in subpopulation networks. What remains missing, though, is an analogue to an effect size measure that quantifies differences in psychometric networks. We address this gap by complementing recent advances for investigating whether psychometric networks differ with an intuitive similarity measure quantifying the extent to which networks differ...

For over three decades, methodologists have cautioned against the use of cross-sectional mediation analyses because they yield biased parameter estimates. Yet, cross-sectional mediation models persist in practice and sometimes represent the only analytic option. We propose a sensitivity analysis procedure to encourage a more principled use of cross-sectional mediation analysis, drawing inspiration from Gollob and Reichardt (1987, 1991). The procedure is based on the two-wave longitudinal mediation model and uses phantom variables for the baseline data...