Psychological Methods

To establish a theory one needs cleverly designed and well-executed studies with appropriate and correctly interpreted statistical analyses. Equally important, one also needs replications of such studies and a way to combine the results of several replications into an accumulated state of knowledge. An approach that provides an appropriate and powerful analysis for studies targeting prespecified theories is the use of Bayesian informative hypothesis testing. An additional advantage of the use of this Bayesian approach is that combining the results from multiple studies is straightforward...

Multivariate generalizability theory (GT) represents a comprehensive framework for quantifying score consistency, separating multiple sources contributing to measurement error, correcting correlation coefficients for such error, assessing subscale viability, and determining the best ways to change measurement procedures at different levels of score aggregation. Despite such desirable attributes, multivariate GT has rarely been applied when measuring psychological constructs and far less often than univariate techniques that are subsumed within that framework...

The depth of information collected in participants' daily lives with active (e.g., experience sampling surveys) and passive (e.g., smartphone sensors) ambulatory measurement methods is immense. When measuring participants' behaviors in daily life, the timing of particular events-such as social interactions-is often recorded. These data facilitate the investigation of new types of research questions about the timing of those events, including whether individuals' affective state is associated with the rate of social interactions (binary event occurrence) and what types of social interactions are likely to occur (multicategory event occurrences, e...

The outcomes in single-case experimental designs (SCEDs) are often counts or proportions. In our study, we provided a colloquial illustration for a new class of generalized linear mixed models (GLMMs) to fit count and proportion data from SCEDs. We also addressed important aspects in the GLMM framework including overdispersion, estimation methods, statistical inferences, model selection methods by detecting overdispersion, and interpretations of regression coefficients. We then demonstrated the GLMMs with two empirical examples with count and proportion outcomes in SCEDs...

Decades of published methodological research have shown the chi-square test of model fit performs inconsistently and unreliably as a determinant of structural equation model (SEM) fit. Likewise, SEM indices of model fit, such as comparative fit index (CFI) and root-mean-square error of approximation (RMSEA) also perform inconsistently and unreliably. Despite rather unreliable ways to statistically assess model fit, researchers commonly rely on these methods for lack of a suitable inferential alternative. Marcoulides and Yuan (2017) have proposed the first inferential test of SEM fit in many years: an equivalence test adaptation of the RMSEA and CFI indices (i...

Several theoretical perspectives suggest that dyadic experiences are distinguished by patterns of behavioral change that emerge during interactions. Methods for examining change in behavior over time are well elaborated for the study of change along continuous dimensions. Extensions for charting increases and decreases in individuals' use of specific, categorically defined behaviors, however, are rarely invoked. Greater accessibility of Bayesian frameworks that facilitate formulation and estimation of the requisite models is opening new possibilities...

Replication studies are essential for assessing the credibility of claims from original studies. A critical aspect of designing replication studies is determining their sample size; a too-small sample size may lead to inconclusive studies whereas a too-large sample size may waste resources that could be allocated better in other studies. Here, we show how Bayesian approaches can be used for tackling this problem. The Bayesian framework allows researchers to combine the original data and external knowledge in a design prior distribution for the underlying parameters...

Intensive longitudinal studies are becoming increasingly popular because of their potential for studying the individual dynamics of psychological processes. However, measures used in such studies are quite susceptible to measurement error due to the short lengths and therefore their psychometric properties, such as reliability, are of great concern. Most existing approaches for assessing reliability are not appropriate for the intensive longitudinal data (ILD) because of the conflation of inter- and intra-individual variations or the difficulty in handling interindividual differences...

With the rising popularity of intensive longitudinal research, the modeling techniques for such data are increasingly focused on individual differences. Here we present mixture multilevel vector-autoregressive modeling, which extends multilevel vector-autoregressive modeling by including a mixture, to identify individuals with similar traits and dynamic processes. This exploratory model identifies mixture components, where each component refers to individuals with similarities in means (expressing traits), autoregressions, and cross-regressions (expressing dynamics), while allowing for some interindividual differences in these attributes...

Psychology has seen an increase in the use of machine learning (ML) methods. In many applications, observations are classified into one of two groups (binary classification). Off-the-shelf classification algorithms assume that the costs of a misclassification (false positive or false negative) are equal. Because this is often not reasonable (e.g., in clinical psychology), cost-sensitive machine learning (CSL) methods can take different cost ratios into account. We present the mathematical foundations and introduce a taxonomy of the most commonly used CSL methods, before demonstrating their application and usefulness on psychological data, that is, the drug consumption data set ( N = 1, 885) from the University of California Irvine ML Repository...

Meta-analyses in the psychological sciences typically examine moderators that may explain heterogeneity in effect sizes. One of the most commonly examined moderators is gender. Overall, tests of gender as a moderator are rarely significant, which may be because effects rarely differ substantially between men and women. While this may be true in some cases, we also suggest that the lack of significant findings may be attributable to the way in which gender is examined as a meta-analytic moderator, such that detecting moderating effects is very unlikely even when such effects are substantial in magnitude...

Given recent evidence challenging the replicability of results in the social and behavioral sciences, critical questions have been raised about appropriate measures for determining replication success in comparing effect estimates across studies. At issue is the fact that conclusions about replication success often depend on the measure used for evaluating correspondence in results. Despite the importance of choosing an appropriate measure, there is still no widespread agreement about which measures should be used...

Integrating regularization methods into structural equation modeling is gaining increasing popularity. The purpose of regularization is to improve variable selection, model estimation, and prediction accuracy. In this study, we aim to: (a) compare Bayesian regularization methods for exploring covariate effects in multiple-indicators multiple-causes models, (b) examine the sensitivity of results to hyperparameter settings of penalty priors, and (c) investigate prediction accuracy through cross-validation. The Bayesian regularization methods examined included: ridge, lasso, adaptive lasso, spike-and-slab prior (SSP) and its variants, and horseshoe and its variants...

Many psychological theories assume that observable responses are determined by multiple latent processes. Multinomial processing tree (MPT) models are a class of cognitive models for discrete responses that allow researchers to disentangle and measure such processes. Before applying MPT models to specific psychological theories, it is necessary to tailor a model to specific experimental designs. In this tutorial, we explain how to develop, fit, and test MPT models using the classical pair-clustering model as a running example...

In psychology, researchers often predict a dependent variable (DV) consisting of multiple measurements (e.g., scale items measuring a concept). To analyze the data, researchers typically aggregate (sum/average) scores across items and use this as a DV. Alternatively, they may define the DV as a common factor using structural equation modeling. However, both approaches neglect the possibility that an independent variable (IV) may have different relationships to individual items. This variance in individual item slopes arises because items are randomly sampled from an infinite pool of items reflecting the construct that the scale purports to measure...

We propose a novel method to analyze time-constrained yes/no questions about a target behavior (e.g., "Did you take sleeping pills during the last 12 months?"). A drawback of these questions is that the relative frequency of answering these questions with "yes" does not allow one to draw definite conclusions about the frequency of the target behavior (i.e., how often sleeping pills were taken) nor about the prevalence of trait carriers (i.e., percentage of people that take sleeping pills). Here we show how this information can be extracted from the results of such questions employing a prevalence curve and a Poisson model...

Latent state-trait (LST) models are increasingly applied in psychology. Although existing LST models offer many possibilities for analyzing variability and change, they do not allow researchers to relate time-varying or time-invariant covariates, or a combination of both, to loading, intercept, and factor variance parameters in LST models. We present a general framework for the inclusion of nominal and/or continuous time-varying and time-invariant covariates in LST models. The new framework builds on modern LST theory and Bayesian moderated nonlinear factor analysis and is termed moderated nonlinear LST (MN-LST) framework...

The generally small but touted as "statistically significant" correlation coefficients in the social sciences jeopardize theory testing and prediction. To investigate these small coefficients' underlying causes, traditional equations such as Spearman's (1904) classic attenuation formula, Cronbach's (1951) alpha, and Guilford and Fruchter's (1973) equation for the effect of additional items on a scale's predictive power are considered. These equations' implications differ regarding large interitem correlations enhancing or diminishing predictive power...

Omega squared (ω^2) is a measure of effect size for analysis of variance (ANOVA) designs. It is less biased than eta squared, but reported less often. This is in part due to lack of clear guidance on how to calculate it. In this paper, we discuss the logic behind effect size measures, the problem with eta squared, the history of omega squared, and why it has been underused. We then provide a user-friendly guide to omega squared and partial omega squared for ANOVA designs with fixed factors, including one-way, two-way, and three-way designs, using within-subjects factors and/or between-subjects factors...

Several authors have recommended adopting the receiver operator characteristic (ROC) area under the curve (AUC) or mean ridit as an effect size, arguing that it measures an important and interpretable type of effect that conventional effect-size measures do not. It is base-rate insensitive, robust to outliers, and invariant under order-preserving transformations. However, applications have been limited to group comparisons, and usually just two groups, in line with the popular interpretation of the AUC as measuring the probability that a randomly chosen case from one group will score higher on the dependent variable than a randomly chosen case from another group...