Journal of Applied Measurement

The purpose of the present report was to assess congruence between a language-based national examination (termed English placement test - EPT) and the Common European Framework of Reference for Languages (CEFR) levels. To this end, a series of methodological steps were put forth to accumulate evidence suggesting that language performance based on the EPT instrument can be split onto meaningful subgroups based on theoretical (expert judgement on difficulty level and CEFR correspondence) and empirical considerations (i...

A consensus has developed that high-quality teacher evaluation systems require multiple measures. We examine multiple measures from a large urban school district, which has included observational ratings and value-added ratings in its system since 2010. Evaluation systems that do not account for observer severity, classroom context, and other factors may yield different results from systems that do account for these factors. Choosing a simpler system involves a trade-off regarding a system's robustness or defensibility...

The Burn-Specific Pain Anxiety Scale (BSPAS) estimates pain-related anxiety and determines the effect of treatment in patients with burns, especially regarding wound care. This study aimed to analyze the 9-item and the abbreviated 5-item BSPAS by the Rasch model. This prospective study included 161 patients admitted to Dutch burn centres. The BSPAS was administered during hospital stay resulting in 314 self-reports and was analysed using the Rasch unidimensional measurement model 2030 (RUMM 2030). Unidimensionality of the 9-item and 5-item BSPAS was confirmed...

AIM: To explore the psychometric properties of the general movements optimality score (GMOS) by examining its dimensionality, rating scale functioning, and item hierarchies using Rasch measurement. METHODS: Secondary data analysis of the GMOS data for video-recording of 383 infants with uni-, multidimensional, and mixed Rasch partial credit models. Videos were scored based on the global General Movement Assessment categories, and on the amplitude, speed, spatial range, proximal and distal rotations, onset and offset, tremulous and cramped components of the upper and lower extremities (21 items), resulting in the GMOS...

Rasch's logistic model for growth is explained by reviewing his analysis of piglet growth. An early formulation was given in India when Rasch visited with Rao to describe the statistic metameter as the distinguishing characteristic for determining the rate of growth. Next, several examples are given demonstrating growth with plots of growth over time using characteristic time with truncated data. The results of these growth plots and analyses are summarized given their implications and restraints for using this approach in determining rate of growth...

Thanks to the wide range of benefits it provides to teaching and learning, portfolio assessment has maintained widespread popularity in language education over the last few decades. However, the practical use of this assessment method is still subject to debates, particularly about the lack of clear definitions and empirical validations of the constructs underlying the assessment. This problem can be addressed by research into portfolio scale development and examination and this article reports on the process of investigating the psychometric properties of a scale assessing the portfolio-based writing competence of Vietnamese students who speak English as a foreign language (EFL)...

Proposals for incorporating information on the quality of human, social, and environmental conditions in more authentic and comprehensive versions of the Gross National Product (GNP) or Gross Domestic Product (GDP) date back to the foundations of econometrics. Typically treated as external to markets, these domains have lately been objects of renewed interest. Calls for accountability and transparency have expanded to include the now topical but previously neglected economic implications of human, social, and natural capital...

The purpose of this study is to evaluate the quality of observer ratings of neighborhood disorder using a many-facet Rasch model (MFRM). Our goal is to investigate observer severity and observer consistency. Observers trained in the use of a systematic social observation visited and rated residential neighborhoods. Data for this study are drawn from the Family and Community Health Study (FACHS). The FACHS sample consisted of 673 neighborhoods. Two observers, out of a total of 67 observers used for this study, rated each residential neighborhood...

The concept of IRT (Item Response Theory), offering several models which at least guarantee if they hold, that a scoring rule in question is indeed fair, can be referred to in regard to the pertinent scoring rule of the IQ (intelligence quotient) in many intelligence test-batteries. Muller's continuous Rasch model (1987, 1999) applies. Analyses were carried out for three test-batteries, that is the German and the English version of AID 3 and a respective version for group administration, to show this in an exemplary way...

A review of various priors used in Bayesian estimation under the Rasch model is presented together with clear mathematical definitions of the hierarchical prior distributions. A Bayesian estimation method, Gibbs sampling, was compared with conditional, marginal, and joint maximum likelihood estimation methods using the Knox Cube Test data under the Rasch model. The shrinkage effect of the priors on item and ability parameter estimates was also investigated using the Knox Cube Test data. In addition, item response data for a mathematics test with 14 items by 765 examinees were analyzed with the joint maximum likelihood estimation method and Gibbs sampling under the Rasch model...

While youth bullying is a critical public health problem, standardized exposure screening is not routinely practiced. The Child-Adolescent Bullying Scale, (CABS), a psychometrically robust 22-item tool, was designed and evaluated for this purpose using classical test theory. The goals of the present study were to examine and optimize the measurement properties of the CABS using a Rasch psychometric analysis to develop a brief screening tool appropriate for clinical use. A methodologic design and the Rasch rating scale model were employed...

The purpose of this paper is to examine the sensitivity of commonly used Rasch fit measures to different distributions of error in item responses. Using Monte Carlo methods, we generated 10 different measurement error conditions within the Rasch rating scale model or partial credit model, and we recorded the estimates of INFIT MNSQ, OUTFIT MNSQ, and person separation reliability for each error distribution condition. INFIT MNSQ and OUTFIT MNSQ were not sensitive to error distributions when the distribution was the same across items...

This paper presents a methodology for evaluating the judgments of participants of a Completely Structured Training (CST) on the Angoff standard setting method based on the Rasch measurement model. The CST was designed for K-12 teachers on the strategy of judging their students' performance. The evaluation examines judgments for reflecting realistic expectations based on bootstrap sampling distribution of the Rasch measurement model difficulty estimates of the test items. Additionally, the report includes a study application of the methodology and recommendations...

The evaluation of outcomes in mental health care embraces evaluation, quality assurance, and progress measurement of treatments. The Clinical Outcome in Routine Evaluation - Outcome Measure (CORE-OM) is an outcome focused self-assessment instrument, comprising 34 items covering four scales well-being, problems, functioning, and risk. The questionnaire has been translated into 52 languages, including Russian. Despite its broad application, the dimensionality of the CORE-OM deserves some further research. Thus, the present study examines the dimensionality of the Russian CORE-OM using the multidimensional random coefficients multinomial logit model (MRCMLM) based on data of N = 240 patients...

In this study, parameter estimation error was examined when three dimensional tests of a semi-mixed structure were estimated unidimensionally. Since previous studies have generally focused on two-dimensional mixed structured tests or three-dimensional approximately simple structured tests, this study adds to the literature by considering the impact of fitting a unidimensional model to multidimensional data using a test structure that has not previously been considered. Test structure, interdimensional correlation, difficulty of the test, and different underlying distributions of ability were considered...

The purpose of the present paper was twofold: (a) to use 95% confidence intervals of the item and test information functions as a means of visualizing differences between groups on the information provided at the item and test levels, and, (b) to statistically compare item and test information functions as a method for evaluating differential item and differential test functioning. Participants were 2,305 high school students who took a Mathematics National entrance examination in Saudi Arabia. Item and test information functions, conditional standard errors of measurement and reliability were estimated for both males and females...

The aim of this study is to construct a measure of numeracy skill use in the workplace for incarcerated and household adults. The 2012/2014 Programme for the International Assessment of Adult Competencies (PIAAC) Survey of Adult Skills asked about the type and frequency of numeracy tasks performed as part of one's job to nationally-representative incarcerated and household adult samples. This paper takes these items from this survey and focuses on the validation of a measure of numeracy skill use in the workplace using the principles of the Rasch rating scale model...

Tracking changes over time, especially as schools transition to a new operating model, is important to understand the effects of the model on students' perceptions and experiences. An accurate measure of such changes requires a stable measure of item difficulty, which the Rasch model can provide. The University of Chicago Consortium on School Research has applied Rasch analysis to surveys in Chicago Public Schools since 1991; these surveys (in whole or in part) are now used in many non-public schools as well...

The purpose of this research was to investigate the reliability of the scores produced and validity of the inferences drawn from the American Association for the Advancement of Science (AAAS, 2018) force and motion sub-topic assessment for middle school students. The assessment of student outcomes in STEM is an international focus in K-12 education. Project 2061, initiated by the AAAS, focuses on addressing challenges related to standards and assessments. This study informs this effort through testing a 14-item multiple-choice test constructed of questions from the AAAS item bank...

Food insecurity is defined as inadequate access to food due to limited resources. Studies regarding college student food insecurity have shown consistently higher rates than the rest of the nation. Many of these studies measure food insecurity using the United States Department of Agriculture's Adult Food Security Survey Module. Despite its prevalence, the module has not been evaluated for use with the college student population. This study uses Rasch analysis, which underlies the current food insecurity classification approach used by the Department of Agriculture, to investigate the Adult Food Security Survey Module's psychometric properties...