Validity of constructed-response situational judgment tests in training programs for the health professions: A systematic review and meta-analysis protocol

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Background

Situational judgments tests have been increasingly used to help training programs for the health professions incorporate professionalism attributes into their admissions process. While such tests have strong psychometric properties for testing professional attributes and are feasible to implement in high-volume, high-stakes selection, little is known about constructed-response situational judgment tests and their validity.

Methods

We will conduct a systematic review of primary published or unpublished studies reporting on the association between scores on constructed-response situational judgment tests and scores on other tests that measure personal, interpersonal, or professional attributes in training programs for the health professions. In addition to searching electronic databases, we will contact academics and researchers and undertake backward and forward searching. Two reviewers will independently screen the papers and decide on their inclusion, first based on the titles and abstracts of all citations, and then according to the full texts. Data extraction will be done independently by two reviewers using a data extraction form to chart study details and key findings. Studies will be assessed for the risk of bias and quality by two reviewers using the “Quality In Prognosis Studies” tool. To synthesize evidence, we will test the statistical heterogeneity and conduct a psychometric meta-analysis using a random-effects model. If adequate data are available, we will explore whether the meta-analytic correlation varies across different subgroups (e.g., race, gender).

Discussion

The findings of this study will inform best practices for admission and selection of applicants for training programs for the health professions and encourage further research on constructed-response situational judgment tests, in particular their validity.

Trial registration

The protocol for this systematic review has been registered in PROSPERO [ CRD42022314561]. https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022314561.

Related collections

Most cited references 30

Record: found
Abstract: found
Article: not found

Assessing bias in studies of prognostic factors.

Jill Hayden, Danielle van der Windt, Jennifer L Cartwright … (2013)

Previous work has identified 6 important areas to consider when evaluating validity and bias in studies of prognostic factors: participation, attrition, prognostic factor measurement, confounding measurement and account, outcome measurement, and analysis and reporting. This article describes the Quality In Prognosis Studies tool, which includes questions related to these areas that can inform judgments of risk of bias in prognostic research.A working group comprising epidemiologists, statisticians, and clinicians developed the tool as they considered prognosis studies of low back pain. Forty-three groups reviewing studies addressing prognosis in other topic areas used the tool and provided feedback. Most reviewers (74%) reported that reaching consensus on judgments was easy. Median completion time per study was 20 minutes; interrater agreement (κ statistic) reported by 9 review teams varied from 0.56 to 0.82 (median, 0.75). Some reviewers reported challenges making judgments across prompting items, which were addressed by providing comprehensive guidance and examples. The refined Quality In Prognosis Studies tool may be useful to assess the risk of bias in studies of prognostic factors.

0 comments Cited 772 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Disciplinary action by medical boards and prior behavior in medical school.

Maxine A Papadakis, Arianne Teherani, Mary Banach … (2005)

Evidence supporting professionalism as a critical measure of competence in medical education is limited. In this case-control study, we investigated the association of disciplinary action against practicing physicians with prior unprofessional behavior in medical school. We also examined the specific types of behavior that are most predictive of disciplinary action against practicing physicians with unprofessional behavior in medical school. The study included 235 graduates of three medical schools who were disciplined by one of 40 state medical boards between 1990 and 2003 (case physicians). The 469 control physicians were matched with the case physicians according to medical school and graduation year. Predictor variables from medical school included the presence or absence of narratives describing unprofessional behavior, grades, standardized-test scores, and demographic characteristics. Narratives were assigned an overall rating for unprofessional behavior. Those that met the threshold for unprofessional behavior were further classified among eight types of behavior and assigned a severity rating (moderate to severe). Disciplinary action by a medical board was strongly associated with prior unprofessional behavior in medical school (odds ratio, 3.0; 95 percent confidence interval, 1.9 to 4.8), for a population attributable risk of disciplinary action of 26 percent. The types of unprofessional behavior most strongly linked with disciplinary action were severe irresponsibility (odds ratio, 8.5; 95 percent confidence interval, 1.8 to 40.1) and severely diminished capacity for self-improvement (odds ratio, 3.1; 95 percent confidence interval, 1.2 to 8.2). Disciplinary action by a medical board was also associated with low scores on the Medical College Admission Test and poor grades in the first two years of medical school (1 percent and 7 percent population attributable risk, respectively), but the association with these variables was less strong than that with unprofessional behavior. In this case-control study, disciplinary action among practicing physicians by medical boards was strongly associated with unprofessional behavior in medical school. Students with the strongest association were those who were described as irresponsible or as having diminished ability to improve their behavior. Professionalism should have a central role in medical academics and throughout one's medical career. Copyright 2005 Massachusetts Medical Society.

0 comments Cited 160 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Validity: on meaningful interpretation of assessment data.

Susan Downing (2003)

All assessments in medical education require evidence of validity to be interpreted meaningfully. In contemporary usage, all validity is construct validity, which requires multiple sources of evidence; construct validity is the whole of validity, but has multiple facets. Five sources--content, response process, internal structure, relationship to other variables and consequences--are noted by the Standards for Educational and Psychological Testing as fruitful areas to seek validity evidence. The purpose of this article is to discuss construct validity in the context of medical education and to summarize, through example, some typical sources of validity evidence for a written and a performance examination. Assessments are not valid or invalid; rather, the scores or outcomes of assessments have more or less evidence to support (or refute) a specific interpretation (such as passing or failing a course). Validity is approached as hypothesis and uses theory, logic and the scientific method to collect and assemble data to support or fail to support the proposed score interpretations, at a given point in time. Data and logic are assembled into arguments--pro and con--for some specific interpretation of assessment data. Examples of types of validity evidence, data and information from each source are discussed in the context of a high-stakes written and performance examination in medical education. All assessments require evidence of the reasonableness of the proposed interpretation, as test data in education have little or no intrinsic meaning. The constructs purported to be measured by our assessments are important to students, faculty, administrators, patients and society and require solid scientific evidence of their meaning.

0 comments Cited 78 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

Sara Mortaz Hejri:

ORCID: https://orcid.org/0000-0002-9979-0529

Role: ConceptualizationRole: MethodologyRole: Writing – original draftRole: Writing – review & editing

Jordan L. Ho:

ORCID: https://orcid.org/0000-0002-0216-395X

Role: MethodologyRole: Project administrationRole: Writing – original draftRole: Writing – review & editing

Xuan Pan: Role: InvestigationRole: MethodologyRole: Writing – original draft

Yoon Soo Park: Role: ConceptualizationRole: InvestigationRole: MethodologyRole: Writing – review & editing

Amir H. Sam: Role: InvestigationRole: MethodologyRole: Writing – review & editing

Haykaz Mangardich: Role: ConceptualizationRole: InvestigationRole: ResourcesRole: Writing – review & editing

Alexander MacIntosh:

ORCID: https://orcid.org/0000-0002-5094-3774

Role: ConceptualizationRole: SupervisionRole: Writing – review & editing

Somayeh Delavari: Role: Editor

Journal

Journal ID (nlm-ta): PLoS One

Journal ID (iso-abbrev): PLoS One

Journal ID (publisher-id): plos

Title: PLOS ONE

Publisher: Public Library of Science (San Francisco, CA USA )

ISSN (Electronic): 1932-6203

Publication date (Electronic): 26 January 2023

Publication date Collection: 2023

Volume: 18

Issue: 1

Electronic Location Identifier: e0280493

Affiliations

[1 ] Acuity Insights, Toronto, Canada

[2 ] Deloitte, Toronto, Canada

[3 ] Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, United States of America

[4 ] Imperial College School of Medicine, Imperial College London, London, United Kingdom

Iran University of Medical Sciences, ISLAMIC REPUBLIC OF IRAN

Author notes

Competing Interests: YSP, AHS, and JLH have no disclosures to declare. SMH, XP, HM, and AM disclose they are salaried employees of Altus Assessments which administers a situational judgment test called Casper. The authors receive no reimbursements, fees, or funding related to this study or its outcomes. This does not alter our adherence to PLOS ONE policies on sharing data and materials.

* E-mail: amacintosh@ 123456acuityinsights.com

Author information

Sara Mortaz Hejri https://orcid.org/0000-0002-9979-0529

Jordan L. Ho https://orcid.org/0000-0002-0216-395X

Alexander MacIntosh https://orcid.org/0000-0002-5094-3774

Article

Publisher ID: PONE-D-22-08111

DOI: 10.1371/journal.pone.0280493

PMC ID: 9879421

PubMed ID: 36701397

SO-VID: 28e67afe-2915-49b7-b950-49eede738ca5

License:

This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

History

Date received : 18 March 2022

Date accepted : 29 December 2022

Page count

Figures: 0, Tables: 1, Pages: 9

Funding

The authors received no specific funding for this work.

Custom metadata

Data Availability No datasets were generated or analysed during the current study. All relevant data from this study will be made available upon study completion.

Validity of constructed-response situational judgment tests in training programs for the health professions: A systematic review and meta-analysis protocol

Read this article at

Abstract

Background

Methods

Discussion

Trial registration

Related collections

PLOS Climate

Most cited references 30

Assessing bias in studies of prognostic factors.

Disciplinary action by medical boards and prior behavior in medical school.

Validity: on meaningful interpretation of assessment data.

Author and article information

Contributors

Journal

Affiliations

Author notes

Author information

Article

History

Page count

Funding

Categories

Custom metadata

Comments

Comment on this article

Similar content 176

Cited by 2

Most referenced authors 563