REACT: Rapid Evaluation Assessment of Clinical Reasoning

Tool
Brian D. Peterson, MD1 , Charles D. Magee, MD, MPH2 , James R. Martindale, Ph.D.2,
Jessica J. Dreicer, MD2, M. Kathryn Mutter, MD, MPH2 , Gregory Young, MD2,
Melissa Jerdonek Sacco, MD, MS2, Laura C. Parsons, MD, MPH2 ,
Stephen R. Collins, MD, MSc2, Karen M. Warburton, MD2, and Andrew S. Parsons,
MD, MPH2
1
University of Virginia Health, Charlottesville, VA, USA; 2University of Virginia School of Medicine, Charlottesville, VA, USA.

INTRODUCTION: Clinical reasoning encompasses the INTRODUCTION

process of data collection, synthesis, and interpretation
to generate a working diagnosis and make management Clinical reasoning encompasses the process of data collection,
decisions. Situated cognition theory suggests that knowl- synthesis, and interpretation to generate a working diagnosis,
edge is relative to contextual factors, and clinical reason- facilitating management decisions. The bulk of research on
ing in urgent situations is framed by pressure of conse- teaching and assessment focuses on diagnosis, namely the
quential, time-sensitive decision-making for diagnosis process by which a differential diagnosis is generated and
and management. These unique aspects of urgent clinical narrowed through data gathering.1–5 Recently, Cook et al.6
care may limit the effectiveness of traditional tools to as-
described management reasoning as a necessary companion to
sess, teach, and remediate clinical reasoning.
METHODS: Using two validated frameworks, a multidis- diagnostic reasoning, accounting for patient preferences, soci-
ciplinary group of clinicians trained to remediate clinical etal values, logistical constraints, and resource availability
reasoning and with experience in urgent clinical care en- when making testing and treatment decisions for patients.
counters designed the novel Rapid Evaluation Assess- Urgent clinical situations, those in which the patient’s clinical
ment of Clinical Reasoning Tool (REACT). REACT is a condition is rapidly declining, require accelerated decision-
behaviorally anchored assessment tool scoring five do- making with respect to both diagnosis and management.7,8
mains used to provide formative feedback to learners Errors in clinical reasoning among practicing clinicians are
evaluating patients during urgent clinical situations. A
pilot study was performed to assess fourth-year medical
common, estimated to occur in up to 10–15% of hospitalized
students during simulated urgent clinical scenarios. patient encounters.9 Learners who struggle with urgent clinical
Learners were scored using REACT by a separate, multi- situations may be labeled as not recognizing “sick vs not sick”
disciplinary group of clinician educators with no addition- or as lacking in communication skills or clinical knowledge.
al training in the clinical reasoning process. REACT scores Clinical reasoning deficits have been commonly identified
were analyzed for internal consistency across raters and among struggling medical trainees described in single-center
observations. remediation programs.10,11 The University of Colorado re-
RESULTS: Overall internal consistency for the 41 patient
ports that clinical reasoning was the primary deficit in 25–
simulations as measured by Cronbach’s alpha was 0.86.
A weighted kappa statistic was used to assess the overall 30% of residents and 40–45% of medical students referred to
score inter-rater reliability. Moderate reliability was ob- their remediation program over a 6-year period.12 Over a 4-
served at 0.56. year period, the University of Virginia identified that 34% of
DISCUSSION: To our knowledge, REACT is the first tool learners referred to a Graduate Medical Education (GME)
designed specifically for formative assessment of a learner’s remediation program struggled with clinical reasoning.13 A
clinical reasoning performance during simulated urgent true estimate of prevalence data is difficult to establish as
clinical situations. With evidence of reliability and content
validated clinical reasoning assessment tools have
validity, this tool guides feedback to learners during high-
risk urgent clinical scenarios, with the goal of reducing limitations.14
diagnostic and management errors to limit patient harm. Dual-process theory is a commonly understood cognitive
model for clinical reasoning wherein decision-making occurs
J Gen Intern Med 37(9):2224–9 through a combination of system 1 (heuristic processes) and
DOI: 10.1007/s11606-022-07513-5
system 2 (analytical processes).15 Urgent clinical situations are
© The Author(s) under exclusive licence to Society of General Internal
Medicine 2022
contextualized well with dual-process theory, requiring heu-
ristics and efficient analytic reasoning for time-sensitive diag-
Received September 15, 2021
nosis as well as rapid assessment, stabilization, and manage-
Accepted March 25, 2022 ment prior to the determination of a diagnosis. To improve
Published online June 16, 2022 clinical reasoning in urgent situations, algorithms for specific
2224
JGIM Peterson et al.: REACT: Rapid Evaluation Assessment of Clinical Reasoning Tool 2225

clinical scenarios have been developed to facilitate manage- reasoning. This group, composed of primary care and
ment reasoning and improve patient outcomes. For example, specialist clinician educators from internal and hospital
the Advanced Cardiovascular Life Support algorithms guide medicine, emergency medicine, pediatrics, anesthesiology,
management in “code” situations and offer a method for family medicine, critical care, and obstetrics and gynecol-
analytical diagnosis while the patient is being resuscitated ogy, met monthly to review struggling learners, discuss
(e.g., the H’s and T’s).16 However, the majority of urgent best practices in clinical reasoning assessment and reme-
clinical situations do not reach this final common pathway, diation, and review the clinical reasoning literature.18,19
and as such, require nuanced clinical reasoning without aid The subcommittee serves as a pool of clinical reasoning
from established algorithms for management and diagnosis. coaches, available as needed, to provide one-on-one
Situated cognition theory (SCT) provides an attractive optic coaching to struggling trainees and students. To address
for the assessment of clinical reasoning performance relative an identified need, the group began a joint effort in late
to the myriad interacting factors impacting formative evalua- 2020 to design an evidence-based tool to assess and pro-
tion in urgent clinical situations. Rencic et al.17 proposed a vide formative feedback to learners during urgent patient
conceptual framework that considers six clinical reasoning care situations.
performance assessment elements: the clinician or assessee, The REACT tool (Fig. 1) was designed by a multidisciplin-
patient, rater, assessment method, task, and environment. ary group of clinician educators from the COACH subcom-
Through this conceptual lens, direct observation of clinical mittee with expertise in teaching and assessing clinical rea-
reasoning performance offers the most authentic assessment, soning across both undergraduate medical education (UME)
but requires rigor to identify and manage the numerous and GME. REACT was named to represent the rapid patient
interacting factors that influence clinical reasoning perfor- evaluation required during urgent clinical scenarios. The
mance. Assessment of clinical reasoning in urgent clinical group met on four occasions to design the tool, first identifying
situations, characterized by high acuity or decompensation of evidence-based domains of diagnosis and management rea-
a patient’s clinical status, is particularly challenging due to the soning specific to urgent patient care situations and then
unplanned nature and multitude of uncontrollable factors that associating a range of behavioral anchors with each domain.
may have distracting or detrimental effects, including the The group began with two validated frameworks: (1) the
subjugation of educational goals for the urgent provision of Society to Improve Diagnosis in Medicine’s (SIDM) Assess-
care. A simulated patient encounter that offers an urgent ment of Reasoning Tool (ART) to assess clinical reasoning
patient care situation in a high-fidelity simulation environment during oral presentations and (2) the Association of American
therefore offers an ideal surrogate model, allowing for control Medical College’s (AAMC) Entrustable Professional Activity
of many factors across each of the six assessment elements. 10 (EPA 10) designed to formatively assess a learner’s recog-
Herein, we describe a novel tool for formative assessment nition of patients requiring urgent care. A priority of the group
and feedback of learner performance during urgent clinical was to design a tool applicable to learners in a variety of
situations that require rapid, time-sensitive diagnostic and clinical settings consistent with patient care provided by mul-
management reasoning. This behaviorally anchored tool, tiple specialties.
known as REACT (Rapid Evaluation Assessment of Clinical Thammasitboon et al.20 and SIDM developed and validated
Reasoning Tool), was designed by content experts based on the Assessment of Reasoning Tool (ART) to facilitate clinical
domain-specific frameworks to guide feedback to learners teaching for oral presentations and clinical reasoning, specif-
during high-risk urgent clinical scenarios, with the goal of ically assessing the learner’s proficiency in the domains of
reducing diagnostic and management errors. data gathering, interpretation, synthesis, and metacognition.
This behaviorally anchored tool provides a general framework
for assessing and correcting errors in clinical reasoning. Ob-
serving a learner in the context of a clinical scenario allows for
METHODS the assessment of nonverbal and tonal cues, which vary de-
In 2016, the Committee on Seeking Competence through pending on situational stressors.21 In contrast, the AAMC
Help (COACH) was formed. COACH is a unique peer designed a toolkit for assessing a range of clinical competen-
support program at the University of Virginia (UVA) cies among learners in real-world settings. These core compe-
aimed to help medical learners who are referred for, or tencies, termed EPAs, include validated proficiencies expect-
who request, help with clinical performance. Since its ed of medical students prior to starting residency.22 EPA 10
creation, COACH has worked with more than 100 trainees focuses on “recognizing a patient requiring urgent or emergent
in 14 different departments with generally positive out- care and initiating evaluation and management.” This toolkit
comes. Subsequently, UVA School of Medicine imple- introduces a standard set of behaviors expected in the man-
mented a clinical remediation program employing the agement of urgent clinical scenarios.
same framework and much of the same personnel. In A pilot study was performed at UVA during the 2021 Intern
2018, a subcommittee was formed to focus on strategies Readiness Course (IRC) for 87 fourth-year medical students
to identify and coach learners who struggle with clinical who are preparing to transition into internal medicine,
2226 Peterson et al.: REACT: Rapid Evaluation Assessment of Clinical Reasoning Tool JGIM

Figure 1 Rapid Evaluation Assessment of Clinical Reasoning Tool (REACT).

psychiatry, family medicine, emergency medicine, or anesthe- The UVA Institutional Review Board reviewed this project
sia residencies. A core goal of the IRC is to provide fourth- and determined that it met the criteria for exempt review (ref #
year medical students with the skills to appropriately respond 4234).
to and manage common urgent clinical situations such as
hypotension, chest pain, hypoxemia, or altered mental status.
Much of this education is accomplished through simulated
scenarios with manikins. Each student plays the role of an RESULTS
intern in at least two unique case scenarios, and the scenario is REACT is comprised of four learner functions essential to the
curated by a nurse with relevant clinical experience. At the clinical reasoning process during urgent patient care: data
conclusion of the simulation, students are immediately led collecting, interpreting, managing, and communicating. A
through debriefing exercises with a clinician who directly fifth learner function, reflecting, highlights the centrality of
observed the simulation. The simulations for the 2021 IRC metacognition in effective clinical reasoning.25 For each func-
were recorded. Table 1 provides a description of each case. An tion, specified tasks are described. A range of exemplar be-
independent, multidisciplinary group of clinicians with no havioral anchors are described for each function to allow for
additional training in the clinical reasoning process observed formative feedback.
these recordings and scored each medical student’s perfor- Seven raters comprising clinicians from internal and hospi-
mance using REACT. A scoring system was added to the tal medicine, obstetrics and gynecology, pediatric critical care,
behavioral anchors in order to analyze the tool’s performance emergency medicine, and anesthesiology scored 41 recorded
for internal consistency across raters and observations. RE- case scenarios representing 41 individual students. Each rater
ACT scores were generated using a 3-point scale for each scored the same 41 case scenarios in full. Internal consistency
behavioral domain, with a maximum total score of 15 and a as assessed by Cronbach’s alpha was measured for the
minimum score of 5. summed overall rating (score) for the 41 video clips and was
Determination of optimal sample size for the study utilized .86, a value considered sufficient for high-stakes assessment.26
estimates based on minimizing measurement error, both in the Due to the ordinal nature of the ratings, a weighted kappa
number of observations and the number of raters used.23 statistic was used to measure inter-rater reliability which for
Cronbach’s alpha was used to assess the internal consistency overall ratings was .56, generally interpreted as a moderate
across the group of raters. Inter-rater reliability for the overall degree of agreement.27 Supplemental Table 1 provides
rating score among the group of raters was assessed using the domain-specific weighted kappa data and Supplemental Ta-
weighted kappa statistic. All analyses were performed using ble 2 provides descriptive statistics of domain-specific scores
SPSS v28.24 for individual raters.
JGIM Peterson et al.: REACT: Rapid Evaluation Assessment of Clinical Reasoning Tool 2227

Table 1 Description of Clinical Cases Requiring Rapid Evaluation

Case Patient description Objectives

Asthma exacerbation 67-year-old female with sudden onset shortness of • Recognize tachycardia, tachypnea, and hypoxia
breath • Identify asthma exacerbation as a most likely diagnosis
• Initiate management for asthma exacerbation
Ruptured ectopic 28-year-old female with nausea, vomiting, and lower • Demonstrate an organized approach to a patient with
pregnancy abdominal pain hypotension
• Recognize ectopic pregnancy as a possible cause of
abdominal pain and hypotension
• Call for obstetric consultation and initiate hypovolemic shock
management
Myocardial infarction 72-year-old female with slight pressure in her • Recognize acute coronary syndrome may present atypically in
epigastrium female patients
• Obtain EKG and call “STEMI” alert
• Initiate management of acute coronary syndrome
Transfusion reaction 45-year-old male with dizziness, nausea, abdominal • Recognize hypoxia, hypotension, tachycardia, and fever as
pain, and shortness of breath possible reactions to transfusion
• Stop the blood transfusion and initiate management of
possible transfusion reaction
Anaphylaxis 70-year-old female with dizziness, shortness of • Recognize hypotension and tachycardia
breath, and pruritis • Identify anaphylaxis as a potential etiology with recent
antibiotic administration
• Initiate a care plan for the decompensating patient
Septic shock 76-year-old male with altered mental status • Recognize fever, hypotension, tachycardia, and altered mental
status
• Prioritize septic shock as the most likely diagnosis
• Initiate stabilization management for septic shock
Cardiac arrest 57-year-old male with shortness of breath and chest • Create a differential for acute chest pain and shortness of
pain breath
• Recognize PEA arrest and create a differential for the causes
• Initiate management for cardiac arrest
COVID-19 pneumonia 65-year-old female with cough and fever • Recognize symptoms of hypoxia and fever as potential viral
pneumonia
• Initiate management of worsening hypoxia
Heart failure exacerbation 58-year-old male with hypotension and chest pain • Demonstrate an organized approach to a patient with
hypotension
• Collect an organized history to determine potential causes
• Initiate appropriate management for hypotension
Hypertensive emergency 52-year-old male with confusion • Demonstrate an organized approach to a patient with altered
mental status
• Recognize hypertensive emergency and hypertensive
encephalopathy
• Initiate management of hypertension and recognize risks of
rapid blood pressure reduction
Acute alcohol withdrawal 52-year-old male with agitation • Recognize alcohol withdrawal syndrome and ensure patient
syndrome and staff safety
• Demonstrate understanding of behavioral emergency
medications
• Initiate management of alcohol withdrawal syndrome
Hypoglycemic seizure 46-year-old female with altered mental status • Demonstrate an organized approach to a patient with altered
mental status
• Consider hypoglycemia on the differential
• Initiate glucose replacement therapy
EKG electrocardiogram
STEMI ST elevation myocardial infarction
PEA pulseless electrical activity

DISCUSSION clinical reasoning performance in simulated urgent clinical

To our knowledge, REACT is the first tool specifically de- situations. This was notably achieved with no additional rater
signed for formative assessment of a learner’s clinical reason- training or standard setting and among a population of clini-
ing performance during simulated urgent clinical situations. cian educators from multiple medical specialties. This finding
Built on the strength of validated instruments specific to is a particularly intriguing observation in contrast to guidelines
clinical reasoning and clinical urgency, REACT was thought- informing best practice for direct observation of clinical skills
fully designed by a multidisciplinary group of clinician edu- in medical education, recommending both rater and frame of
cators with expertise in teaching and assessing clinical reason- reference training.28
ing across the spectrum of UME and GME education. This SCT indicates that numerous potential variables may influ-
approach provides evidence of content validity and our anal- ence clinical reasoning performance in urgent clinical situa-
ysis demonstrates both moderate inter-rater reliability and a tions. Variables include those intrinsic to the clinician such as
high degree of internal consistency of REACT to assess years of experience or training, as well as variables intrinsic to
2228 Peterson et al.: REACT: Rapid Evaluation Assessment of Clinical Reasoning Tool JGIM

the patient, the rater, the clinical reasoning task, and the necessary to determine the broadest application for this for-
environment.17 For example, urgent clinical encounters can mative assessment instrument.
in part be defined by the need for early management and an
accelerated response to dynamic information. Although these Corresponding Author: Andrew S. Parsons, MD, MPH; University of
Vi rg i n i a S c h o o l o f M e d i c i n e , C h a r l o t te s v i l l e , VA , U S A
variables exist uncontrolled in authentic urgent clinical situa- (e-mail: [email protected]).
tions, implementation of REACT in a simulated urgent clinical
situation affords an environment in which control over many
Supplementary Information The online version contains supple-
of these variables generates an opportunity to isolate and mentary material available at https://doi.org/10.1007/s11606-022-
measure variables of interest. 07513-5.
REACT, in fact, was not designed to directly assess the
myriad relationships and interactions between variables that
Declarations:
might affect the performance of clinical reasoning in urgent
clinical scenarios. Rather, REACT is focused on the empiric Conflict of Interest: The authors have no conflicts of interest to
tasks essential to the formative assessment of clinical reason- disclose.
ing performance in urgent clinical situations. It is therefore our
hypothesis that REACT may not perform as well in authentic,
real-life urgent clinical situations where the rapid interactions
may at times compete with the educational goals, threatening REFERENCES
the ability to achieve educational goals and objectives. In 1. Gruppen LD. Clinical Reasoning: Defining It, Teaching It, Assessing It,
Studying It. West J Emerg Med 2017;18(1):4-7.
simulated urgent clinical situations, where educational goals 2. Geoff R, Norman CPMvdV, David I Newble, Diana H.J.M Dolmans, et al.
maintain primacy, the numerous factors surrounding the cli- International Handbook of Research in Medical Education. Dordrecht,
the Netherlands: Kluwer Academic Publishers, 2002.
nician, patient, rater, assessment method, task, and environ- 3. Norman G. Research in clinical reasoning: past history and current
ment are all controllable and modifiable. In a limited way, trends. Med Educ 2005;39(4):418-27.
REACT provides a tool to isolate and study variance attribut- 4. Norman GR, Eva KW. Diagnostic error and clinical reasoning. Med Educ
2010;44(1):94-100.
able to individual factors, creating an understanding of how 5. Patel JJ, Bergl PA. Diagnostic vs Management Reasoning. JAMA
each is associated with clinical reasoning performance. Such 2018;320(17):1818.
6. Cook DA, Sherbino J, Durning SJ. Management Reasoning: Beyond the
analysis may inform other potential relationships and interac-
Diagnosis. JAMA 2018;319(22):2267-68.
tions in non-urgent clinical environments. 7. Lighthall GK, Vazquez-Guillamet C. Understanding Decision Making in
Future directions include the study of REACT as a tool to Critical Care. Clin Med Res 2015;13(3-4):156-68.
8. Ozel F. Time pressure and stress as a factor during emergency egress.
predict the need for remediation among different levels of Safety Science 2001;38(2):95-107.
learners and as a summative assessment tool before and after 9. Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in
medicine. Am J Med 2008;121(5 Suppl):S2-23.
clinical reasoning coaching. This includes the creation of a
10. Yao DC, Wright SM. National survey of internal medicine residency
quantitative system with performance thresholds for scoring in program directors regarding problem residents. JAMA 2000;284(9):1099-
addition to the current behavioral anchors. Friedman et al.29 104.
11. Dupras DM, Edson RS, Halvorsen AJ, Hopkins RH, Jr., McDonald FS.
found that residents are more often over-confident in their "Problem residents": prevalence, problems and remediation in the era of
diagnoses and more prone to errors in diagnostic reasoning. core competencies. Am J Med 2012;125(4):421-25.
In contrast, medical students made more diagnostic errors but 12. Guerrasio J, Garrity MJ, Aagaard EM. Learner deficits and academic
outcomes of medical students, residents, fellows, and attending physi-
had the least confidence; attending physicians made mistakes, cians referred to a remediation program, 2006-2012. Acad Med
albeit with the highest confidence and accuracy. These find- 2014;89(2):352-8.
13. Warburton KM, Shahane AA. Mental Health Conditions Among Strug-
ings suggest that opportunities for improved clinical reasoning gling GME Learners: Results From a Single Center Remediation Program.
exist throughout the UME and GME continuum and beyond. J Grad Med Educ 2020;12(6):773-77.
Future studies may also employ faculty development for raters 14. Lessing JN, Rendon P, Durning SJ, Roesch JJ. Approaches to Clinical
Reasoning Assessment. Acad Med 2020;95(8):1285.
on the basics of diagnostic and management reasoning. 15. Monteiro SM, Norman G. Diagnostic reasoning: where we've been, where
we're going. Teach Learn Med 2013;25 Suppl 1:S26-32.
16. American-Heart-Association. Advanced cardiovascular life support: Pro-
vider Manual. Dallas, Texas: 2016.
17. Rencic J, Schuwirth LWT, Gruppen LD, Durning SJ. Clinical reasoning
CONCLUSION performance assessment: using situated cognition theory as a conceptual
framework. Diagnosis (Berl) 2020;7(3):241-49.
REACT is a novel tool designed for formative assessment and 18. Parsons A, Warburton K. A novel clinical reasoning coaching program for
feedback of a learner’s clinical reasoning performance during the medicine learner in need. MedEdPublish, ed. 2019.
19. Parsons AS, Clancy CB, Rencic JJ, Warburton KM. Targeted Strategies to
urgent clinical situations. This tool can reliably serve as a
Remediate Diagnostic Reasoning Deficits. Acad Med 2021.
guide to clinician educators in their assessment of learners 20. Thammasitboon S, Rencic JJ, Trowbridge RL, Olson APJ, Sur M,
and may assist in the identification of learners who struggle Dhaliwal G. The Assessment of Reasoning Tool (ART): structuring the
conversation between teachers and learners. Diagnosis (Berl)
with clinical reasoning skills in high-risk urgent clinical sce- 2018;5(4):197-203.
narios, with the goal of reducing diagnostic and management 21. Ende J. Feedback in clinical medical education. JAMA 1983;250(6):777-
errors. A study of validity in other learner populations is 81.
JGIM Peterson et al.: REACT: Rapid Evaluation Assessment of Clinical Reasoning Tool 2229

22. Obeso V BD, Aiyer M, Barron B, Bull J, et al. Core Entrustable 26. Bland JM, Altman DG. Cronbach's alpha. BMJ 1997;314(7080):572.
Professional Activities for Entering Residency—EPA 10 Schematic: Rec- 27. Landis JR, Koch GG. The measurement of observer agreement for
ognize a Patient Requiring Urgent or Emergent Care and Initiate categorical data. Biometrics 1977;33(1):159-74.
Evaluation and Management. Washington, DC: Association of American 28. Kogan JR, Hatala R, Hauer KE, Holmboe E. Guidelines: The do's, don'ts
Medical Colleges; 2017. and don't knows of direct observation of clinical skills in medical
23. Gwet KL. Handbook of Inter-Rater Reliability. 4th ed. Advanced Analytics education. Perspect Med Educ 2017;6(5):286-305.
LLC: 2014. 29. Friedman CP, Gatti GG, Franz TM, et al. Do physicians know when their
24. IBM-Corp. IBM SPSS Statistics for Windows Version 28. 28 ed. Armonk, diagnoses are correct? Implications for decision support and error
NY: IBM Corp; 2021. reduction. J Gen Intern Med 2005;20(4):334-39.
25. Marcum JA. An integrated model of clinical reasoning: dual-process
theory of cognition and metacognition. J Eval Clin Pract 2012;18(5):954- Publisher’s Note Springer Nature remains neutral with regard to
61. jurisdictional claims in published maps and institutional affiliations.