Received: 27 December 2023 | Revised: 28 February 2024 | Accepted: 4 March 2024

DOI: 10.1002/aet2.10974

ORIGINAL CONTRIBUTION

Better together: A multistakeholder approach to developing

specialty-­wide entrustable professional activities in emergency
medicine

Holly A. Caretta-­Weyer MD, MHPE1 | Stefanie S. Sebok-­Syer PhD1 |

1 2
Amanda M. Morris MA | Benjamin H. Schnapp MD, MEd | Abra L. Fant MD, MS3 |
Kevin R. Scott MD, MSEd4 | Matthew Pirotte MD5 | Michael A. Gisondi MD1 |
6
Lalena M. Yarris MD, MCR

1
Department of Emergency Medicine,
Stanford University School of Medicine, Abstract
Palo Alto, California, USA
Purpose: Entrustable professional activities (EPAs) are a widely used framework for
2
Berbee Walsh Department of Emergency
Medicine, University of Wisconsin,
curriculum and assessment, yet the variability in emergency medicine (EM) training
Madison, Wisconsin, USA programs mandates the development of EPAs that meet the needs of the specialty as
3
Department of Emergency Medicine, a whole. This requires eliciting and incorporating the perspectives of multiple stake-
Northwestern University Feinberg School
of Medicine, McGaw Medical Center at holders (i.e., faculty, residents, and patients) in the development of EPAs. Without a
Northwestern University, Chicago, Illinois, shared understanding of what a resident must be able to do upon graduation, we run
USA
4 the risk of advancing ill-­prepared residents that may provide inconsistent care.
Department of Emergency Medicine,
University of Pennsylvania Perelman Methods: In an effort to address these challenges, beginning in February 2020, the
School of Medicine, Philadelphia,
authors assembled an advisory board of 25 EM faculty to draft and reach consensus
Pennsylvania, USA
5
Department of Emergency Medicine, on a final list of EPAs that can be used across all training programs within the spe-
Vanderbilt University Medical Center, cialty of EM. Using modified Delphi methodology, the authors came to consensus on
Nashville, Tennessee, USA
6 an initial list of 22 EPAs. The authors presented these EPAs to faculty supervisors,
Department of Emergency Medicine,
Oregon Health & Science University, residents, and patients for refinement. The authors collated and analyzed feedback
Portland, Oregon, USA
from focus groups of residents and patients using thematic analysis. The EPAs were
Correspondence subsequently refined based on this feedback.
Holly A. Caretta-­Weyer, MD, MHPE,
Results: Stakeholders in EM residency training endorsed a final revised list of 22
Department of Emergency Medicine,
Stanford University School of Medicine, EPAs. Stakeholder focus groups highlighted two main thematic considerations that
900 Welch Road, Suite 350, Palo Alto, CA
helped shape the finalized list of EM EPAs: attention to the meaningful nuances of
94304, USA.
Email: hcweyer@[Link] EPA language and contextualizing the EPAs and viewing them developmentally.
Conclusions: To foreground all key stakeholders within the EPA process for EM, the
Funding information
American Medical Association authors chose within the development process to draft; come to consensus; and refine
EPAs for EM in collaboration with relevant faculty, patient, and resident stakehold-
ers. Each stakeholder group contributed meaningfully to the content and intended

Holly A. Caretta-­Weyer and Stefanie S. Sebok-­Syer are co–first authors.

Presented at the biannual meeting of the Ottawa Assessment Conference, Lyon, France, August 2022; and the Council of Residency Directors in Emergency Medicine Academic
Assembly, Las Vegas, NV, March 2023.

Supervising Editor: Stephen John Cico

This study was approved by the Stanford University Institutional Review Board on October 28, 2019 (File No. 51828).

AEM Educ Train. 2024;8:e10974. [Link]/journal/aet2 © 2024 Society for Academic Emergency Medicine. | 1 of 8
[Link]
2 of 8 | A MULTISTAKEHOLDER APPROACH TO DEVELOPING SPECIALTY-­WIDE EPAs IN EM

implementation of the EPAs. This process may serve as a model for others in develop-
ing stakeholder-­responsive EPAs.

I NTRO D U C TI O N EPAs using consensus methods. Additionally, to integrate stake-

holder perspectives to ensure relevant outcomes for all, we authen-
Entrustable professional activities (EPAs) are units of professional ticated our EPAs with a group of primary stakeholders, including
practice that are observable and can be entrusted to an individ- residents, faculty, and patients.
ual with a specified level of supervision.1 They are widely used
as a framework for curriculum development and assessment in
competency-­based medical education (CBME). CBME requires five M E TH O D S
core components as part of its organizing framework. These core
components include an outcomes competency framework, progres- Study design
sive sequencing of competencies, tailored learning experiences,
competency-­focused instruction, and programmatic assessment. 2 Using the core components framework for CBME, 2 this study in-
The EPA framework allows frontline faculty to assess components volved four distinct, but interconnected, phases: (1) a modified
of everyday practice while also facilitating the integration of com- Delphi to develop EPAs for EM; (2) feedback from patients; (3) re-
petencies and milestones to allow programs to track progression. finement of EPAs from the EM resident perspective; and (4) feed-
EPAs additionally provide the scaffolding for directed feedback that back from faculty for consolidation and synthesis to finalize EPAs.
facilitates coaching and individualized learning while also provid- The goal of this work was to develop EPAs that can be used by all EM
ing structure around which to build out programmatic assessment. residency programs within the United States regardless of program
Many specialties across graduate medical education in the United length (3-­year and 4-­year) or training setting (university, county, and
States have developed or are in the process of developing their own community hospital based). These EPAs reflect activities that resi-
specialty EPAs.3–6 Despite the widespread incorporation of EPAs dents can routinely perform with varying levels of supervision from
into programmatic assessment in other fields, emergency medi- day one of residency until they graduate and transition to unsuper-
cine (EM) programs have yet to incorporate a shared framework of vised practice. The institutional review board of Stanford University
specialty-­outcome EPAs for workplace-­based assessment. While a approved this study as exempt (Protocol #51828).
previous set was drafted, these remain unimplemented due to their
significant breadth and lack of ease in translation and applicability
to workplace-­based assessments.7 Without a shared vision of out- Modified Delphi phase
comes within the specialty of EM for what we should be teaching
and assessing, resident entrustability will be highly variable and pa- In February 2020, we convened an advisory board of 25 EM faculty
tients will receive inconsistent care. representing the following geographical regions in the United States:
The variability in EM programs and training settings mandate Northwest, Southwest, Midwest, Northeast, and Southeast. These
the development of EPAs that are designed to meet the needs of 14 men and 11 women represented both 3-­ and 4-­year EM residency
the specialty as a whole. To be relevant, the perspectives of multi- programs and the full spectrum of EM training and practice envi-
ple stakeholders (i.e., faculty supervisors, residents, and patients/ ronments including university, county, and community hospital set-
families) must be considered in EPA development.8,9 CBME is both tings. Our board included those with residency program leadership
a learner-­centered and a patient-­focused paradigm. Given these roles, ACGME Milestone development experience, and oversight
dual foci, the development of EPAs must take both perspectives in EM program accreditation. Initial members were selected based
into account. However, the vast majority of EPA development pro- on their expertise or roles within EM organizations or on specific
cesses focus exclusively on the faculty perspective of what must projects such as Milestone development or work on the Residency
be accomplished within specialty training.10–15 To ensure societally Review Committee. The remainder were recruited via snowball sam-
relevant outcomes for trainees, the patient perspective is required pling with attention to representation as noted above. A subset of 10
throughout CBME development and implementation, including the members of our advisory board, including the members representa-
crafting of the EPA framework. As learners are required to meet tive of the ACGME and EM program oversight, drafted EPAs that
those outcomes to progress through training, it is also necessary responded to the following question: What should an EM resident
to ascertain whether these outcomes are consistent with residents’ be able to do to graduate and enter into unsupervised practice? For
growth and development throughout their graduate medical educa- guidance, they used previously drafted EM EPAs (six),7 the Canadian
16–18
tion training. EM EPAs,19 AAMC Core EPAs, 20,21 and the ACGME Milestones 2.0
As part of our broader aim of implementing CBME within EM for EM. 22 This process yielded 60 EPAs, which were put forth in the
training in the United States, we sought to develop EM specialty initial voting questionnaire to the entire advisory board.
CARETTA-­WEYER et al. | 3 of 8

The Delphi method is a systematic approach used to gather family members could share ideas and benefit from hearing others'
23–25
consensus among a group of individuals. On March 6, 2020, 18 experiences and perspectives. Eight participants consented to par-
members of the advisory board met in New York to conduct a Delphi ticipate in the focus group during which they reviewed the EPAs
panel; the other seven participated remotely via Zoom. There were and offered their feedback regarding whether these EPAs captured
a total of three rounds performed as detailed below, each of which their expectations for EM providers. Patients and their families also
was followed by an anonymous vote, tallying of results, and review shared what they hoped EM residents would be learning through-
of comments prior to the next round of voting. out their residency training. We analyzed these data using inductive
The panel decided that an EPA would be adopted in the process thematic analysis.26 The focus group was transcribed verbatim and
if there was 80% agreement among the group. For the first round deidentified prior to analysis. After reading through the transcript
of voting, we asked panelists to vote on 60 EPAs that resulted from to become familiar with the data, two members of the team (SSS
the initial drafting. They did so anonymously and prior to arriving for and AMM) independently coded the data. The two raters worked in
the discussion, consistent with the Delphi process. We additionally parallel and then looked for commonalities. After developing a list
encouraged them to write in any EPAs they felt were missing from of codes, we searched for themes from the list of identified codes.
the initial draft set of EPAs. No additional EPAs were added as part We then reviewed, defined, and named these themes. We did not
of the first round. The panel then engaged in discussion and debate assess for a level of agreement given the intent was not a rigorous
about the EPAs after the first round ended. Twenty-­five EPAs moved qualitative analysis but simply to identify themes to iterate upon the
on from the first round. Of these, 12 met the automatic 80% thresh- developed EPAs.
old for adoption and 13 were advanced for further group discussion
for the next round vote. The others were discarded as the panelists
determined they would not meet the threshold based on the group Resident focus groups
discussion. Once all panel members were given an opportunity to
provide feedback, participants completed a second round of voting From January 28, 2021, to February 11, 2021, we recruited and
where they were asked to anonymously vote on the 13 remaining conducted three separate focus groups with residents across all
EPAs that were not automatically adopted in the first round. Once postgraduate years from six different EM programs throughout the
again, discussion and debate occurred around the remaining EPAs. United States (n = 9). These residents were recruited from a variety
The EPAs that generated the most discussion were ones in which of programs (3-­year vs. 4-­year, geographically variable across the
the panel could not agree that all training programs had opportu- United States, and representing a variety of patient care settings)
nities for residents to perform the activities, such as “accept and by an education researcher (SSS) who does not practice clinically.
transfer patients between health systems.” There was also signifi- During these focus groups, residents were shown the list of 22 EPAs
cant discussion about what constituted a concrete “activity” for the and asked the following questions: (1) To what extent do these EPAs
purposes of creating an EPA (e.g., is it possible to be fully entrustable encompass the knowledge, skills, and abilities of an EM physician?
in “ultrasound” or “communication”?). There was a final round of vot- What is missing or what would you change? (2) Where are each of
ing requested by the group that resulted in consensus around 22 these EPAs learned, practiced, and assessed? What are some of the
EPAs for the specialty of EM. In the following 2 months, the advisory challenges in assessing these activities? During these focus groups,
board members edited the 22 EPAs for clarity and language. They which ran for an average of 64 min, the education researcher who
additionally developed descriptions of key features expected of a moderated the sessions (SSS) also shared the named themes and
learner, preliminary subcompetency mapping, and recommended definitions from the patient and family focus group to elicit reac-
assessment tools for each of the EPAs (Data S1). tions and responses from residents. We conducted a thematic analy-
sis using the same approach described in the Patient and Families
Advisory Council section.
Stakeholder authentication phase

Patient and families advocacy council focus groups Faculty feedback sessions

On January 14, 2021, SSS presented the list of 22 EPAs to the EM Throughout August 11, 2021, to January 24, 2022, a group of seven
Patient and Families Advocacy Council at XX Health Care and in- EM faculty members (BS, AF, KS, MP, MAG, LY, HCW) representing
vited individuals to voluntarily participate in a focus group around different residency programs throughout the United States met on
the EPAs. This group was selected due to access to the study team four separate occasions to review and refine the EPAs. These indi-
and willingness to participate. This group is representative of the pa- viduals again represent 3-­ and 4-­year training programs, variable ge-
tient population served as ensured by the institution given the work ographic settings, and different practice settings. In these feedback
that they traditionally perform around quality improvement initia- sessions, SSS presented the list of 22 EPAs developed by the advi-
tives and provision of feedback about core patient care and service sory board. Participants shared their initial reactions and considered
initiatives. We chose a focus group method so that patients and their what assessment scales might be used to assess each of the EPAs.
4 of 8 | A MULTISTAKEHOLDER APPROACH TO DEVELOPING SPECIALTY-­WIDE EPAs IN EM

TA B L E 1 Final EPAs for the specialty of EM.

Then, SSS presented themes from both the patient and the fam-
ily and resident focus groups for discussion; this led to refinement 1. Initiate treatment for a patient requiring emergent/immediate
intervention.
in language of some EPAs as well as conversation about whether
2. Lead the resuscitation of a critically ill or injured patient.
any additional EPAs should be added. It also led to refinement of
the scales used to assess residents within the EPA framework. The 3. Obtain and interpret a focused history using data from all
necessary sources.
final EPAs after all rounds of refinement were re-­presented to the
4. Perform and interpret a focused physical examination.
25-­member advisory board for final review and feedback.
5. Create and prioritize a differential diagnosis.
6. Order and interpret diagnostic tests.

Consolidation and synthesis phase 7. Apply best available evidence to guide patient care.
8. Manage clinical or diagnostic uncertainty when caring for
Data from all the feedback phases underwent final thematic analy- patients.

sis where SSS and HCW produced a summary of the final list of 22 9. Utilize observation and reassessment to guide decision making.

EPAs; the language and content of the EPAs represented the feed- 10. Develop and implement an appropriate disposition and aftercare
back from the different stakeholder groups. The study team consist- plan.

ing of the author group provided final approval of the 22 EPAs after 11. Perform the diagnostic and therapeutic procedures of an
emergency physician.
all of the above rounds of feedback. An EM assessment research
12. Provide invasive and noninvasive airway management.
expert (HCW) and a PhD research scientist (SSS) led the study team,
which also included a communication science and qualitative re- 13. Perform and interpret point-­of-­c are ultrasound.

search expert (AMM) and six EM clinical faculty (BS, ALF, KS, MP, 14. Perform procedural sedation.

MAG, LY) which brought expertise in education and administration. 15. Implement pharmacologic and therapeutic management plans.
16. Provide palliative and end-­of-­life care for patients and their
families.

R E S U LT S 17. Document the EE encounter.

18. Communicate with other health care professionals about patient
We engaged all stakeholders in the process of developing and refin- care.

ing the content of the EM EPAs. Through analysis of data collected 19. Communicate with the patient, family, and caregivers.
from patients, residents, and faculty, we finalized the list of 22 EPAs 20. Provide supervision or consultation for other health care
for EM (see Table 1). We identified meaningful nuances in EPA lan- professionals.

guage, as well as the importance of contextualizing EPAs develop- 21. Manage the ED flow to optimize patient care.

mentally, through eliciting stakeholders’ feedback. Specific EPAs are 22. Fulfill professional obligations and adhere to professional
identified as EPA#, faculty participants are identified as F#, resident standards.

participants as R#, and patient/families as P#. Abbreviation: EPAs, entrustable professional activities.
The majority of stakeholders’ feedback centered around clarify-
ing clinical and nonclinical aspects of EM residency training. All of
the EPAs initially presented to stakeholders passed the test of rel- EPA8, EPA11, EPA15, EPA 18, EPA 19, and EPA 21). Yet, it was not
evance (i.e., no one suggested removing any of the EPAs). Although uncommon for different stakeholder groups to vary in their reactions
patients and families supported the proposed EPAs, most expressed and responses to a particular EPA. In addition to commenting on the
not feeling confident in their ability to comment on the more clinical specifics of a particular EPA, many participants suggested language
EPAs (e.g., procedures or diagnostic tests) that a resident should be revisions focused on developmental or contextual aspects, to align
expected to demonstrate before they graduate: their expectations for how these would be used when implemented.
We further elaborate on these themes throughout the results section.
If it is really clinical related alone, the patient doesn't
have a lot of input. But say if it is related to how you
interact with patients or have you tried to calm pa- Meaningful nuances in EPA language
tients down where they are brought into the ED given
it is such an exciting situation, those I think would be Despite stakeholders’ general support for the proposed EPAs, their
important and helpful to give perspective. But if it is feedback suggested that the language needed to be tightened, clari-
really strictly this clinical aspect then we can't com- fied, or in some instances changed completely. Common words that
ment on whether you know how to do intubation. (P3) made stakeholders pause were “recognizing” and “mobilizing” (R2),
“interpreting” (F16), “focused” (F21), “implement” (F10), and “man-
Our analysis highlighted that the EPAs that received the most attention age” (P1, R12). For example, a patient who read the list of EPAs im-
appeared to have a collaborative element to the activity (i.e., EPA1, mediately honed in on the word “manage” in EPA8 and questioned
CARETTA-­WEYER et al. | 5 of 8

whether patients or one's uncertainty should be “the thing” which like honoring their trust, [and] making them want to
residents are expected to manage: “You know from the patient help you if it's your team, or making them believe you
perspective, I don't want to be managed […] I want you to help me in a strange situation or something like that if it's your
understand what is going on” (P1). This participant highlights the patient. (R3)
distinction between having a physician manage them versus hav-
ing a physician partner with them around aspects of clinical care. Throughout the focus groups, residents repeatedly used the word
Residents were also drawn to EPA8 and felt the language most ac- “communication” as one of the most important activities that they
curately reflected their role as residents: need to master during residency training. As this resident describes, it
is communication with everyone on the team:
There is a lot of uncertainty in small decisions too,
but I agree […] that it's appropriate to apply these to I feel like a huge part of what we do is we manage
higher stakes things. […] I think naturally in anyone's the care of this patient through the recommendations
education, you're going to start doing these things that consultants [provide] when [the patient needs]
more independently like having a small shared de- specialized recommendations and interventions. (R7)
cision making about [something like] moderate risk
chest pain with our patient without as much super-
vision. (R1) Contextualizing EPAs and viewing them
developmentally
When residents were made aware of the feedback that patients
did not care for the term “manage,” one resident responded: “We All of the stakeholders, but faculty and residents in particular, com-
[…] manage patients. I know that is not a happy term, but we do” mented on the contextual or developmental aspects of these EPAs.
(R12). In light of what patients and residents had to say about Some examples were evident: “I think recognizing the patient need-
EPA8, faculty viewed this as a difficult EPA that routinely chal- ing emergent intervention is something that people learn pretty
lenges many graduating residents: “[I] still see many or all graduat- early, but then mobilizing resources is probably the more difficult
ing residents struggle with this, most residents mid-­residency are part” (R2, EPA1) or “Order is early; proper interpretation is more of
clueless with this, not independent” (F10). a later skill, but also context dependent” (F11, EPA6). One faculty
Other instances where language was important, related to even started every point of feedback with, “For this EPA…” (F13) and
EPA18 and EPA19: mainly because of the word “communicate” with then followed up with statements such as: “I am considering that the
both other health care professionals and patients and families. In resident is leading the resuscitation from start to finish” (F13, EPA2)
discussing communication, patients and families expressed the im- or “I'm thinking of scenarios that are generally more clear-­cut and
portance of listening: accepted as best evidence and less about more complex, nuanced, or
controversial evidence” (F13, EPA7) to illustrate that context matters
Communication is talking, but it's also listening. And I when we think about assessing residents on these EPAs.
think that is very important with the communication Context is salient in EM because of the heterogeneity across
is that they get that they don't have to talk all the residency training programs. When it came to EPA15, one resident
time. They also need to actively listen to the patient shared: “I think most people learn it intern year” (R3) to which an-
and the caregivers (P4). other resident responded:

Here the patient is conveying that communication requires a resident I disagree a little, at least in our program, interns are
to possess multiple skills to be deemed entrustable. This point was […] being monitored very closely, so you're never re-
echoed by faculty who felt the “complexity of communication required ally getting the opportunity to not give your thera-
mattered” (F19). Residents also agreed and elaborated: peutics timely, and I think it takes until you're a junior,
and sometimes even a late junior, for when you're
For 18 and 19, it says “effectively communicate with starting to get a bit more autonomy, and you have the
other health care professionals and with patients, opportunity to really make a mistake […] And as an
family, and caregivers,” but I think that there's so intern, I think it's really hard to get away with some-
much more to that interaction than communica- thing that's not timely or not appropriate because of
tion. And I think that leaving the language like that the amount of supervision that you have. (R4)
might help people, “oh, I just need to speak clearly in
a way that people understand.” But communication The difference of opinion between these two residents suggests that
with the healthcare team is a lot more than that […] “this is very broad and runs a wide spectrum of complexity” (F20).
Because there's a huge difference between saying Although an intern (first-­year resident) might be able to make a pharma-
what you need, and making sure people understand; cologic and therapeutic management plan, it was likely to be revised.
6 of 8 | A MULTISTAKEHOLDER APPROACH TO DEVELOPING SPECIALTY-­WIDE EPAs IN EM

Another area that highlighted the contextual aspect of EPAs was In the initial step of crafting an outcomes framework, in our case
the opportunity for residents to demonstrate their procedural com- using EPAs, patients and residents have often been overlooked in the
petence, which did not always have to be on a patient in the clinical development process. Patients have additionally been relegated to
workplace. As this resident describes, some EM procedures (EPA11) beneficiaries (direct or indirect) of an EPA or, more often, hidden fig-
are rare; therefore, we often rely on the simulation context to help ures when CBME is meant to focus on relevant health outcomes of
our residents gain knowledge and exposure: patients and society. Sebok-­Syer et al.18 examined the phenomenon
of foregrounding patients specifically within EPAs. In their work, they
There's numbers of procedures, and then there's just proposed an approach to shift the focus away from the supervisor and
the general approach to emergency procedures. I more toward patients in this case, but also with consideration for learn-
think by going with the latter more, thinking of the ers. This requires a focus on patients and learners within the develop-
reason we have EPAs, to encourage programs to teach ment of EPAs, the content of EPAs, and the implementation of EPAs.
their residents on these essential things to emer- As such, we chose within our development process to draft, come
gency medicine. So as an intern, even if you haven't to consensus, and refine EPAs for EM in collaboration with relevant
done a bunch of central lines, you should absolutely patient and resident stakeholders. This was done in addition to the
know how to do a central line because your program traditionally employed representative group of faculty with the in-
taught you in a simulated setting, like knowing how to tention to foreground patients and residents in the process. In our
set it up. And just because you didn't get exposed to approach, we utilized faculty supervisors in the initial drafting and
a bunch of them and you did three by the end of your Delphi for selecting the EPAs and then convened focus groups of
intern year […] that's fine, but you do know how to do residents and patients to react to the EPAs. Faculty remain an ideal
it by the end of your intern year. (R1) stakeholder group to create initial drafts of EPAs given that they are
practicing educators within the specialty and have the viewpoint of
Another aspect that creates heterogeneity in EM is that residency relevant outcomes for competent patient care. However, it is essen-
training can be 3 or 4 years depending on the particular program. This tial to ensure rigorous resident and patient input as these are the
division of programs was apparent when residents spoke about their primary stakeholders as it relates to the outcomes framework—res-
second, third, and fourth years of residency training. When speaking idents are the ones striving to develop the skillset to meet the out-
about EPA21, one resident shared: comes that patients require as part of the delivery of high-­quality
care. This approach led to significant changes to the content of the
I think this is a senior, like later senior, one when you EPAs such that they are representative of all three key stakeholder
have more departmental awareness, like being aware groups.
of other residents’, patients, even the intern's patients, Patients focused primarily on the content of the EPAs, particu-
continuing to move things like to have more of a sense larly nuances centered around language regarding communication,
of the entire department. For all of you guys at four-­ trust, and partnership between patient and provider. This further
year programs, this is like fourth year, ultimately. (R2) reflected the need to foreground the patient within each EPA, which
participants in this focus group picked up on within several of the
This resident recognized that 4-­year programs are designed with a EPAs. For example, the patient focus group highlighted the idea
different developmental trajectory than 3-­year programs and that the that a sense of partnership occurs through effective communication
fourth year for many is really an opportunity to manage all aspects and results in a shared understanding and a sense of collaboration,
of an ED. As this faculty member stated, “do they learn it as seniors, which results in trust. The patient focus group expressed that they
absolutely. But, they would be a level 3 [on EPA21] at best” (F6). As have more confidence in health care providers who can tell them
these examples suggest, context varies and this cannot be ignored why they have made a specific diagnosis, even if that diagnosis may
when creating the shared mental model surrounding the developmen- not be what the patient wants to hear. This foregrounds the patient
tal trajectory of EPAs. within the content of the encounter and subsequently should be re-
flected in the EPA.
Perhaps even more clear was the directionality or what was fore-
DISCUSSION grounded in the language of the eighth EPA. This was originally phrased
as “manage patients in the face of clinical or diagnostic uncertainty.”
CBME is, at its core, about relevant outcomes: for patients and so- Residents felt that they often were tasked with the management of pa-
ciety to receive optimal care and for learners to work toward rel- tients but acknowledged that the language did not feel good. Patients
evant outcomes of their educational training. We must begin with picked up on the directionality of the management asking if they as
the core components framework when we consider the adoption of the patient with the uncertain diagnosis were being managed or if
a competency-­based approach to training within a specialty. An out- the uncertainty being managed was on the part of the resident. They
comes framework is subsequently an essential first step in adopting noted that they wanted the resident to help them understand what
the other four core components of CBME. was going on and to create a partnership in managing the uncertainty
CARETTA-­WEYER et al. | 7 of 8

of their diagnosis. To reflect that, the EPA was subsequently rephrased opportunities into account. This will require an integration of assess-
to “manage clinical or diagnostic uncertainty when caring for patients.” ment and program evaluation to ensure adequate opportunity for prac-
Residents focused primarily on the implementation of the EPAs, par- ticing each EPA as well as individualized learning pathways to address
ticularly as it relates to the variable developmental trajectory of com- areas for continued growth and improvement for each resident. While
petence and the context in which that development occurs. Residents this is apparent now, we did not take this into consideration when draft-
from several different programs highlighted that expectations vary ing or voting on EPAs for inclusion. The EPAs may require additional
based on a host of factors including the practice environment, patient iteration once implemented within a variety of programs and different
mix, curricular opportunities, and faculty comfort with departmental contexts. Finally, we did not tackle making broad recommendations for
volume and acuity. This suggests that there is more than one accept- implementation of the EPAs as we intend to pilot these across a variety
able developmental trajectory within each EPA that is interdependent of training contexts to make more informed recommendations for their
with the context in which each resident trains. The implications of this implementation across the entire specialty of EM in the United States.
are that there cannot be universal “cut points” or clear “categorization”
of expectations for a given phase of training for progress as a develop-
mental “trajectory” inherently implies variability. This would suggest CO N C LU S I O N S
that hard and fast expectations around EPAs to be completed at spe-
cific time points prior to the transition to unsupervised practice would To foreground all key stakeholders within the entrustable profes-
likely be problematic if applied specialty-­wide as they could not be and sional activity process for emergency medicine, the authors chose
should not be forced to be universal across a variable specialty with a within the development process to draft; come to consensus; and
high degree of context specificity. refine entrustable professional activities for emergency medicine in
Additionally, the promise of CBME is the evolution of medical edu- collaboration with relevant faculty, patient, and resident stakehold-
cation toward a continuum of training and practice. As such, rigorously ers. Each stakeholder group contributed meaningfully to the devel-
defining thresholds for competence to be achieved prior to progression opment, content, and intended implementation of the entrustable
present a multitude of concerns within the core components frame- professional activities. This process may serve as a model for others
work. The developmental trajectory of each EPA will not be linear for attempting to develop stakeholder-­responsive entrustable profes-
each resident, which requires educators to leave room for interpretation sional activities within other specialties or contexts.
by competency committees within a given program for acceptable vari-
ation. Furthermore, progress is highly interdependent on the context in AC K N OW L E D G M E N T S
which a resident trains. Residents at one program may perform intuba- The authors thank the faculty, residents, and patients and their families
tions on trauma patients as interns while another may only allow this for who participated in the advisory board and focus groups for this study.
their senior residents. One program may have a rigorous systems-­based
structure leveraging case management support for complex disposition F U N D I N G I N FO R M AT I O N
planning in the ED while another may have no resources and require res- This research was funded by the American Medical Association
idents to do this work. This would lead to residents at the first program Reimagining Residency Grant.
likely being viewed as more efficient while residents at the latter pro-
gram would be viewed as more adept with complex systems issues and C O N F L I C T O F I N T E R E S T S TAT E M E N T
considering social determinants of health in disposition planning. Finally, The authors declare no conflicts of interest.
a resident may be allowed to perform an EPA with lower acuity patients
on a given shift but on higher acuity patients on a different shift, entirely ORCID
dependent on faculty comfort with the volume, acuity, and support staff Holly A. Caretta-­Weyer [Link]
associated with the context of that specific shift. These variations in su- Stefanie S. Sebok-­Syer [Link]
pervision are natural and must be accounted for within the acceptable Amanda M. Morris [Link]
variation of a developmental trajectory and when crafting individualized Benjamin H. Schnapp [Link]
learning pathways for residents using these EPAs. Abra L. Fant [Link]
Kevin R. Scott [Link]
Michael A. Gisondi [Link]
LI M ITATI O N S Lalena M. Yarris [Link]

First, by foregrounding the patient and resident within the EPA, we REFERENCES
involved a broad selection of stakeholders in the development pro- 1. Ten Cate O. Nuts and bolts of entrustable professional activities. J
cess. However, we did not involve all stakeholders in the drafting of Grad Med Educ. 2013;5(1):157-­158. doi:10.4300/JGME-­D-­12-­0 0380.1
2. Van Melle E, Frank JR, Holmboe ES, et al. A Core components
the EPAs which may have altered the number and composition of the
framework for evaluating implementation of competency-­based
original EPAs. Second, the developmental trajectory of each resident medical education programs. Acad Med. 2019;94(7):1002-­1009.
should be transparent and take context and programmatic support or doi:10.1097/ACM.0000000000002743
8 of 8 | A MULTISTAKEHOLDER APPROACH TO DEVELOPING SPECIALTY-­WIDE EPAs IN EM

3. Royal College of Physicians and Surgeons of Canada. Anesthesiology professional activities in complex care. Acad Med. 2023;98(3):342-­
Entrustable Professional Activity Guide. Accessed December 13, 347. doi:10.1097/ACM.0000000000005095
2022. [Link] ww.​royal​colle​ge.​c a/​rcsite/​docum​ents/​cbd/​epa-­​ 18. Sebok-­Syer SS, Gingerich A, Holmboe ES, Lingard L, Turner DA,
guide​-­​anest​hesio​logy-­​v3-­​e.​pdf Schumacher DJ. Distant and hidden figures: foregrounding patients
4. Entrustable Professional Activities for Psychiatry. Royal College of in the development, content, and implementation of Entrustable
Physicians and Surgeons of Canada. Accessed December 13, 2022. professional activities. Acad Med. 2021;96(7S):S76-­S80.
[Link] a/​sites/​​defau​lt/​f iles/​​inlin​e-­​files/​​epa-­​ doi:10.1097/ACM.0000000000004094
guide​-­​psych​iatry​-­​e.​pdf 19. Emergency Medicine Specialty Committee. EPA Guide: Emergency
5. Entrustable Professional Activities. The American Board of Medicine. Royal College of Physicians and Surgeons of Canada;
Pediatrics. Accessed December 13, 2022. [Link] ww.​abp.​org/​ 2017.
sites/​​p ublic/​f iles/​​p df/​a ll-­​gen-­​peds-­​epas-­​and-­​curri​c ular​- ­​compo​ 20. The Core Entrustable Professional Activities (EPAs) for Entering
nents.​pdf Residency. Association of American Medical Colleges. Accessed
6. Entrustable Professional Activities–General Surgery. The American December 13, 2022. [Link] ww.​aamc.​org/​what-­​we-­​do/​missi​on-­​
Board of Surgery. Accessed December 13, 2022. [Link] ww.​ areas/​​medic​al-­​educa​tion/​cbme/​core-­​epas
absur​gery.​org/​defau​lt.​jsp?​epa_​gs 21. Obeso V, Brown D, Aiyer M, et al. Core EPAs for Entering Residency
7. Hart D, Franzen D, Beeson M, et al. Integration of Entrustable Pilot Program. Toolkits for the 13 Core Entrustable Professional
professional activities with the milestones for emergency medi- Activities for Entering Residency. 2017. Association of American
cine residents. West J Emerg Med. 2019;20(1):35-­42. doi:10.5811/ Medical Colleges. Accessed December 21, 2022. aamc.​org/​initi​
westjem.2018.11.38912 atives/​coree​pas/​publi​c atio​nsand​prese​ntations
8. Yarris LM. Defining trainee competence: value is in the eye of 22. Emergency Medicine Milestones. Accreditation Council for
the stakeholder. Acad Med. 2019;94(6):760-­762. doi:10.1097/ Graduate Medical Education. 2021. Accessed December 21, 2022.
ACM.0000000000002643 [Link] ww.​a cgme.​o rg/​g loba​l asse​t s/​p dfs/​m iles​t ones/​​e merg​
9. Lundsgaard KS, Tolsgaard MG, Mortensen OS, Mylopoulos M, encym​edici​nemil​eston​es.​pdf
Østergaard D. Embracing multiple stakeholder perspectives in 23. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using
defining trainee competence. Acad Med. 2019;94(6):838-­8 46. and reporting the Delphi method for selecting healthcare qual-
doi:10.1097/ACM.0000000000002642 ity indicators: a systematic review. PLoS One. 2011;6(6):e20476.
10. Tanaka P, Marty A, Park YS, et al. Defining entrustable professional doi:10.1371/[Link].0020476
activities for first year anesthesiology residents: a Delphi study. J 24. Humphrey-­Murto S, Wood TJ, Gonsalves C, Mascioli K, Varpio
Clin Anesth. 2023;88:111116. doi:10.1016/[Link].2023.111116 L. The Delphi method. Acad Med. 2020;95(1):168. doi:10.1097/
11. Amare EM, Siebeck M, Sendekie TY, Fischer MR, Berndt M. ACM.0000000000002887
Development of an Entrustable professional activities (EPA) 25. Waggoner J, Carline JD, Durning SJ. Is there a consensus on consen-
framework to inform surgical residency training programs in sus methodology? Descriptions and recommendations for future
Ethiopia: a three-­round National Delphi Method Study. J Surg Educ. consensus research. Acad Med. 2016;91(5):663-­668. doi:10.1097/
2022;79(1):56-­68. doi:10.1016/[Link].2021.06.023 ACM.0000000000001092
12. Acker S, Noelke A, Huckabee M, Rieck KM. Development of the 26. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res
proposed 13 Entrustable professional activities for physician as- Psychol. 2006;3(2):77-­101. doi:10.1191/1478088706qp063oa
sistant graduates. J Physician Assist Educ. 2021;32(4):232-­236.
doi:10.1097/JPA.0000000000000387
13. Woodworth GE, Marty AP, Tanaka PP, et al. Development and S U P P O R T I N G I N FO R M AT I O N
pilot testing of Entrustable professional activities for US anesthe- Additional supporting information can be found online in the
siology residency training. Anesth Analg. 2021;132(6):1579-­1591. Supporting Information section at the end of this article.
doi:10.1213/ANE.0000000000005434
14. Young JQ, Hasser C, Hung EK, et al. Developing end-­of-­training
Entrustable professional activities for psychiatry: results and meth-
odological lessons. Acad Med. 2018;93(7):1048-­1054. doi:10.1097/
ACM.0000000000002058 How to cite this article: Caretta-­Weyer HA, Sebok-­Syer SS,
15. Pagano MB, Treml A, Stephens LD, et al. Entrustable professional Morris AM, et al. Better together: A multistakeholder
activities for apheresis medicine education. Transfusion (Paris). approach to developing specialty-­wide entrustable
2020;60(10):2432-­2440. doi:10.1111/trf.15983
professional activities in emergency medicine. AEM Educ
16. El-­Haddad C, Damodaran A, McNeil HP, Hu W. A patient-­centered
approach to developing entrustable professional activities. Acad Train. 2024;8:e10974. doi:10.1002/aet2.10974
Med. 2017;92(6):800-­8 08. doi:10.1097/ACM.0000000000001616
17. Huth K, Henry D, Cribb Fabersunne C, et al. Family–ed-
ucator partnership in the development of entrustable