1059644

research-article2021
CPJXXX10.1177/00099228211059644Clinical PediatricsHarrison et al

Article
Clinical Pediatrics

Addressing Pediatric Developmental 2022, Vol. 61(1) 46­–55

© The Author(s) 2021
Article reuse guidelines:
and Mental Health in Primary Care sagepub.com/journals-permissions
DOI: 10.1177/00099228211059644
https://doi.org/10.1177/00099228211059644

Using Tele-Education journals.sagepub.com/home/cpj

Joyce N. Harrison, MD1,2, Janna Steinberg, MA1,

Anna Maria Louise Wilms Floet, MD1,2, Nancy Grace, PhD1,2,
Deepa Menon, MB BS1,2, Rebecca German, MSW1, Belinda Chen, MA1 ,
Gayane Yenokyan, MD, MPH, PhD3,
and Mary L. O’Connor Leppert, MB BCH1,2

Abstract
This study evaluates the effectiveness of an early childhood tele-education program in preparing community pediatric
clinicians to manage developmental and mental health disorders in young children. Community pediatric clinicians
from rural, underserved, or school-based health center practices in the mid-Atlantic region participated in a weekly
tele-education videoconference. There was a significant knowledge gain evidenced by the percentage of questions
answered correctly from pre- to post- didactic exposure (P < .001). Participants reported an increase in knowledge
from pre- (P < .001) and in confidence from pre- to post- participation (P < .001). Practice management changes
demonstrated an encouraging trend toward managing patients in the Medical Home, as compared with immediately
deferring to specialists following participation. This early childhood tele-education videoconferencing program is a
promising response to the urgent need to confidently increase the role of pediatricians in the provision of care for
childhood developmental and mental health disorders.

Keywords
medical education, developmental and behavioral pediatrics, project ECHO

Introduction of Pediatrics offered subspecialty certification for those

with specific training in Neurodevelopmental
Pediatric clinicians are struggling to meet the demands Disabilities (NDD) in 2001, and Developmental-
of children with developmental and behavioral disor- Behavioral Pediatrics (DBP) in 2002. The introduction
ders. Presently, 17% of children in the United States of subspecialists trained in chronic disorders of develop-
have a disability,1 and 10% to 20% have a disorder of ment and behavior became a valued referral resource for
behavior or mental health.2 The top 5 functionally limit- primary care physicians, but the number of certified spe-
ing chronic conditions confronting pediatricians are cialists remains insufficient to effectively care for the
developmental and behavioral in nature.3 In 2000, the population of children with developmental or behavioral
Future of Pediatric Education II (FOPE II) reported that disorders.6
the rise in developmental etiologies of chronic condi-
tions would have significant implications on pediatric
practice and education.4 The final report of the FOPE II
1
recommended that pediatricians must be prepared to Kennedy Krieger Institute, Baltimore, MD, USA
2
Johns Hopkins University School of Medicine, Baltimore, MD, USA
care for the “new morbidities” of pediatrics including 3
Johns Hopkins Bloomberg School of Public Health, Baltimore, MD,
chronic medical, mental health, behavioral, and devel- USA
opmental disorders.4 Since 1997, pediatric residency
Corresponding Author:
training mandates a 1 month rotation in behavioral/ Mary L. O’Connor Leppert, Kennedy Krieger Institute, 707 North
developmental pediatrics.5 Shortly after implementing Broadway, Baltimore, MD 21205, USA.
the recommendations of FOPE II, the American Board Email: [email protected]
Harrison et al 47

Barriers to the care of these vulnerable children access programs, and telehealth and telemedicine efforts
remain despite the efforts to address the rising tide of to provide consultation and direct care. Despite the
developmental and behavioral disorders. Presently, the widespread growth of interventions to improve the inte-
American Board of Medical Subspecialties reports that gration and delivery of behavioral and mental health in
there are a total of 907 NDD or DBP pediatricians with primary care, the need to fund and expand as well as
active certificates in the United States,7 insufficient to systematically evaluate promising programs remains.17
meet the needs of the relevant pediatric population. The early childhood tele-education videoconferenc-
Furthermore, these subspecialists tend to be located in ing program described in this study is the first develop-
urban academic centers where waitlists are long, and mental and behavioral health Extension for Community
distance and insurance preclude accessibility to whole Healthcare Outcomes (ECHO) program. It combines
populations of children in poverty or in underserved or longitudinal CME and integrated care through case-
rural areas. Consequently, community pediatric clini- based consultation. This study aimed to evaluate the
cians (CPCs) are called upon for the evaluation and impact of this program on CPC’s self-reported confi-
management of children with complex developmental dence and knowledge in addressing developmental and
and behavioral health conditions. A 2007 American behavioral health concerns among children ages birth 6
Academy of Pediatrics survey of pediatricians indicated years. Other measures included self-reported learning
that the majority believe that primary care pediatricians priorities, barriers to caring for children with develop-
should be responsible for the identification of selected mental and behavioral health diagnoses, and practice
developmental and behavioral disorders and for refer- management.
ring them to subspecialists.8 With the exception of atten-
tion-deficit/hyperactivity disorder (ADHD), most
Methods
believed that they were not responsible for the treatment
of anxiety, depression, behavioral problems, or learning Participants
difficulties.8,9 A survey of general pediatricians 4 to 5
years out of residency training indicated that a minority Four cohorts consisted of 61 participants who practice in
of practitioners were comfortable providing care for the community setting, and included 33 physicians (32
children with developmental disorders (31%) and men- pediatric, 1 family medicine), 17 nurse practitioners (9
tal health conditions (7%) independently.10 Pediatricians family, 7 pediatric, 1 psychiatric mental health), 1 regis-
cite gaps in training as a barrier to identifying or manag- tered nurse, 4 physician assistants, 2 clinical psycholo-
ing developmental or mental health disorders in their gists (1 master’s level and 1 doctoral), and 4 clinical
practices.10-12 The constellation of the increasing num- social workers from rural, underserved, or school-based
ber of children with developmental and behavioral dis- health center (SBHC) practices in the mid-Atlantic area.
orders, a workforce shortage of subspecialists, and an Of those participants who reported practice information,
under-prepared workforce of pediatric primary care cli- 22 were private practices, 10 were federally qualified
nicians has brought this pediatric health crisis to the health centers (FQHCs), and 12 were SBHCs.
fore. Participants included 50 females and 11 males and aver-
Stakeholders from the American Academy of Pediatrics, aged 47.6 years of age and 15.5 years in practice. This
the Association of Pediatric Program Directors, and the study was approved by the appropriate institutional
National Academies of Sciences, Engineering, and review board (IRB #00108505).
Medicine have convened to address the crisis in access to
care for behavioral and mental health problems.13-15 A Early Childhood Extension for Community
recently published special report presents strategies, guide- Healthcare Outcomes (ECHO) Program
lines, and updates to efforts to address these barriers.16
Although much progress has been made, the authors con- Establishing the ECHO program required 3 phases;
clude that there is still an urgent need for CPCs to confi- planning, preparation, and delivery (Figure 1). The plan-
dently increase their role in the provision of care for ning phase included honing the topic of the ECHO,
behavioral and mental health disorders.16 defining the team, and seeking funding. Four years of
Outlined strategies that have met with variable suc- funding was awarded from the Health Services Research
cess include increased residency training in develop- Administration through a competitive grant process.
mental disabilities and mental health disorders, increased The rural health grant was specifically targeted to feder-
opportunities for continuing medical education (CME) ally designated rural areas and SBHCs in Maryland,
in these topics, integration and colocation of mental with extension to West Virginia after the second year. In
health providers in pediatric practices, child psychiatry the preparation phase, 5 hub team members participated
48 Clinical Pediatrics 61(1)

Figure 1. Planning, preparation, and delivery phases of ECHO (Extension for Community Healthcare Outcomes) development.

in a 9-month curriculum development course offered hub team, recommendations from the CPCs and hub
through the Johns Hopkins School of Medicine’s team, and finally a summary of the recommendations,
Institute for Excellence in Education. Curriculum devel- which are sent to participants after the clinic. The cases
opment was based on Kern’s 6-step model of general were often complex and required 30 to 40 minutes for
and targeted needs assessment, goals and objectives, discussion. The case reviews provoked robust conversa-
educational strategies, implementation, and evaluation tion among all participants on diagnoses, medical, edu-
and feedback.18 Team members also completed ECHO cational, psychosocial interventions, and community
training at the University of New Mexico.19 Also in the resources. Hub experts supplemented the discussion
preparation phase, the team recruited partners and built with evidence-based instruction. A 15- to 20-minute
templates and tools for data collection, including case didactic followed the case discussion. CME credits were
forms, surveys, and registration forms. Recruitment was available for all registered participants.
aided and supported by the state American Academy of The teleECHO sessions followed an academic year
Pediatrics chapters but also included many in-person with weekly sessions from September to mid-June. A
visits to practices and schools to describe the project and 4-week miniseries was offered during the summer.
register participants. Ongoing recruitment for succes- The didactic modules were from a semi-structured
sive cohorts was through presentations at meetings and educational curriculum consisting of learning modules
grand rounds, and word of mouth. on neurodevelopmental and behavioral disorders
The final phase, preparing for delivery, included authored by faculty members. The curriculum con-
reviewing the survey of participant learning preferences, tains 15- to 20-minute learning modules in 5 semis-
finalizing the syllabus, and practicing with a mock tructured levels, which range from introductory (level
ECHO before the formal launch of the program. 1) to expert (level 5) content (M. L. O’Connor Leppert,
In fidelity to the ECHO model, the program com- unpublished data, 2021). The majority of modules
bined case-based discussion and didactic teaching in a were from levels 2, 3 and 4 in the curriculum, which
“hub and spoke” model, with multidirectional exchange cover screening and differential diagnosis (level 2),
of information so that “all teach and all learn.” The evaluation and identification of co-occurring condi-
“hub” in this Early Childhood ECHO consisted of neu- tions (level 3), and intervention strategies (level 4).
rodevelopmental and developmental-behavioral pedia- Content materials were largely drawn from learner
tricians, a child and adolescent psychiatrist, and a preferences. They included topics on adverse child-
behavioral psychologist. The community participants hood experiences (ACEs), ADHD, anxiety disorders,
described above served as the “spokes.” autism spectrum disorder (ASD), developmental
Weekly, 1-hour teleECHO sessions were conducted delay, disruptive behavior, educational law, genetics,
with the standard ECHO structure. A de-identified case, intellectual disability, speech and language disorders,
which is usually submitted in advance on a form devel- sleep, substance exposure, transition, and the behav-
oped by the hub team, is presented by a CPC. This is ioral and educational challenges brought on by the
followed by questions from the participants, then the COVID-19 pandemic.
Harrison et al 49

Table 1. Number of Available Survey Responses by Measurement and Timepoint.

Knowledge Barriers to care Confidence Practice management

Pre 0 15 15 12
Retro Pre 14 7 12 9
Pre total 14 22 27 21
Post 14 15 24 15

Participant Evaluation Post-participation surveys also assessed retrospective

pretest (retro-pre) ratings. These ratings are commonly
Participant data included demographics (certification used in evaluation of professional development pro-
type, practice location, and estimate of children in their grams to reduce bias and assess change.20 Pre-participation
practices), attendance, and the number of patient travel surveys queried learning priorities to inform the curricu-
miles saved. One-way patient travel miles saved was the lum. Specific topics were categorized as developmental
distance from the practice location, as a proxy for patient or mental/behavioral health by team consensus. The
location, to the nearest center in which developmental developmental topics listed in the survey were ADHD,
pediatrics or child psychiatrists were on staff and accept- ASD, developmental delay, motor delay, speech and lan-
ing new patients. Likert-type scales measured partici- guage disorders, and in utero substance exposure. The
pants’ self-report of learning priority topics from not at mental/behavioral health topics listed in the survey were
all helpful (1) to extremely helpful (5). ACEs, anxiety, disruptive behavior, feeding issues,
obsessive compulsive disorder, parent-child relational
Session Evaluation issues, psychotropic medications, sexual acting out,
sleep disruption, and toileting issues. An open-text
Content knowledge questions, collected using Poll
option was available for participants to suggest addi-
Everywhere, an audience response system, measured
tional learning priorities and provide informal feedback,
objective knowledge gain. Questions were composed by
which was not qualitatively analyzed.
authors of the modules to reflect the top 3 take-away
Likert-type scales measured participants’ self-report
points that the authors wanted to convey. All questions
were peer-reviewed to ensure alignment with specifica- of barriers to care from not at all (1) to greatly (5),
tions for American Board of Pediatrics certification and knowledge from none (1) to a great deal (5), and confi-
were formatted for consistency. The multiple-choice dence from not at all (1) to very (5) confident. Practice
questions were asked during the didactic presentation management was measured by self-report of participants
and repeated 1 week after content delivery. Responses to as to whether they typically deferred all management of
questions posed during the didactic (pre-), and a week patients with developmental and behavioral health dis-
later in the post-test were collected anonymously and orders to a specialist, comanaged with a specialist, or
aggregated to enhance participation. Participants not managed on their own.
present for the didactic the week before were effectively To compensate for a small sample size, retro-pre data
contributing pre-test responses to the aggregated post- provided supplemental data for barriers to care, confi-
test data. Results of the aggregate participant percentage dence, and practice management (see Table 1). Retro-
of correct responses were reported. Data on how well pre data and pre-data were compared at face value and
learning objectives were met were collected. were similar enough to justify using retro-pre data to fill
in missing data. Only retro-pre data existed for knowl-
edge as this measure was added later in the project.
Overall Program Evaluation Data were analyzed using IBM SPSS Statistics21 and
22
Surveys prior to and following participation in the R. Likert-type scale data are presented as the 25th per-
ECHO program were collected via Qualtrics and were centile (Q1), median (Mdn), and 75th percentile (Q3).
de-identified before analysis. Surveys were sent to 53 of Objective knowledge scores were calculated as the per-
the 61 total participants, as 8 participants enrolled in centage of correct responses and were summarized with
4-week summer sessions were excluded because of the mean (M) and standard deviation (SD). Comparison of
brevity of their participation. Pre- and post-participation participants’ pre- to post- ratings of barriers to care were
surveys assessed participants’ barriers to caring for and analyzed with Wilcoxon paired signed-rank tests because
knowledge, confidence, and practice management of of the small sample size of the survey responses.
children with developmental and behavioral disorders. Objective knowledge change was analyzed with a paired
50 Clinical Pediatrics 61(1)

t test. Pre- to post- self-report of knowledge in each of the Program Evaluation

domains of developmental and behavioral health were
calculated by averaging knowledge ratings and also ana- Surveys were completed by 31 of the 53 eligible partici-
lyzed with Wilcoxon paired signed-ranks tests because pants (58.49% response rate) despite repeated email and
of the small sample size. Pre- to post- self-report of con- verbal reminders. The response rate was representative
fidence was similarly calculated and analyzed. Change in of the participants by certification type. Due to response
practice management is reported as observed trends. drop-off, sample sizes were smaller for measures at the
end of the survey. Also, the knowledge gain measure
was added to the survey later in the program so it was
Results only sent to 24 of the 53 eligible participants.
Barriers to care included CPC, systems, and patient
Participant Evaluation factors. Of the 10 barriers assessed, the only statistically
The pilot consisted of 14 one-hour weekly sessions. All significant change was CPC confidence/comfort in car-
subsequent cohorts were offered an academic year of ing for children with developmental and/or mental
weekly sessions, averaging 38 sessions per cohort. Over health concerns. The participants reported that their con-
4 cohorts, 134 sessions were delivered, employing 67 fidence and comfort in providing care was less of a bar-
unique learning modules. Average attendance was 8.26 rier from pre- (Q1 = 1.00, Mdn = 1.00, Q3 = 2.00) to
participants per session, with a range of 2 to 18. Summer post-participation (Q1 = 0.00, Mdn = 1.00, Q3 = 1.00),
miniseries participants were not enrolled or surveyed Z = −2.49, P = .008).
under the same protocols as longitudinal program par- Median knowledge ratings were significantly higher
ticipants, and were therefore excluded from analysis. A on participant post-test ratings as compared with retro
total of 53 unique CPCs from the longitudinal ECHO pre-test ratings in both developmental and mental health
series attended an average of 41% of sessions (range 6% domains. Overall, participants reported an increase in
to 96%) of semesters in which they enrolled. Frequency knowledge from pre- (Mdn = 2.00) to post- (Mdn =
of attendance was highest in the physician group (50%), 2.50), Z = 3.07, P < .001. Self-reported knowledge
followed by nurse practitioners (30%). Pre-participation could not be correlated with objective knowledge mea-
surveys (n = 21) of participants’ learning priorities sures because the objective pre-post responses were
revealed that the top 5 rated topics were obsessive com- aggregate and anonymous. Participants reported a simi-
pulsive behavior, sexual acting out, aggression, ASD, lar increase in knowledge in the developmental domain
and parent-child relational issues. from pre- (Mdn = 2.00) to post- (Mdn = 2.50), Z =
2.93, P = .001, as well in the mental health domain pre-
(Mdn = 2.00) to post- (Mdn = 2.50), Z = 3.19, P <
Session Evaluation .001. The post-test score distributions were statistically
In the 134 teleECHO sessions, 94 cases (85 unique) significantly higher than the median retro pre-test scores
were presented by participants. Ninety-eight percent (n for the following developmental and behavioral disor-
= 83) of unique cases were managed in the Medical ders: ASD, speech language delay, disruptive behavior,
Home and did not require referral to subspecialty care. anxiety, sleep disruption, psychotropic medications, and
Patients were referred to specialty clinics with specific ACEs (see Table 2).
targeted needs. Medical Home management saved each Median confidence ratings were significantly higher
family an average of 119.85 miles in one-way travel to on participants’ on post-test in both developmental and
the nearest specialist. mental health domains. Overall, participants reported
Evaluation of each didactic module included ratings an increase in confidence from pre- (Mdn = 1.82) to
of how well learning objectives were met and objective post-participation (Mdn = 2.52), Z = 3.72, P < .001.
measure of knowledge gain. On average, 96% of partici- Participants reported a similar increase in confidence in
pants agreed or strongly agreed that the didactics met the developmental domain from pre- (Mdn = 2.00) to
the learning objectives. Participant knowledge gain post- (Mdn = 2.50), Z = 3.55, P < .001, as well as in
across the didactics was available for 181 questions the mental health domain from pre- (Mdn = 1.86) to
(range of percent correct was 0% to 100%). An average post- (Mdn = 2.50), Z = 3.73, P < .001. The post-test
of 6.21 participants answered pre- and an average of distributions were statistically significantly higher than
5.89 participants answered post- per ECHO sessions. the median pre-test scores for the following develop-
There was a significant gain in the percentage of ques- mental and behavioral disorders: ASD, developmental
tions answered correctly from pre- (M = 49.84, SD = delay, hyperactivity, ADHD, tantrums, disruptive
31.90) to post- (M = 75.88, SD = 25.60), t(180) = behavior, anxiety, psychotropic medications, and ACEs
10.50, P < .001. (see Table 2).
 Table 2. Self-Reported Knowledge Change (n = 14) and Confidence Change (n = 24) by Topic.

Knowledge Confidence

Pre Post Post Pre Post

Pre Q1 Post Q1 median median Pre Q3 Q3 Z P Pre Q1 Post Q1 median median Pre Q3 Post Q3 Z P
ASD 2.00 2.00 2.00 2.00 2.50 3.00 2.45 .016 1.00 2.00 1.50 2.00 2.00 3.00 3.27 <.001
Speech language delay 1.50 2.00 2.00 2.00 2.00 3.00 3.16 .001 2.00 2.00 2.00 2.00 3.00 3.00 1.67 .090
Developmental delay 2.00 2.00 2.00 2.00 2.50 3.00 1.89 .063 2.00 2.00 2.00 3.00 3.00 3.00 3.00 .002
Hyperactivity 2.00 2.00 2.00 2.00 3.00 3.00 2.12 .035 1.00 2.00 2.00 3.00 3.00 3.00 3.64 <.001
ADHD 2.00 2.50 2.00 2.75 3.00 3.00 2.33 .020 1.00 2.00 2.00 3.00 3.00 3.00 3.50 <.001
Tantrums 2.00 2.00 2.00 2.00 2.50 3.00 1.34 .188 2.00 2.00 2.00 3.00 3.00 3.00 2.50 .010
Disruptive behavior 1.50 2.00 2.00 2.00 2.50 3.00 2.83 .004 1.00 2.00 2.00 2.00 2.00 3.00 2.81 .002
Anxiety 1.50 2.50 2.00 2.75 2.00 3.00 3.46 <.001 1.00 2.00 2.00 2.25 2.00 3.00 3.02 <.001
Sleep disruption 1.00 2.00 2.00 2.00 2.00 3.00 3.00 .002 1.00 2.00 2.00 2.50 3.00 3.00 2.00 .036
Psychotropic medications 1.00 2.00 1.00 2.00 2.00 3.00 3.22 <.001 1.00 2.00 1.00 2.00 2.00 2.00 3.60 <.001
ACEs 2.00 2.00 2.00 2.00 2.00 3.00 2.53 .008 1.00 2.00 2.00 3.00 2.00 3.00 3.34 <.001

Abbreviations: ASD, autism spectrum disorder; ADHD, attention-deficit/hyperactivity disorder; ACEs, adverse childhood experiences.

51
52 Clinical Pediatrics 61(1)

Table 3. Self-Reported Management Percent Point Change by Topic (n = 15).

Defer all management to Comanage patient with a Manage patient on my

specialist, % specialist, % own, %
ASD −24 +16 +8
Speech language delay −18 −4 +22
Developmental delay −10 −7 +17
Hyperactivity −12 −10 +22
ADHD −14 −16 +31
Tantrums −10 0 +10
Disruptive behavior −14 +24 −10
Anxiety −14 −31 +45
Sleep disruption −10 −16 +26
Psychotropic medications −15 +29 −15
ACEs −19 +2 +17

Abbreviations: ASD, autism spectrum disorder; ADHD, attention-deficit/hyperactivity disorder; ACEs, adverse childhood experiences.

Changes in practice management demonstrated an its effectiveness has led to the Expanding Capacity for
encouraging trend toward CPC management of patients Health Outcomes Act of 2019.26
in the medical home, as compared with immediately An additional strength of this professional educa-
deferring to specialists (see Table 3). Although partici- tional paradigm is the opportunity for continuous evalu-
pants varied in professional backgrounds, 93% of the ation and collection of feedback, which led to curriculum
respondents to the question about management were of adjustments in response to the needs and interests of the
professional backgrounds that include the capacity to learners, as well as the content of the cases presented.
prescribe psychotropic medications. Psychologists and For example, after recognizing the frequency of cases
social workers presumably would continue to defer with a history of substance exposure, several modules
management of prescribing to specialists. on the consequences of in utero substance exposure
were added. COVID-19-related topics on the impact of
school closures and behavioral changes in children dur-
Discussion ing the pandemic were added at the request of partici-
The ultimate goal of ECHO programs is to improve spe- pants. As the hub team identified additional needs,
cialty care delivery by enhancing primary care provid- modules on managing stress, sleep, and providing struc-
ers’ knowledge and confidence. Measuring success via ture during COVID-19 were also included.
patient outcomes and practice change is subjective and This early childhood tele-education videoconferenc-
challenging. This early childhood tele-education video- ing program and similar collaborative learning programs
conferencing program has the potential to grow local provide benefits beyond the quantitatively measured
experts, reduce barriers to care, and improve access to changes reported in this study.24,25 Informal feedback
knowledgeable, confident practitioners who can care for from participants included decreased feelings of isola-
children with developmental and behavioral disorders in tion and identification as a behavioral/mental health
the medical home. The most promising indication of the expert within the practice and/or regionally. The collab-
effectiveness of the program is the trend toward orative environment establishes connections within the
increased management in the medical home. The contri- community and between systems, improving awareness
bution of the didactics is demonstrated by the statically of local resources, and enhancing the understanding of
significant improvement in knowledge gain measured the roles or services provided by various professionals.
objectively from pre- to post-test quizzes. In addition to The combination of learning and collaboration improves
improved objective knowledge, participants self- the access to and quality of care for children with devel-
reported more confidence in their knowledge and abili- opmental and behavioral disorders within communities.
ties after participating. This combination of case-based The greatest limitation to this study was the small
learning and didactic instruction follows the ECHO sample size. Furthermore, the uneven distribution of par-
model of professional development and appears to be ticipants by certification type (33 physicians, 17 nurse
quite effective. The success of this model23-25 is evi- practitioners, 1 nurse, 4 physician assistants, 4 social
denced by its global expansion and positive evidence of workers, and 2 psychologists) prohibited the analysis of
Harrison et al 53

ratings of knowledge gain, confidence, and practice man- community of professionals with the knowledge and
agement changes among participants by professional confidence to care for children with neurodevelopmen-
certification. Small sample size further prohibited the tal and behavioral disorders within the medical home.
comparison of survey results by participant location or
practice size. Consistency of attendance was variable, Acknowledgments
with an average attendance of 41% of sessions, perhaps We are so very appreciative of the pediatric primary care clini-
due to busy practice schedules, which may have limited cians who contributed so much to the success of this program.
participation in this mid-day ECHO. Additional limita- The complexity of the children for whom they care, and their
tions include lack of a control group and potential for dedication to and advocacy for their patients was inspiring.
selection bias as the questions may have been answered This publication was made possible by the Johns Hopkins
by participants who felt more confident in their answers. Institute for Clinical and Translational Research (ICTR),
The objective knowledge post-test measure was lim- which is funded in part by Grant Number UL1 TR003098 from
ited by our method of data collection. The post-test aver- the National Center for Advancing Translational Sciences
(NCATS), a component of the National Institutes of Health
age is likely an underestimate of the participants’
(NIH), and NIH Roadmap for Medical Research. Its contents
knowledge gain, as participants absent for the pre-test are solely the responsibility of the authors and do not necessar-
and didactic that followed it, but present for the post-test, ily represent the official view of the Johns Hopkins ICTR,
were effectively adding pre-test responses to our post- NCATS, or NIH.
test aggregate score. In addition, this study relied on
self-report measures and did not include direct measures Author Contributions
of practice change or patient outcomes.
MLOL: Contributed to conception and design; Contributed to
ECHO and ECHO-like models (EELM) typically
interpretation; drafted manuscript; critically revised manu-
report on participants’ own assessment of the benefits of script; gave final approval; agrees to be accountable for all
such programs. Evaluation of the impact on child health aspects of work ensuring integrity and accuracy.
and educational outcomes, family outcomes, and the JNH: Contributed to conception and design; contributed to
collaborations between medical and educational sys- interpretation; drafted manuscript; critically revised manu-
tems would add great value to future assessments of this script; gave final approval; agrees to be accountable for all
clinical education model. The model would also benefit aspects of work ensuring integrity and accuracy.
from the measurement of long-term knowledge reten- NG: Contributed to conception and design; critically revised
tion, as this evaluation assessed retention only 1-week manuscript; gave final approval; agrees to be accountable for
post content exposure. Furthermore, the curriculum all aspects of work ensuring integrity and accuracy.
employed is designed for multi-level learners of diverse DM: Contributed to conception and design; critically revised
manuscript; gave final approval; agrees to be accountable for
clinical or educational backgrounds. As the care of chil-
all aspects of work ensuring integrity and accuracy.
dren with developmental and behavioral disorders is AMWF: Contributed to conception and design; critically
multifaceted and interdisciplinary in nature, the use of a revised manuscript; gave final approval; agrees to be account-
versatile curriculum in the EELM holds great promise able for all aspects of work ensuring integrity and accuracy.
for the robust education of all disciplines privileged with JS: Contributed to acquisition; drafted manuscript; critically
the care of this vulnerable population of children. revised manuscript; gave final approval; agrees to be account-
able for all aspects of work ensuring integrity and accuracy.
RG: Contributed to acquisition; critically revised manuscript;
Conclusion gave final approval; agrees to be accountable for all aspects of
This early childhood tele-education program is a struc- work ensuring integrity and accuracy.
tured, longitudinal CME program that provides guided BC: Contributed to acquisition and analysis; critically revised
manuscript; gave final approval; agrees to be accountable for
practice to pediatric primary care clinicians. It is a prom-
all aspects of work ensuring integrity and accuracy.
ising response to the call to action to address the urgent GY: Contributed to analysis and interpretation; critically
need to confidently increase the role of pediatricians in revised manuscript; gave final approval; agrees to be account-
the provision of care for childhood developmental and able for all aspects of work ensuring integrity and accuracy.
mental health disorders.16 This program and the ECHO
model employ several of the strategies suggested to Declaration of Conflicting Interests
address the growing crisis. The provision of case-based The author(s) declared no potential conflicts of interest with
learning and topic-specific didactic learning from a respect to the research, authorship, and/or publication of this
structured curriculum equips providers and creates a article.
54 Clinical Pediatrics 61(1)

Funding 8. Horwitz SM, Caspary G, Storfer-Isser A, et al. Is devel-

opmental and behavioral pediatrics training related to
The author(s) disclosed receipt of the following financial sup-
perceived responsibility for treating mental health prob-
port for the research, authorship, and/or publication of this
lems? Acad Pediatr. 2010;10:252-259. doi:10.1016/j.
article: This study was funded by a Health Resources and
acap.2010.03.003
Services Administration Telehealth Network Grant Program
9. Stein REK, Horwitz SM, Storfer-Isser A, et al. Do pedia-
Award, H2ARH30299-02-01. Dr Yenokyan was supported by
tricians think they are responsible for identification and
the National Institutes of Health, National Center for
management of child mental health problems? Results of
Advancing Translational Sciences (NIH/NCATS), Grant
the AAP periodic survey. Ambul Pediatr. 2008;8:11-17.
Number UL1TR0030 and the NIH/NICHD Intellectual and
doi:10.1016/j.ambp.2007.10.006
Developmental Disabilities Research Center, Grant Number
10. Freed GL, Dunham KM, Switalski KE, et al. Recently
U54HD079123-05S1.
trained general pediatricians: perspectives on residency
training and scope of practice. Pediatrics. 2009;123(suppl
Ethical Approval 1):S38-S43. doi:10.1542/peds.2008-1578J
This study was approved by Johns Hopkins Institutional 11. Green C, Hampton E, Ward MJ, Shao H, Bostwick S.
Review Board: 00108505 The current and ideal state of mental health training:
pediatric program director perspectives. Acad Pediatr.
ORCID iDs 2014;14:526-532. doi:10.1016/j.acap.2014.05.011
12. Rosenberg AA, Kamin C, Glicken AD, Jones MD,
Belinda Chen https://orcid.org/0000-0001-7330-8602
Jr. Training gaps for pediatric residents planning a
Mary L. O’Connor Leppert https://orcid.org/0000-0003-
career in primary care: a qualitative and quantitative
0679-4895
study. J Grad Med Educ. 2011;3:309-314. doi:10.4300/
JGME-D-10-00151.1
References 13. Boat TF, Land ML, Jr, Leslie LK. Health care workforce
1. Zablotsky B, Black LI, Maenner MJ, et al. Prevalence development to enhance mental and behavioral health of
and trends of developmental disabilities among chil- children and youths. JAMA Pediatr. 2017;171:1031-1032.
dren in the United States: 2009-2017. Pediatrics. doi:10.1001/jamapediatrics.2017.2473
2019;144:e20190811. doi:10.1542/peds.2019-0811 14. McMillan JA, Land ML, Jr, Rodday AM, Wills K, Green
2. Weitzman C, Wegner L; The Section on Developmental CM, Leslie LK. Report of a joint association of pediat-
and Behavioral Pediatrics, Committee on Psychosocial ric program directors—American Board of Pediatrics
Aspects of Child and Family Health; Council on Early Workshop: preparing future pediatricians for the mental
Childhood and Society for Developmental and Behavioral health crisis. J Pediatr. 2018;201:285-291. doi:10.1016/j.
Pediatrics. Promoting optimal development: screen- jpeds.2018.06.044
ing for behavioral and emotional problems. Pediatrics. 15. National Academies of Sciences, Engineering, and
2015;135:384-395. doi:10.1542/peds.2014-3716 Medicine. Training the Future Child Health Care
3. Houtrow AJ, Larson K, Olson LM, Newacheck PW, Workforce to Improve Behavioral Health Outcomes
Halfon N. Changing trends of childhood disability, for Children, Youth, and Families: Proceedings of a
2001-2011. Pediatrics. 2014;134:530-538. doi:10.1542/ Workshop—in Brief. Olson S., ed. National Academies
peds.2014-0594 Press; 2017. doi:10.17226/24789
4. Simon JL, Chesney RW, Alden ER, et al. The future 16. McMillan JA, Land M, Jr, Tucker AE, Leslie LK.
of pediatric education II: organizing pediatric educa- Preparing future pediatricians to meet the behavioral and
tion to meet the needs of infants, children, adoles- mental health needs of children. Pediatrics. 2020;145.
cents and young adults in the 21st century. Pediatrics. e20183796. doi:10.1542/peds.2018-3796
2000;105(suppl):163-212. Accessed November 24, 2020. 17. Ader J, Stille CJ, Keller D, et al. The medical home and
http://pediatrics.aappublications.org/content/pediat- integrated behavioral health: advancing the policy agenda.
rics/105/Supplement_1/163.full.pdf Pediatrics. 2015;135:909-917. doi:10.1542/peds.2014-
5. Johnson RL, Charney E, Cheng TL, et al. Final report 3941
of the FOPE II education of the pediatrician workshop. 18. Kern DE, Thomas PA, Hughes MT. Curriculum
Pediatrics. 2000;106:1175-1198. Development for Medical Education: A Six-Step Approach.
6. Bridgemohan C, Bauer NS, Nielsen BA, et al. A work- 2nd ed. The Johns Hopkins University Press; 2009.
force survey on developmental-behavioral pediatrics. 19. Project ECHO. University of New Mexico School of
Pediatrics. 2018;141:e20172164. doi:10.1542/peds.2017- Medicine. Become a partner. Accessed October 2, 2021.
2164 https://hsc.unm.edu/echo/become-a-partner/
7. American Board of Medical Specialties. ABMS Board 20. Little TD, Chang R, Gorrall BK, et al. The retrospective
Certification Report 2017-2018. Accessed November 2, pretest-posttest design redux: on its validity as an alterna-
2021. https://www.abms.org/wp-content/uploads/2020/10/ tive to traditional pretest-posttest measurement. Int J Behav
abms-board-certification-report-2017-2018.pdf Dev. 2020;44:175-183. doi:10.1177/0165025419877973
Harrison et al 55

21. IBM Corp. IBM SPSS Statistics for Windows, Version 24. Zhou C, Crawford A, Serhal E, Kurdyak P, Sockalingam
26.0. Accessed November 24, 2020. http://www.ibm. S. The impact of Project ECHO on participant and patient
com/analytics/spss-statistics-software outcomes: a systematic review. Acad Med. 2016;91:
22. R Core Team. R: A Language and Environment for 1439-1461. doi:10.1097/ACM.0000000000001328
25. McBain RK, Sousa JL, Rose AJ, et al. Impact of proj-
Statistical Computing. R Core Team; 2020. Accessed
ect ECHO models of medical tele-education: a system-
November 24, 2020. http://www.R-project.org/ atic review. J Gen Intern Med. 2019;34:2842-2857.
23. Arora S, Thornton K, Murata G, et al. Outcomes of treat- doi:10.1007/s11606-019-05291-1
ment for hepatitis C virus infection by primary care pro- 26. Schatz B. ECHO 2019 Act; 2019. Accessed November
viders. N Engl J Med. 2011;364:2199-2207. doi:10.1056/ 2, 2021. https://www.congress.gov/bill/116th-congress/
NEJMoa1009370 senate-bill/1618