The Evaluation of Physicians’ Communication Skills

From Multiple Perspectives

Jenni Burt, PhD1 ABSTRACT
Gary Abel, PhD2 PURPOSE To examine how family physicians’, patients’, and trained clinical raters’
Marc N. Elliott, PhD3 assessments of physician-patient communication compare by analysis of indi-
vidual appointments.
Natasha Elmore, MSc1
METHODS Analysis of survey data from patients attending face-to-face appoint-
Jennifer Newbould, PhD4 ments with 45 family physicians at 13 practices in England. Immediately post-
Antoinette Davey, MPhil2 appointment, patients and physicians independently completed a questionnaire
including 7 items assessing communication quality. A sample of videotaped
Nadia Llanwarne, MPhil5 appointments was assessed by trained clinical raters, using the same 7 communi-
Inocencio Maramba, MSc2 cation items. Patient, physician, and rater communication scores were compared
using correlation coefficients.
Charlotte Paddison, PhD5
RESULTS Included were 503 physician-patient pairs; of those, 55 appointments
John Campbell, MD, FRCGP2 were also evaluated by trained clinical raters. Physicians scored themselves, on
Martin Roland, DM, FRCGP5 average, lower than patients (mean physician score 74.5; mean patient score
1
The Healthcare Improvement Studies 94.4); 63.4% (319) of patient-reported scores were the maximum of 100. The
Institute (THIS Institute), University mean of rater scores from 55 appointments was 57.3. There was a near-zero cor-
of Cambridge, Cambridge Biomedical relation coefficient between physician-reported and patient-reported communica-
Campus, Cambridge, United Kingdom tion scores (0.009, P = .854), and between physician-reported and trained rater-
2
University of Exeter Medical School, reported communication scores (-0.006, P = .69). There was a moderate and
St Luke’s Campus, Exeter, United Kingdom statistically significant association, however, between patient and trained-rater
scores (0.35, P = .042).
RAND Corporation, Santa Monica,
3

California CONCLUSIONS The lack of correlation between physician scores and those of
RAND Europe, Cambridge, United
4 others indicates that physicians’ perceptions of good communication during
Kingdom their appointments may differ from those of external peer raters and patients.
Physicians may not be aware of how patients experience their communication
5
Cambridge Centre for Health Services
practices; peer assessment of communication skills is an important approach in
Research, University of Cambridge School
of Clinical Medicine, Cambridge, United
identifying areas for improvement.
Kingdom
Ann Fam Med 2018;16:330-337. https://doi.org/10.1370/afm.2241.

INTRODUCTION

P
atient-centered communication is fundamental to the practice of
family medicine.1,2 While good communication itself is an impor-
Conflicts of interest: M.R. and J.C. have acted as tant outcome, it is associated with benefits such as improvement
advisors to Ipsos MORI, the Department of Health of clinical outcomes, reduction in medical errors, and facilitation of
and subsequently NHS England on the development
self-management and preventive behaviors.3-11 Internationally, the evalu-
of the English GP Patient Survey. J.B. currently acts
as an advisor to NHS England on the GP Patient ation of physicians’ communication skills is increasing as part of efforts
Survey. No other authors report a conflict of interest. to improve the quality of health care.12-14 Approaches to evaluating and
benchmarking standards of communication have typically relied on
CORRESPONDING AUTHOR patient experience surveys, the results of which are often made public.15,16
Jenni Burt, PhD At the level of the individual, physicians may need to reflect on their own
The Healthcare Improvement Studies Insti- performance alongside ratings from peers, coworkers, and patients as
tute (THIS Institute) part of both regulation and continuing professional development.17-20 For
University of Cambridge example, in the UK, the General Medical Council requires all doctors to
Cambridge Biomedical Campus
Clifford Allbutt Building complete 360-degree evaluation of the care they provide, with patient
Cambridge CB2 0AH UK and colleague feedback used as supporting information for the renewal of
[email protected] their license to practice.21

A NNALS O F FAMILY MEDICINE ✦ WWW.AN N FA MME D.O R G ✦ VO L. 16, N O. 4 ✦ J ULY/AUG UST 2018

330
 P H Y S I C I A N CO M M U N I C AT I O N S K I L L S

Confidence in the instruments used to assess— Sampling and Practice Recruitment

and commonly compare—performance is essential if Practices were eligible if they (1) had more than 1 fam-
they are to contribute meaningfully to quality assur- ily physician (physician) working a minimum of 2 days
ance.22 Extensive work on the reliability and validity a week in direct clinical contact with patients, and
of patient questionnaires has been conducted.23-28 (2) had low scores on physician-patient communica-
Despite this, research shows that doctors often strug- tion items used in the 2009-2010 national GP (general
gle to trust, make sense of, and subsequently respond practitioner) Patient Survey. Low scores were defined
to, feedback from patient surveys.29-31 In fact, evidence as below the lower quartile for mean communication
from evaluations of performance (aggregated across a score, adjusted for patient case-mix (age, sex, ethnicity,
series of appointments) suggests that physicians tend self-rated health, and an indicator of area-level depriva-
to rate themselves more negatively than patients or tion/disadvantage).40 This study was part of a research
peers.32-33 Indeed, physicians’ perceptions of their own program concerned with understanding the full range
competence are frequently out of line with external of patient experiences of communication, from poor
assessments, as patients tend to give particularly favor- to good.41 In England, however, 94% of patients score
able assessments of care in comparison to the physi- all questions addressing GP communication during
cian self-assessments.34-37 The greatest divergence appointments as good or very good in the GP Patient
between self-assessments of physicians and others, Survey: therefore, we specifically sought low-scoring
however, is with those physicians who are, by external practices to maximize the chance of some appoint-
evaluation, the least skilled but the most confident in ments within the practice being given low patient
their abilities (a phenomenon not confined to physi- ratings for communication. We approached eligible
cians alone).34,38,39 practices within the study areas until we had recruited
To date, research in the area of reliability and 13 practices: some practices were known to us from
validity of patient questionnaires has focused on the participation in a previous study in the program.40
evaluation of overall performance assessed across a
series of appointments.18,19 We compared physician, Patient Recruitment
patient, and rater assessments of communication for Data collection took place between August 2012 and
individual appointments to discover where discrepan- July 2014, with recruitment of 1 or 2 physicians at
cies in assessment of care originate and to learn about a time in each practice. As the primary component
physicians’ insight into patients’ perceptions of care of the study involved video-recording the encoun-
during a single encounter. While we considered dif- ter (reported elsewhere42), we based researchers in
ferences in the distribution of scores given by raters, the practice to recruit patients into the study. The
patients, and physicians, our main focus was on cor- research team approached adult patients on their
relations of scores at the appointment level. The cor- arrival in the practice for a face-to-face appointment
relations were considered more important for assessing with a participating physician. Patients received a
the extent to which physicians are able to distinguish summary, a detailed information sheet, and a consent
(1) appointments that more fully met communications form. A member of the research team discussed these
standards from those appointments that did so to a with each participating patient in order to obtain
lesser extent, and (2) appointments that resulted in informed consent.
better patient experiences from those that resulted in
worse patient experiences. The correlation of patient Patient and Physician Ratings
and rater scores is also of interest as it illuminates the Immediately following the appointment, the patient
extent to which use of communication best practices was asked to complete a short questionnaire. The
may improve patient experience. questionnaire included a set of 7 items taken from the
national GP Patient Survey to assess physician-patient
communication (Table 1) and basic sociodemographic
METHODS questions. Also, following the appointment, physicians
We present an analysis of data collected in a study answered the same 7 items about their own communi-
conducted in family practices in England in 2 broad cation performance in that encounter. From these, we
geographic areas (Devon, Cornwall, Bristol, and Som- calculated separate scores of communication during the
erset; and Cambridgeshire, Bedford, Luton, and North appointment, from the patient responses and from the
London). Approval for the study was obtained from physician responses. In line with previous work, each
the National Research Ethics Service Committee East was calculated by linearly rescaling responses from
of England – Hertfordshire on 11 October 2011 (ref: 0 to 100 and calculating the mean of all informative
11/EE/0353). responses where 4 or more informative answers were

A NNALS O F FAMILY MEDICINE ✦ WWW.AN N FA MME D.O R G ✦ VO L. 16, N O. 4 ✦ J ULY/AUG UST 2018

331
 P H Y S I C I A N CO M M U N I C AT I O N S K I L L S

given.40,43 Responses of “doesn’t apply” were considered whether individual physicians’ scores for particular
uninformative and excluded. appointments increased when patients also rated them
higher. This mixed model was performed initially
Trained Clinical Rater Ratings with a single fixed effect (patient-reported scores) and
In addition to physician self ratings and patient rat- subsequently adjusted for patient demographics (age,
ings, 56 of the consultations were selected for rating sex, ethnicity, and self-rated health) to account for
by experienced, trained clinical raters (all family phy- the fact that some types of patients were more likely
sicians). The selection of appointments was made on to give positive ratings. Another model performed
the basis of patient ratings of communication, with the included physician sex, whether they were UK quali-
aim of maximizing the variation of patient-reported fied, and the years since they qualified, to adjust for
communication quality. To increase reliability, 4 rat- any differences not captured by the random effect for
ers scored each appointment, using both the Global physician. Standardized regression coefficients (betas)
Consultation Rating Scale44 and the same set of 7 items are reported. These are directly comparable to (and
taken from the GP Patient Survey used by patients in the case of models with a single exposure, equal
and physicians. Full details of the rating process were to) correlation coefficients. Because of potential con-
reported in a previous publication, which showed a cerns over normality assumptions, bootstrapping was
weak correlation between patient ratings of physician used in all analyses with 500 bootstrap samples. To
communication and trained raters scores using the account for the nonindependence of observations due
Global Consultation Rating Scale.42 In this analysis, to physicians being represented more than once, we
we made use of the items derived from the GP Patient performed the bootstrap sampling clustered by physi-
Survey, calculating scores as described above. Each of cian. All analysis was carried out using Stata V13.1
the raters scored appointments in a different random (StataCorp LP).
order to minimize any order effects (using simple
randomization) and the same raters were used for all
appointments. The mean of the scores from the 4 rat- RESULTS
ers was calculated for each appointment. A total of 908 patients had face-to-face appoint-
ments with 45 participating physicians during periods
Statistical Analyses of patient recruitment. Of these, 167 (18.4%) were
We calculated correlation coefficients comparing ineligible (mostly children) and, of the remainder,
physician and patient scores for the full sample and 529 completed a questionnaire (71.4% response rate).
physician, patient, and rater scores for a subsample. An additional 26 (4.9%) appointments were excluded
To evaluate the within-physician association between due to missing data, leaving 503 physician-patient
patient and physician scores, we used a mixed linear appointment pairings in the data set (Supplemental
regression with a random effect (intercept) for each Figure 1, available at http://www.annfammed.org/
physician on the full sample. This model accounts content/16/4/330/suppl/DC1/). Table 2 shows self-
for the fact that some physicians may be more gener- reported demographic characteristics of patients. For
ous or more critical than others, and thus assessed 4 physicians, data on sex, country of qualification,

Table 1. Physician-Patient Communication Items

Instructions given to patients: “Thinking about the consultation which took place today, how good was the doctor at each of the
following?a Please put an ✗ in 1 box for each row.”

Very Neither good Very Doesn’t

good Good nor poor Poor poor applyb
Giving you enough time      
Asking about your symptoms      
Listening to you      
Explaining tests and treatments      
Involving you in decisions about your care      
Treating you with care and concern      
Taking your problems seriously      
a
Amended for physicians to read “How good were you at each of the following?” Items were also reworded, for example, “Giving the patient enough time.” Raters
were asked, “Thinking about this consultation, how good was the ...?”
b
Considered to be uninformative for the purposes of our analysis.

A NNALS O F FAMILY MEDICINE ✦ WWW.AN N FA MME D.O R G ✦ VO L. 16, N O. 4 ✦ J ULY/AUG UST 2018

332
 P H Y S I C I A N CO M M U N I C AT I O N S K I L L S

50, on average they gave themselves much

Table 2. Self-Reported Demographics for Patients Who
lower scores than patients. The lack of any
Completed a Questionnaire
clear relationship in Figure 2 is reflected in
Characteristic N (%) Characteristic N (%) the very low correlation coefficient shown in
Table 3, again with no evidence of an associa-
Sex Self-rated health
Male 199 (39.56) Excellent 49 (9.74)
tion (P = .854). The lack of association persists
Female 304 (60.44) Very good 163 (32.41)
when considering within-physician associa-
Age, y Good 179 (35.59) tions and when further adjusting for patient
18-24 36 (7.16) Fair 78 (15.51) demographics (Table 3). Additional adjust-
25-34 76 (15.11) Poor 34 (6.76) ment for physician factors had no meaningful
35-44 61 (12.13) Ethnicity impact on the regression coefficient or P value
45-54 78 (15.51) White 458 (91.05) for physician self-rating.
55-64 83 (16.50) Mixed 4 (0.80)
65-74 93 (18.89) Asian or Asian British 15 (2.98)Physician, Patient, and Rater Comparison
75-84 59 (11.73) Black or Black British 21 (4.17)
Figure 3 shows the distribution of physician-
85+ 15 (2.98) Chinese 4 (0.80)
reported, patient-reported, and rater-reported
Other 1 (0.20)
scores for the 55 appointment subsample. The
bi-modal distribution of patient scores reflects
and date of qualification were not available. Of the the way appointments were sampled, while the physi-
56 appointments selected for evaluation by raters, 55 cian self-rated scores were distributed similarly to the
(98%) had complete physician and patient scores and full sample. The raters scored appointments over a
the subsample analysis was restricted to these appoint- wider range than either patients or physicians, from
ments. The individual rater scores for the 55 consulta- 23.2 to 87.5 (mean 57.3), and their scores were less
tions were strongly correlated with each other (pair- skewed than those of patients. Figure 4 shows scatter
wise Spearman correlation coefficients varied between plots comparing the 3 sets of ratings. Similar to the
0.54 and 0.67, P <.0001 for all, see Supplemental full data set shown in Figure 2, there is no associa-
Table 1, available at http://www.annfammed.org/con- tion between physician scores and patient scores
tent/16/4/330/suppl/DC1/) giving confidence that the in the subset of appointments evaluated by raters.
scale was being used consistently and that using the Furthermore, there is no association between physi-
mean of the 4 rater scores was appropriate. cian scores and the scores of raters, although there
is a tendency for patient scores to be higher when
Physician and Patient Comparison the rater scores were also higher. These relationships
Figure 1 shows the distribution of physician-reported are reflected in the correlation coefficients of 0.015
and patient-reported scores for the full sample (503 (P = .91) for physicians and patients, -0.006 (P = .69)
appointments). Physician scores of their performance for physicians and raters, and 0.35 (P = .042) for
were fairly symmetrically distributed and ranged patients and raters. The only pair with any statisti-
from 39.3 to 100 (mean 74.5), with only 5.4% (27) cally significant and nontrivial association is between
of appointments being given the maximum score of the scores of patients and raters.
100. In contrast, the distri-
bution of patient-reported Figure 1. Distribution of scores for the full sample.
scores is highly skewed, with
63.4% (319) of patients giv- 250 400
ing the maximum score of
200
100 (range 32.1 to 100, mean 300
94.4). A scatter plot com-
Frequency
Frequency

150
paring physician-reported 200
scores with patient-reported 100

scores of the same appoint- 50

100
ment is shown in Figure 2.
The skewed nature of patient 0 0
scores is evident in this fig- 0 20 40 60 80 100 0 20 40 60 80 100

ure, which also shows that, Physician scores Patient scores

while physicians do not often Full sample (n = 503)
score themselves lower than

A NNALS O F FAMILY MEDICINE ✦ WWW.AN N FA MME D.O R G ✦ VO L. 16, N O. 4 ✦ J ULY/AUG UST 2018

333
 P H Y S I C I A N CO M M U N I C AT I O N S K I L L S

DISCUSSION found no correlation between physician and patient

In this examination of family physicians’, patients’, and scores or between physician and rater scores, and a
trained clinical raters’ assessments of physician-patient moderate correlation between patient and rater scores.
communication during individual appointments, we Our results suggest that family physicians draw
on different constructs of good
Figure 2. Scatterplot illustrating the association between physician communication compared with
and patient scores. patients and trained clinical rat-
ers, when asked to complete the
100
same evaluation items. Previ-
ous research has documented a
mismatch between physicians’
assessments of patient expecta-
80
tions, their subsequent commu-
nication behaviors, and patient
Physician scores

perceptions of these behav-

iors, most notably in pediatric
appointments.34-37 Our find-
60 ings suggest that a divergence
between physician and patient
expectations of communication
practices may be common in pri-
mary care. Therefore, physicians’
40 self-perceptions alone may be
of limited value for identifying
20 40 60 80 100 aspects of their patient-centered
Patient scores communication practices which
could be strengthened or
improved. Raters are more likely
Table 3. Standardized Regression Coefficients for Physician and
to share patient perceptions of
Patient Scores (n = 503)
what good communication looks
Standardised Regression P like. Additionally, raters may
Association Coefficients (95% CI) value pinpoint aspects of physicians’
Overall association 0.009 (–0.086 to 0.104) .854 communication behaviors which
Within-physician association 0.025 (–0.060 to 0.110) .565 are not perceived by patients,
Within-physician association adjusted for patient 0.023 (–0.064 to 0.110) .608 or at least not reported in a
sociodemographics
post-consultation survey.33 Mul-
Within-physician association adjusted for patient 0.051 (–0.044 to 0.146) .291
sociodemographics and physician factors a tisource feedback for the assess-
ment of physician performance
Restricted to 451 consultations where physician data were available.
a

is now an established tool for

Figure 3. Distribution of scores for the subsample.

30 30 15

20 20 10
Frequency
Frequency

Frequency

10 10 5

0 0 0
0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100
Physician scores Patient scores Rater scores

Subsample rated by trained raters (n = 55)

A NNALS O F FAMILY MEDICINE ✦ WWW.AN N FA MME D.O R G ✦ VO L. 16, N O. 4 ✦ J ULY/AUG UST 2018

334
 P H Y S I C I A N CO M M U N I C AT I O N S K I L L S

evaluating the quality of care, with increasing evi- interest, although they must be interpreted with some
dence of impact on physician behaviors.45,46 Our study caution. Patients provided more generous scores, on
provides further evidence for the importance of exter- average, than raters or physicians. High patient scores
nal assessment of physicians’ communication skills by reflect, in part, the fact that some patients are inhib-
trained peers as a first step in improving the standard ited about identifying poor communication on patient
of physician-patient communication. experience questionnaires.47 The reluctance of some
The differences we observed in the distribution of patients to report poor experiences is likely to result in
scores used by raters, patients, and physicians are of weaker correlations between patient and rater assess-
ments of communication than would otherwise occur.
In aggregate, patient ratings are able to distinguish the
Figure 4. Scatterplots illustrating associations quality of physician performance overall.48,49 Given the
between physician, patient, and rater scores. different range of scores used by each group (patients,
100
physicians, raters) on the same response scale, how-
ever, we suggest that patient experience scores are
90 best interpreted as a relative measure of the patient
experience, rather than being interpreted on an abso-
Physician scores

80 lute scale.42 This further supports the need for external

peer assessment of communication skills, as patient
70 feedback alone is unlikely to identify specific needs for
support and training in this area.
60 Our study has a number of limitations. We selected
practices to increase the likelihood of identifying
50 appointments with lower patient scores. Within each
20 40 60 80 100 practice, not all physicians took part. If physicians who
Patient scores participated were more skilled at communicating with
patients, we may have reduced the variation in quality
100 of communication in our sample, thus reducing study
power and the strength of the observed correlations.
90 We asked physicians to assess their communica-
tion performance immediately after each appointment,
Physician scores

80
when time may have been short. Thus our findings
may not be generalizable to other forms of self-
70
reflection where more time is taken, for example, in
review of video-recorded appointments. On the other
60
hand, our method of data collection may be repre-
50
sentative of the informal self-evaluation that routinely
20 40 60 80 100 occurs among physicians. We additionally note that
we did not assess the compliance of each participating
Rater scores
physician to our request to complete an assessment
after every appointment. While we collected assess-
100
ments at the end of each surgery, reliability may have
been reduced if physicians completed assessments in
80 batches following a series of appointments. Patients
Patient scores

completed questionnaires immediately following their

60 appointment, usually in the practice waiting area.
Social desirability bias may have increased the likeli-
hood of patients giving positive assessments of care.
40
Additionally, most patients in the study self-reported
as white and our findings may not generalize well to
20 patients of different racial and ethnic backgrounds.
20 40 60 80 100 Patient feedback is, and should remain, a central
Rater scores component of assessments of the quality of care. Our
Note: The gray lines are lines of best fit.
findings, however, support the role of trained peer
assessors in examining the communication practices of

A NNALS O F FAMILY MEDICINE ✦ WWW.AN N FA MME D.O R G ✦ VO L. 16, N O. 4 ✦ J ULY/AUG UST 2018

335
 P H Y S I C I A N CO M M U N I C AT I O N S K I L L S

physicians in any multisource assessment investigat- 9. Kennedy A, Gask L, Rogers A. Training professionals to engage
with and promote self-management. Health Educ Res. 2005;​20(5):​
ing standards of care. We would further suggest that 567-578.
the presentation of feedback from such assessments 10. Cohen D, Longo MF, Hood K, Edwards A, Elwyn G. Resource effects
should include support for physicians to better attune of training general practitioners in risk communication skills and
themselves to the perceptions and communication shared decision making competences. J Eval Clin Pract. 2004;​10(3):​
439-445.
needs of their patients.
11. Kroenke K. A practical and evidence-based approach to common
To read or post commentaries in response to this article, see it symptoms:​a narrative review. Ann Intern Med. 2014;​161(8):​579-586.
online at http://www.AnnFamMed.org/content/16/4/330.
12. Health Research Institute. Scoring Healthcare:​Navigating Customer
Experience Ratings. New York, NY:​PricewaterhouseCoopers;​2013.
Key words: physician-patient relations; health care surveys; quality of
13. Department of Health. Hard Truths:​The Journey to Putting Patients
health care; patient satisfaction; patient experience; physician-patient
First. London, UK:​The Stationery Office;​2014.
communication; health care quality measurement
14. Ahmed F, Burt J, Roland M. Measuring patient experience:​con-
cepts and methods. Patient. 2014;​7(3):​235-241.
Submitted September 1, 2017; submitted, revised, January 30, 2018;
accepted February 27, 2018. 15. Agency for Healthcare Research and Quality (AHCRQ). CAHPS Clini-
cian and Group Surveys. https:​//cahps.ahrq.gov/surveys-guidance/
cg/index.html. Published 2018. Accessed Jan 27, 2018.
Funding support: This work was funded by a National Institute for
Health Research Programme Grant for Applied Research (NIHR PGfAR) 16. Ipsos MORI. GP Patient Survey. https:​//gp-patient.co.uk/. Published
program (RP-PG-0608-10050). 2018. Accessed Jan 27, 2018.
17. Lockyer J. Multisource feedback in the assessment of physician
Department of Health disclaimer: The views expressed are those of the competencies. J Contin Educ Health Prof. 2003;​23(1):​4 -12.
author(s) and not necessarily those of the National Health Service, the 18. Overeem K, Wollersheim HC, Arah OA, Cruijsberg JK, Grol RP, Lom-
National Institute for Health Research, or the Department of Health. barts KM. Evaluation of physicians’ professional performance:​an
iterative development and validation study of multisource feedback
Acknowledgments: We would like to thank the patients, practice man- instruments. BMC Health Serv Res. 2012;​12:​80.
agers, family physicians, and other staff of the general practices who 19. Wright C, Richards SH, Hill JJ, et al. Multisource feedback in evalu-
kindly agreed to participate in this study and without whom the study ating the performance of doctors:​the example of the UK General
would not have been possible. Particular acknowledgment goes to our Medical Council patient and colleague questionnaires. Acad Med.
4 trained clinical raters for their contribution to this work, and to James 2012;​87(12):​1668-1678.
Brimicombe, our data manager, who developed the online rating sys- 20. Lockyer J. Multisource feedback:​can it meet criteria for good
tem. We would also like to thank the Improve Advisory Group for their assessment? J Contin Educ Health Prof. 2013;​33(2):​89-98.
input and support throughout this study. 21. General Medical Council. Colleague and patient feedback for
revalidation. http:​//www.gmc-uk.org/doctors/revalidation/colleague_
 upplementary Materials: Available at http://www.AnnFamMed.
S patient_feedback.asp. Published 2018. Accessed Jan 27, 2018.
org/content/16/4/330/suppl/DC1/. 22. Campbell JL, Richards SH, Dickens A, Greco M, Narayanan A, Brear-
ley S. Assessing the professional performance of UK doctors:​an
evaluation of the utility of the General Medical Council patient and
References colleague questionnaires. Qual Saf Health Care. 2008;​17(3):​187-193.

1. Buetow SA. What do general practitioners and their patients want 23. Campbell J, Wright C, Primary Care Research Group, Peninsula
from general practice and are they receiving it? A framework. Soc College of Medicine & Dentistry. GMC Multi-Source Feedback Ques-
Sci Med. 1995;​40(2):​213-221. tionnaires. Interpreting and handling multisource feedback results:​
Guidance for appraisers. General Medical Council Web site. https:​
2. Wensing M, Jung HP, Mainz J, Olesen F, Grol R. A systematic review //www.gmc-uk.org/Information_for_appraisers.pdf_48212170.pdf
of the literature on patient priorities for general practice care. Part Published Feb 1, 2012. Accessed Jan 27, 2018.
1:​Description of the research domain. Soc Sci Med. 1998;​47(10):​
1573-1588. 24. Elliott MN, Edwards C, Angeles J, Hambarsoomians K, Hays RD.
Patterns of unit and item nonresponse in the CAHPS Hospital Sur-
3. Anhang Price R, Elliott MN, Zaslavsky AM. Valuing patient experi- vey. Health Serv Res. 2005;​40(6 Pt 2):​2096-2119.
ence as a unique and intrinsically important aspect of health care
quality. JAMA Surg. 2013;​148(10):​985-986. 25. Ipsos MORI. GP Patient Survey - Technical Annex:​2016-17 Annual
Report. London, UK:​NHS England;​2017. https:​//gp-patient.
4. Anhang Price R, Elliott MN, Zaslavsky AM, et al. Examining the role co.uk/downloads/archive/2017/GPPS%202017%20Technical%20
of patient experience surveys in measuring health care quality. Med Annex%20PUBLIC.pdf. Pulished Jul 6, 2017. Accessed Jan 27, 2018.
Care Res Rev. 2014;​71(5):​522-554.
26. Klein DJ, Elliott MN, Haviland AM, et al. Understanding nonre-
5. Stewart MA. Effective physician-patient communication and health sponse to the 2007 Medicare CAHPS survey. Gerontologist. 2011;​
outcomes:​a review. CMAJ. 1995;​152(9):​1423-1433. 51(6):​843-855.
6. Kuzel AJ, Woolf SH, Gilchrist VJ, et al. Patient reports of prevent- 27. Martino SC, Weinick RM, Kanouse DE, et al. Reporting CAHPS and
able problems and harms in primary health care. Ann Fam Med. HEDIS data by race/ethnicity for Medicare beneficiaries. Health Serv
2004;2(4):333-340. Res. 2013;​48(2 Pt 1):​417-434.
7. Doyle C, Lennox L, Bell D. A systematic review of evidence on the 28. O’Malley AJ, Zaslavsky AM, Elliott MN, Zaborski L, Cleary PD.
links between patient experience and clinical safety and effective- Case-mix adjustment of the CAHPS Hospital Survey. Health Serv Res.
ness. BMJ Open. 2013;​3(1):​e001570. 2005;​40(6 Pt 2):​2162-2181.
8. Kennedy M, Denise M, Fasolino M, et al. Improving the patient 29. Asprey A, Campbell JL, Newbould J, et al. Challenges to the cred-
experience through provider communication skills building. Patient ibility of patient feedback in primary healthcare settings:​a qualita-
Exp J. 2014;​1(1):​56-60. tive study. Br J Gen Pract. 2013;​63(608):​e200-e208.

A NNALS O F FAMILY MEDICINE ✦ WWW.AN N FA MME D.O R G ✦ VO L. 16, N O. 4 ✦ J ULY/AUG UST 2018

336
 P H Y S I C I A N CO M M U N I C AT I O N S K I L L S

30. Boiko O, Campbell JL, Elmore N, Davey AF, Roland M, Burt J. The 41. Burt J, Campbell J, Abel G, et al. Improving patient experience
role of patient experience surveys in quality assurance and improve- in primary care:​a multimethod programme of research on the
ment:​a focus group study in English general practice. Health measurement and improvement of patient experience. Programme
Expect. 2015;​18(6):​1982-1994. Grants for Applied Research. 2017;​5:​1-452.
31. Farrington C, Burt J, Boiko O, et al. Doctors’ engagements with 42. Burt J, Abel G, Elmore NL, et al. Rating communication in GP con-
patient experience surveys in primary and secondary care:​a quali- sultations:​the association between ratings made by patients and
tative study. Health Expect. 2017;​20(3):​385-394. trained clinical raters. Med Care Res Rev. 2018;​75(2):​201-218.
32. Kenny DA, Veldhuijzen W, Weijden Tv, et al. Interpersonal percep- 43. Lyratzopoulos G, Elliott M, Barbiere JM, et al. Understanding ethnic
tion in the context of doctor-patient relationships:​a dyadic analysis and other socio-demographic differences in patient experience of
of doctor-patient communication. Soc Sci Med. 2010;​70(5):​763-768. primary care:​evidence from the English General Practice Patient
Survey. BMJ Qual Saf. 2012;​21(1):​21-29.
33. Roberts MJ, Campbell JL, Richards SH, Wright C. Self-other agree-
ment in multisource feedback:​the influence of doctor and rater 44. Burt J, Abel G, Elmore N, et al. Assessing communication quality of
group characteristics. J Contin Educ Health Prof. 2013;​33(1):​14-23. consultations in primary care:​initial reliability of the Global Consul-
tation Rating Scale, based on the Calgary-Cambridge Guide to the
34. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE,
Medical Interview. BMJ Open. 2014;​4(3):​e004339.
Perrier L. Accuracy of physician self-assessment compared with
observed measures of competence:​a systematic review. JAMA. 45. Ferguson J, Wakeling J, Bowie P. Factors influencing the effective-
2006;​296(9):​1094-1102. ness of multisource feedback in improving the professional practice
of medical doctors:​a systematic review. BMC Med Educ. 2014;​14:​76.
35. Eva KW, Regehr G. “I’ll never play professional football” and other
fallacies of self-assessment. J Contin Educ Health Prof. 2008;​28(1):​ 46. Al Ansari A, Donnon T, Al Khalifa K, Darwish A, Violato C. The con-
14-19. struct and criterion validity of the multi-source feedback process to
assess physician performance:​a meta-analysis. Adv Med Educ Pract.
36. Gordon MJ. A review of the validity and accuracy of self-
2014;​5:​39-51.
assessments in health professions training. Acad Med. 1991;​66(12):​
762-769. 47. Burt J, Newbould J, Abel G, et al. Investigating the meaning of
‘good’ or ‘very good’ patient evaluations of care in English general
37. Campbell J, Smith P, Nissen S, Bower P, Elliott M, Roland M. The
practice:​a mixed methods study. BMJ Open. 2017;​7(3):​e014718.
GP Patient Survey for use in primary care in the National Health
Service in the UK—development and psychometric characteristics. 48. Lyratzopoulos G, Elliott MN, Barbiere JM, et al. How can health
BMC Fam Pract. 2009;​10:​57. care organizations be reliably compared?:​Lessons from a national
survey of patient experience. Med Care. 2011;​49(8):​724-733.
38. Kruger J, Dunning D. Unskilled and unaware of it:​how difficul-
ties in recognizing one’s own incompetence lead to inflated self- 49. Roland M, Elliott M, Lyratzopoulos G, et al. Reliability of patient
assessments. J Pers Soc Psychol. 1999;​77(6):​1121-1134. responses in pay for performance schemes:​analysis of national
General Practitioner Patient Survey data in England. BMJ. 2009;​339:​
39. Carter TJ, Dunning D. Faulty self-assessment:​why evaluating one’s
b3851.
own competence is an intrinsically difficult task. Soc Personal Psychol
Compass. 2008;​2(1):​346-360.
40. Roberts MJ, Campbell JL, Abel GA, et al. Understanding high and
low patient experience scores in primary care:​analysis of patients’
survey data for general practices and individual doctors. BMJ. 2014;​
349:​g6034.

A NNALS O F FAMILY MEDICINE ✦ WWW.AN N FA MME D.O R G ✦ VO L. 16, N O. 4 ✦ J ULY/AUG UST 2018

337