Journal of Sports Sciences

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/rjsp20

The running shoe comfort assessment tool (RUN-

CAT): Development and evaluation of a new multi-
item assessment tool for evaluating the comfort of
running footwear

Christopher Bishop , Jonathan D. Buckley , Adrian E. Esterman & John B.

Arnold

To cite this article: Christopher Bishop , Jonathan D. Buckley , Adrian E. Esterman & John
B. Arnold (2020): The running shoe comfort assessment tool (RUN-CAT): Development and
evaluation of a new multi-item assessment tool for evaluating the comfort of running footwear,
Journal of Sports Sciences, DOI: 10.1080/02640414.2020.1773613

To link to this article: https://doi.org/10.1080/02640414.2020.1773613

View supplementary material

Published online: 06 Jun 2020.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=rjsp20
JOURNAL OF SPORTS SCIENCES
https://doi.org/10.1080/02640414.2020.1773613

PHYSICAL ACTIVITY, HEALTH AND EXERCISE

The running shoe comfort assessment tool (RUN-CAT): Development and evaluation
of a new multi-item assessment tool for evaluating the comfort of running footwear
Christopher Bishopa,b, Jonathan D. Buckley a
, Adrian E. Esterman c,d
and John B. Arnold a

a
Alliance for Research in Exercise, Nutrition & Activity (ARENA), Allied Health & Human Performance, University of South Australia, Adelaide, Australia;
b
The Biomechanics Lab, Adelaide, Australia; cClinical & Health Sciences, University of South Australia, Adelaide, Australia; dAustralian Institute of
Tropical Health and Medicine, James Cook University, Townsville, Australia

ABSTRACT ARTICLE HISTORY

Comfort is important for running shoe prescription in athletes to enhance performance and potentially Accepted 26 February 2019
decrease injury risk. A three-stage process was used to develop a new running footwear comfort
KEYWORDS
assessment tool (RUN-CAT): (i) a survey of 282 runners to identify meaningful items of comfort, (ii-a) Footwear; comfort; running;
field testing of 100 runners who assessed the comfort of different shoes, (ii-b) item reduction using cushioning; stability
bootstrap aggregation and weightings using multiple regressions to identify a final set of items, and (iii)
defining test-retest reliability, standard error of measurement (SEM), minimal detectable difference
(MDD90) and minimal important difference (MID) values for the final tool. Of the 19 initial items, after
item reduction, four were included in the final tool: heel cushioning, shoe stability, forefoot cushioning
and forefoot flexibility. Reliability of the overall comfort score was good to excellent (within-day ICC 0.88,
between-day 0.70) with all four components displaying good reliability (ICC >0.70). The SEM of the
comfort score was 2.8 points and the MDD90 was 6.5 mm. Subject nominated MID values ranged from 9.3
to 9.9 mm. The RUN-CAT demonstrates excellent reliability, acceptable measurement error and can
discriminate between footwear models. Clinicians and researchers can incorporate the RUN-CAT to
optimise running shoe comfort in athletes.

Introduction (Hagen et al., 2010; Mündermann, Nigg, Stefanyshyn, Humble

et al., 2002). However, there are numerous methodological and
Footwear comfort is an important component of running
practical issues with existing tools and methods of footwear
shoe prescription not only for athletes, but also for retail
comfort assessment. Since different aspects of footwear con­
staff selling shoes and health professionals using footwear
tribute to overall perception of comfort (Tay et al., 2016; Zhang
to treat their patients. Not only are athletes unlikely to pur­
et al., 1996), overly simplistic methods that rely on one
chase uncomfortable footwear, but comfortable footwear can
response have limited content validity (Lynn, 1986) and scores
also reduce the incidence of foot-related complaints (Mailler-
from Likert-type rating scales are not stable for footwear com­
Savage & Adams, 2006; Mundermann et al., 2001; Nigg et al.,
fort assessment between days (Mills et al., 2010). The use of
2015) and improve running performance (Luo et al., 2009). On
a control shoe limits the utility of a tool in real-world settings,
a subjective level, when asking runners about their footwear,
whilst reference to one’s own footwear can skew scores
fit, comfort and cushioning are the most frequently men­
depending on the type or condition of the existing footwear
tioned features of shoes (Schubert et al., 2011) with comfort
(Hirschmüller et al., 2009). While tools with multiple questions
being influenced by factors such as shoe tightness, cushion­
that use visual analogue scales address these issues (Mills et al.,
ing, stability, support, foot alignment and type of activity
2010; Mündermann, Nigg, Stefanyshyn, Humble et al., 2002),
(Hagen et al., 2010; Miller et al., 2000; Mundermann et al.,
none have been prospectively developed taking into account
2001). This suggests that matching shoe design and features
the aspects of comfort that are most meaningful to a broad
to the most comfortable footwear to the individual may be
sample of runners. If such a tool did exist, it could be used for
useful for optimising footwear comfort.
research purposes, and to help identify the most comfortable
Comfort has been measured in a variety of ways, ranging
running shoe for an individual runner prior to purchase.
from a simple Yes/No question for overall perception (Hoerzer
To the authors’ knowledge, no footwear comfort assessment
et al., 2016), to multiple visual analogue scales for different
tool exists that has been developed in a systematic fashion that
footwear design features, such as cushioning and arch support
(Dinato et al., 2015; Mills et al., 2010; Mündermann, Nigg, (1) incorporates meaningful comfort items, (2) is reliable and
Stefanyshyn, Humble et al., 2002). Other methods include rank­ sensitive to detect differences between footwear models, (3)
ing or scoring of multiple shoes in order of preference (Mills can determine the contribution of different shoe design
et al., 2010), relative to one’s own footwear (Hirschmüller et al., aspects to the overall perception of shoe comfort and (4) has
2009) or relative to a standardised control shoe or condition established thresholds for what defines a meaningful change.

CONTACT Christopher Bishop [email protected] University of South Australia, PO Box 2471, Adelaide, SA 5001, Australia
Supplemental data for this article can be accessed here.
© 2020 Informa UK Limited, trading as Taylor & Francis Group
2 C. BISHOP ET AL.

Therefore, the first aim of this study was to develop a new shoe size range Euro 36–47) were recruited to test the comfort of
running footwear comfort assessment tool taking into account three different shoes during a standardised running session.
these requirements. The second aim was to determine the Participants were recruited through expressions of interest from
reliability, discriminative ability, thresholds for measurement notice boards in the general community, running clubs and social
error and important change for the newly developed tool. media outlets. Participants were eligible if they were aged
18–50 years and ran at least 5 km (km) per week. Subjects were
excluded if they reported an injury in the last 3 months, had
Methods
current blisters or skin conditions on their feet, had undergone
This study consisted of three stages. Stage one involved the previous orthopaedic lower limb surgery, or reported a medical
completion of an online questionnaire by recreational runners history that could influence running gait or sensation in their feet.
conducted to identify meaningful items of shoe comfort. Stage
two recruited recreational runners to assess the comfort of Data collection
three different shoes using a draft tool informed by the results All participants completed a self-paced 1.1 km running session
of Stage One. Item reduction from the results of Stage Two was on road in three footwear conditions (Figure 1(c)). The three
performed to formulate a refined list of items and the final shoes were chosen to reflect three distinct levels of footwear
comfort index. Stage Three investigated the reliability, discri­ (and theoretically comfort) in that Shoe A was a top-of the
minative ability and the difference in comfort that was actually range running shoe (AUD 240 USD), Shoe B was an entry-level
deemed important to the runner (represented using MID running shoe (AUD 100 USD) and Shoe C was a casual shoe with
thresholds) of the new comfort index in a separate group of minimal cushioning and no specific technical design features.
runners. Methodological details relating to each stage of devel­ The individual properties of each of the three shoes are out­
opment are reported below. This study was approved by the lined in Table 1. All shoes were fitted using a Brannock device
local Human Research Ethics Committee and all participants (The Brannock Device Company, USA). A stopwatch was used to
provided informed consent prior to participation. ensure consistency in duration of running time between con­
ditions. A 5-min rest period was provided between testing
Stage one conditions. At the end of each run, a comfort assessment was
completed. The order of testing each condition was rando­
Online questionnaire mised using a computer-generated randomisation matrix.
An initial list of 19 items relating to different aspects of
footwear comfort were included in the initial questionnaire Evaluation of preliminary assessment tool (RUN-CAT)
(Figure 1(a)). These items were identified based on; (i) Based on the ranking of importance by the 100 runners who
a search of the literature to identify items used in previous screened the initial list of 19 items, seven items scored less
running footwear comfort scales (Mills et al., 2010; than 6.52 so were excluded. Therefore, the preliminary RUN-
Mündermann, Nigg, Stefanyshyn, Humble et al., 2002), (ii) CAT consisted of 12 original footwear items. Overall shoe
an online survey of 70 technical running shoe retail store comfort was also assessed (total = 13 items). Two items were
employees and 92 Podiatrists from Australia and New removed from the preliminary RUN-CAT as they were con­
Zealand, and (iii) addition of extra items based on the sidered redundant. Firstly, side-to-side stability was deemed
clinical and research experience of two of the authors (CB to assess the same item as shoe-stability. Secondly, shoe
and JA). This approach yielded a total 19 items. To deter­ length was excluded based on the fact that the shoes were
mine the importance of each item, a sample of recreational fitted to each individual. The refined RUN-CAT tool used for
runners was recruited via social media to complete the shoe assessment in Phase Two consisted of 10 items
questionnaire to rate the complete list of items according (Figure 1(b)). Each footwear item was assessed on an
to the question: ‘when determining how comfortable your anchored horizontal 0–100 mm Visual Analogue Scale (VAS,
running shoes are, how important is the following?”. All see example in Figure 1(d)). Overall comfort was anchored as
items were rated by participants on an 11-point Likert- 0 = shoe not comfortable at all and 100 = most comfortable
type scale ranging from “0 – of no importance” to “10 – shoe imaginable. Using heel stability as an example, each
extremely important”. To be deemed meaningful to runners, predictive footwear items were anchored as 0 = not stable at
a mean cut-off score of ≥6.52/10 was required for that item all, 50 = ideal heel stability and 100 = heel too stable.
to be carried forward into stage two. This threshold was
based on the average importance score of all respondents Item and data reduction
for all items assessed. Participants were asked to assess whether each footwear item
was ideal in each shoe. A shoe that was either assessed as “not
Stage two stable” (VAS score of 0) or “too stable” (VAS score of 100) would
be suboptimal, and therefore not ideal. To address this issue,
Footwear comfort data collection and tool refinement the 100 mm VAS data were transformed using the formula:

Participants
One hundred recreational runners (48 male: 52 females, mean
age 32.6 SD 8.9 years; height 1.73 SD 0.09 m; body mass Thus, using the transformed dataset, 0 = not ideal and
72.5 ± 13.4 kg; 1.1 km running time = 4 minutes 43 ± 47 seconds; 100 = most ideal situation relating to each item.
 JOURNAL OF SPORTS SCIENCES 3

Figure 1. The development process of the running footwear comfort assessment tool (RUN-CAT). 1(a) – the initial list of 19 footwear items used in Stage One. 1(b) – the
refined list of footwear items used in data collection in Stage Two. 1(c) – Shoe models used in data collection of Stage Two. 1(d) – Example data collection tool
demonstrating anchoring of 100 mm VAS. 1(e) – the results of Stage One investigating the meaningfulness of each footwear item. Items shaded grey did not meet the
pre-determined threshold of meaningfulness and were excluded from further consideration.

Data analysis
To identify meaningful items, the contribution of each footwear included in the model. Bootstrap aggregation (bagging) on
item to overall footwear comfort was assessed using a multiple the full model was used to eliminate variables not likely to be
regression model. All 10 items assessed in Stage Two were strong predictors of overall comfort (dependent variable).
4 C. BISHOP ET AL.

Table 1. Shoe properties.

Shoe design feature ASICS Nimbus 18 ASICS Contend 4 Onitsuka Tiger Mexico 66
Heel counter Y – External pebax counter Y – Internal TPU counter No counter
Midsole Material Dual layer (SPEVA/Solyte) EVA EVA
Density Shore A 53 Shore A 54 Shore A 55
Heel height differential Male: 10 mm (23/13) Male: 10 mm (23/13) Unisex: 7 mm (14/7)
Female: 13 mm (25/12) Female: 10 mm (22/12)
Cushioning features Rearfoot Gel Rearfoot Gel No features
Forefoot Gel + 1st MTP twist Gel
Upper material Jacquard mesh Mesh Leather
Outsole material AHAR Carbon + Blown Rubber AHAR Carbon + Blown Rubber NC Rubber
Shoe Mass (g) Mens US Size 9 325 300 325
Female US Size 7 265 245 265

Bagging involved taking 10,000 bootstrapped samples from p-values). Bland-Altman plots were used to observe any bias,
the original dataset. Stepwise regression was then undertaken the 95% limits of agreement and heteroscedasticity (Bland &
using each sample. Those items not included in at least 5,000 of Altman, 1986). Intra and inter-day reliability was calculated
the 10,000 repetitions were dropped from further considera­ using intra-class correlation coefficients (ICC). Standard error
tion. The set of remaining items were then subjected to back­ of measurement (SEM) of each item was calculated using inter-
wards elimination using the Akaike Information Criterion (AIC). day reliability coefficients which then informed the minimum
Variables demonstrating multiple collinearity (Variable Inflation detectable difference (MDD90) (Portney & Watkins, 2009). To
Factor >5) were excluded from further analysis due to their allow interpretation of what is actually an important and mean­
relationship with other predictive variables (Kleinbaum et al., ingful change and/or difference in comfort based on the feed­
2013). Standardised regression coefficients from the final back of the runners themselves, minimal important change
model were used to derive weighting factors for each of the (MID) thresholds were established. To establish these thresh­
final items. These weighting factors were applied to each item olds, a participant-derived approach was used (Mills et al.,
score to create a new composite comfort score. 2010). Each participant was asked to nominate the change on
Discriminative ability of the RUN-CAT was assessed in three a 100 mm VAS (mm) that would represent a meaningful change
ways. Firstly, the total VAS comfort score was compared to the in comfort. The mean of these participant nominated values
RUN-CAT score for all condition using a paired-sample t-test. then became the participant-derived MID for each item.
Secondly, the differences in comfort score derived by both
techniques for each condition were assessed using one-way
analysis of variance (ANOVA), with Sidak corrections for multi­ Results
ple comparison. Thirdly, the data were re-classified based on Stage one: Online questionnaire
shoe preference (i.e., most comfortable, second most comfor­
table and least comfortable) rather than explicit shoe model, Two hundred and ninety-eight surveys were completed online.
and the differences in comfort again compared using one-way. Sixteen participants’ data were excluded as seven reported
The level of significance for statistical analyses was set at they did not run and nine were aged less than 18 years.
α = 0.05. All statistical analyses were conducted in STATA A final sample of 282 was used for the analyses (121 males,
Software (Version 14, StataCorp, USA). 161 females, Mean age 35.9 years [SD 11.4], height = 1.72 m
[SD 0.10], body mass 69.9 kg [SD 12.7], weekly running distance
24.9 km [SD 8.6], weekly running frequency = 3.1 days/week
Stage three [SD 1.2] and shoe size range euro 35–48). The results are pre­
Reliability testing, establishment of error thresholds and mini­ sented in Figure 1(e).
mally important difference values
An independent sample of thirty recreational runners Stage two: Development of the running shoe comfort
(15 females: 15 males, mean age 34.6 years SD = 8.5; height index (RUN-CAT)
1.75 m, SD 6.8; body mass 69.3 kg, SD 10.0, mean 1.1 km
running time = 4 min 28 seconds, SD = 42 s, shoe size One hundred runners completed testing and provided com­
range: euro 37–45) were recruited to test the refined tool. The fort data on all three shoes. No missing data were identified.
same recruitment strategies and eligibility criteria as used in Data were normally distributed (p > 0.05). Both raw and
Stage Two were applied. Participants ran in their own shoes normalised data are provided for each research variable in
that were not more than 4 months old and asked to test the Table 2. Running time was not different between conditions
comfort of their shoes using the RUN-CAT. To collect reliability (Shoe A = 283.9 ± 48.3 s, Shoe B = 287.4 ± 50.0 s, Shoe
data, participants were asked to run around a pre-defined C = 294.9 ± 54.4 s, p = 0.298).
1.1 km road track; twice during session one (separated by a five-
minute break) and a third time 1 week later (session two). Effect of different footwear on overall comfort
Paired t-tests were used to assess for systematic differences in Shoe A was deemed to be more comfortable than both Shoe
the mean values for within and between-day comfort scores of B (MD = 12.3 mm [95% CI = 6.3–18.3], p = <0.001, d = 0.73) and
the four items and overall comfort index (Bonferroni adjusted Shoe C (MD = 48.9 mm [95% CI = 42.9–54.9], p = <0.001,
 JOURNAL OF SPORTS SCIENCES 5

Table 2. Raw comfort assessment data for the three shoes tested on a horizontal 100 mm VAS. All values are reported in mm. Raw score refers to the raw data recorded
by the participant at the time of testing. The normalised score has been transformed during data processing. N/A applies to situation where the data were not
transformed for analysis.
Shoe A: Nimbus 18 Shoe B: Contend 4 Shoe C: Onitsuka Tiger
Raw score Normalised score Raw score Normalised score Raw score Normalised score
Footwear Item Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD P-value
Overall Shoe Comfort 74.9 15.4 N/A N/A 62.7 18.1 N/A N/A 26.0 19.0 N/A N/A <0.001
Heel cushioning 60.2 11.8 77.9 21.9 50.9 16.7 75.9 23.0 18.1 14.7 36.0 28.7 <0.001
Forefoot cushioning 58.4 12.3 78.8 20.7 46.9 17.2 74.9 24.3 20.2 15.1 39.5 28.4 <0.001
Shoe stability 54.0 9.5 86.9 16.0 55.0 11.8 81.4 17.6 27.8 20.7 48.4 32.0 <0.001
Heel fit 50.3 8.1 90.8 13.3 51.9 12.4 87.0 21.5 52.3 20.9 70.0 29.3 0.005
Upper flexibility 50.8 9.3 88.7 14.9 49.0 14.6 80.9 21.9 56.8 24.7 58.8 30.3 <0.001
Forefoot width 50.9 11.7 85.2 18.2 51.5 14.7 81.8 23.3 41.9 17.8 71.0 26.3 <0.001
Forefoot flexibility 53.9 9.4 85.5 14.3 45.7 13.7 79.4 19.9 57.6 23.8 59.6 29.2 <0.001
Toe box depth 48.0 11.5 87.4 19.7 49.3 11.1 88.3 18.9 43.8 17.0 74.9 26.0 <0.001
Shoe mass 45.5 13.0 83.1 21.6 46.5 13.8 82.7 22.6 65.1 20.0 60.3 30.4 <0.001
Shoe grip 52.6 7.2 91.8 13.0 51.8 11.1 87.5 18.6 37.4 18.9 65.9 30.0 <0.001

d = 2.83). Compared to Shoe C, Shoe B was more comfortable standardised regression coefficients (see Table 4) were used
(MD = 36.7 mm [95% CI = 30.7–42.7], p = <0.001, d = 1.98). to derive weightings for a composite score. Figure 2 presents
the data collection form used, which allowed raw items to be
Effect of different footwear on individual comfort items calculated, these scores then to normalised relative to their
A main effect of shoe type was noted on all footwear items total contribution to the RUN-CAT score (based on the regres­
(p < 0.05). Both Shoe A and B were shown to have more ideal sion coefficients), and finally, a total RUN-CAT score to be
heel cushioning (p = <0.001), forefoot cushioning (p = <0.001), calculated for each shoe. The final model explained 49% of
shoe stability (p = <0.001), heel fit (p = <0.001), upper flexibility the variability relating to shoe comfort.
(p = <0.001), forefoot width (p = <0.001), forefoot flexibility
(p = <0.001), toe box depth (p = <0.001), shoe mass Discriminative ability of the RUN-CAT
(p = <0.001) and shoe grip (p = <0.001) compared to Shoe Differences in comfort scores when categorising comfort data
C. No significant differences in any footwear item were identi­ by shoe model and individual preference using both assessment
fied between Shoe A and B (p > 0.05). Significant post-hoc techniques are included as Table 5. When analysing shoe com­
comparisons for the effect of footwear condition on each pre­ fort by shoe model, there were a main effect of condition
dictive variable are included as Table 3. (p < 0.05). Using the 100 mm VAS, Shoe A was more comfortable
than both Shoe B (MD = 12.3 mm [6.3–18.2], p = <0.001,
Contribution of predictive variables to overall footwear d = 0.73) and C (MD = 48.9 mm [43.0–54.9], p = <0.001,
comfort d = 2.83). When using the RUN-CAT, although there was a sig­
Of the 10 assessed items, five were included in the resampling nificant difference in comfort between Shoe A and
model at least 50% of occasions (>5000 iterations, see Table 4). C (MD = 36.6 mm [30.2–43.0], p = <0.001, d = 2.59), there was
Forefoot width was excluded from the model due to multiple no difference in comfort between shoe A and B (p = 0.237,
collinearity. Four items constituted the final RUN-CAT. The d = 0.30). However, when analysing by shoe preference,

Table 3. Post-hoc comparisons for the effect of footwear condition on each predictive variable.
Within-group comparison Effect size
Footwear Item Comparison MD (mm) 95% CI P-value Cohen’s d
Heel cushioning Shoe A vs. Shoe C 41.9 33.5–50.3 <0.001 1.64
Shoe B vs. Shoe C 40.0 31.5–48.4 <0.001 1.53
Forefoot cushioning Shoe A vs. Shoe C 39.2 30.8–47.6 <0.001 1.58
Shoe B vs. Shoe C 35.3 26.9–43.7 <0.001 1.34
Shoe stability Shoe A vs. Shoe C 38.5 30.7–46.3 <0.001 1.52
Shoe B vs. Shoe C 33.0 25.1–40.8 <0.001 1.28
Heel fit Shoe A vs. Shoe C 15.2 1.4–29.1 <0.001 0.91
Shoe B vs. Shoe C 17.0 3.2–30.8 <0.001 0.66
Upper flexibility Shoe A vs. Shoe C 29.8 21.9–37.7 <0.001 1.25
Shoe B vs. Shoe C 22.0 14.1–29.9 <0.001 0.84
Forefoot width Shoe A vs. Shoe C 14.3 6.5–22.0 <0.001 0.63
Shoe B vs. Shoe C 10.8 3.0–18.6 <0.001 0.43
Forefoot flexibility Shoe A vs. Shoe C 25.9 18.4–33.4 <0.001 1.13
Shoe B vs. Shoe C 19.8 12.3–27.3 <0.001 0.79
Toe box depth Shoe A vs. Shoe C 12.5 5.1–19.9 <0.001 0.54
Shoe B vs. Shoe C 13.3 5.9–20.7 <0.001 0.59
Shoe mass Shoe A vs. Shoe C 22.8 14.2–31.3 <0.001 0.86
Shoe B vs. Shoe C 22.4 13.8–30.9 <0.001 0.84
Shoe grip Shoe A vs. Shoe C 26.0 18.6–33.4 <0.001 1.12
Shoe B vs. Shoe C 21.6 14.2–29.0 <0.001 0.87
6 C. BISHOP ET AL.

Table 4. The standardised regression coefficient, reliability, SEM, MID and MDD90 for each footwear item in the final RUN-CAT model.
Included in final model Reliability
Item (No. of iterations) Correlation Intra-day [95% CI] Inter-Day [95% CI] SEM MID MDD90
Heel cushioning YES (N = 6785) 0.175 0.913 [0.817–0.958] 0.844 [0.673–0.926] 4.6 9.3 10.6
Forefoot cushioning YES (N = 9728) 0.311 0.934 [0.862–0.969] 0.897 [0.784–0.951] 4.3 9.8 10.0
Shoe stability YES (N = 9895) 0.277 0.944 [0.882–0.973] 0.859 [0.725–0.938] 4.4 9.9 10.3
Heel fit NO (N = 4493) N/A N/A N/A N/A N/A N/A
Upper flexibility NO (N = 1529) N/A N/A N/A N/A N/A N/A
Forefoot width YES* (N = 8486) N/A N/A N/A N/A N/A N/A
Forefoot flexibility YES (N = 9901) 0.247 0.846 [0.675–0.926] 0.702 [0.373–0.858] 4.9 9.7 11.4
Toebox depth NO (N = 1125) N/A N/A N/A N/A N/A N/A
Shoe mass NO (N = 3876) N/A N/A N/A N/A N/A N/A
Shoe grip NO (N = 1764) N/A N/A N/A N/A N/A N/A

Stage three: Reliability testing, establishment of error

thresholds, minimal detectable difference and
minimally important difference
All data were normally distributed. Differences in the mean
comfort scores between sessions were not statistically significant
(p = 0.160–1.000). Bland-Altman plots revealed minimal systema­
tic bias (range −2.3 mm to 0.6 mm) and proportional differences,
with the exception of the inter-day assessment of heel cushion­
ing which was affected by two participants with low ratings
(β 0.37, p = 0.007, Supplementary File 2). Data for each individual
footwear item are provided in Table 3. With regards to the new
composite comfort score (the comfort index), intra-day and inter-
day reliability were good to excellent (intra-day ICC = 0.889 [95%
CI = 0.753–0.948], inter-day ICC 0.705 [95% CI = 0.471–0.847]).
Compared to the 100 mm VAS, the RUN-CAT reported a lower
SEM for overall comfort (2.8 points compared to 4.9 mm with the
VAS) and a smaller MDD90 threshold (6.5 points compared to
11.5 mm with the VAS). The MID values were all <10 mm.

Discussion
Figure 2. The running footwear comfort assessment tool (RUN-CAT) items. The first aim of this study was to develop a new tool to assess
Individual footwear items are assessed on a 100 mm VAS using the anchors footwear comfort in research applications. A review of previous
provided. The RUN-CAT calculation section provides instructions for normalisation
of raw scores relative to overall contributions, culminating in a final RUN-CAT score. comfort tools indicated that none were developed in
a systematic way that incorporated assessment items that are
considered meaningful to runners. The present study
Table 5. Differences in comfort scores when categorising comfort data by shoe addressed previous limitations by both asking runners what
model and individual preference using both assessment techniques. they deemed important when assessing shoe comfort, and
100 mm then rigorously testing these items with different shoe designs
VAS
to identify which items were related to shoe comfort and how
Comfort RUN-CAT
score Score much of a change in comfort is actually important to the
Categorisation Condition Mean SD Mean SD p-Value runner. The removal of redundant footwear items in the final
By shoe model ASICS Nimbus 18 74.9 15.4 82.9 14.1 0.213 tool not only improved efficiency of data collection (which is
ASICS Contend 4 62.7 18.1 78.3 16.5 0.053 important when testing multiple shoes), but also ensured the
ASICS OT Mexico 66 26.0 19.0 46.4 14.1 <0.001 full contribution of each item in the final comfort score was
By shoe Most preferred 79.8 12.2 87.5 11.0 0.551
preference captured.
Second most 57.8 15.8 74.5 16.6 0.167 The second aim of this study was to determine the reliability
preferred of the RUN-CAT tool and its ability to discriminate between
Least preferred 26.0 19.0 46.4 14.1 <0.001
different footwear types. The RUN-CAT demonstrated similar
reliability to a horizontal 100 mm VAS (Mills et al., 2010;
Mündermann, Nigg, Stefanyshyn, Humble et al., 2002), and
significant differences in comfort were identified between the comparable MID thresholds to those previously reported
most preferred and second most preferred shoe using both the (Mills et al., 2010). Importantly, both assessment methods
RUN-CAT (MD = 12.6 mm [6.4–18.8], p = <0.001, d = 0.92) and could discriminate between the three footwear models during
100 mm VAS technique (MD = 18.8 mm [13.6–24.0], p = <0.001, field testing based on individual shoe preference. In light of the
d = 1.56). difference in shoe preferences identified, an interesting finding
 JOURNAL OF SPORTS SCIENCES 7

relates to the shoe-model-specific data, whereby the RUN-CAT Study limitations

did not identify a difference between Shoe A and B based on
The RUN-CAT was developed to provide an immediate
mean population data. This perhaps indicates that the RUN-
assessment of comfort, with particular application to choos­
CAT places less emphasis on traditional features thought to
ing the most comfortable shoe at the point of sale. We have
influence the global perception of comfort such as cushioning
not investigated how comfort measured by the RUN-CAT
and is in fact more sensitive to detecting differences in shoe
changes over time with accumulated shoe mileage. This is
design features and their subject-specific effect.
an application for future research. This is also the first report
of the development and evaluation of the RUN-CAT.
Although many elements of rigorous tool development
Benefits and applications of the RUN-CAT have been addressed, such as aspects of content validity,
reliability, measurement error and interpretability (Mokkink
The development of the weighted comfort index in this study
et al., 2010), further refinement is required. This includes
addresses an important consideration in the prescription of
testing in other cohorts and clinical groups, with subse­
running footwear. It is important to state at this point that the
quent assessment of its psychometric measurement proper­
RUN-CAT does not replace the 100 mm VAS technique, as both
ties and tool validation.
the 100 mm VAS and RUN-CAT are reliable and can discriminate
between footwear conditions. However, where a 100 mm VAS
may give a global perception of overall comfort, it cannot
explain what shoe design components contribute to that per­ Conclusion
ception. The benefit of the RUN-CAT, being a composite score,
This study presents the development and initial evaluation of
is that it evaluated individual items to determine where differ­
a new assessment tool (RUN-CAT) for evaluating the comfort of
ences in assessment score were. The ability to isolate such
running footwear. The RUN-CAT demonstrates excellent relia­
differences has multiple applications in terms of industry/
bility, acceptable measurement error and good discriminative
design and research.
ability between footwear models. Comfort perception is pre­
In terms of shoe design, the ability to isolate the contribu­
sented as a composite score for overall rating using weightings
tion of different items provides the ability to improve overall
derived from four individual items (heel cushioning, shoe sta­
comfort of running footwear through redesign of individual
bility, forefoot cushioning and forefoot flexibility). Based on the
shoe components. Although the concept is in its infancy, recent
data from this cohort, changes in running footwear comfort
research has suggested comfort could be an important over­
exceeding 10 mm were both larger than error thresholds and
riding feature in footwear selection for runners (Nigg et al.,
deemed to be important to runners. The RUN-CAT will benefit
2015). Cushioning has long been considered important in the
from further evaluation and validation in different cohorts to
design of running footwear, yet this discussion has largely
fully elucidate its clinical utility.
centred on the rearfoot based on the forces applied to the
foot at initial contact (Kong et al., 2009; Nigg et al., 1987; Nigg
et al., 1995; Shorten & Mientjes, 2011; Wang et al., 2010). The
current data suggest that optimising forefoot cushioning is the
Practical implications
most important contributor to overall shoe comfort. Given this ● Self-assessed running shoe comfort can be partially predicted by four
finding, the industry may benefit from research investigating explicit footwear items in order to optimise individual shoe choice
what makes ideal forefoot cushioning for the individual. ● Forefoot cushioning is more strongly related to comfort of running
shoes than heel cushioning.
From a footwear selection standpoint, the RUN-CAT may ● A difference in self-assessed comfort between shoes of more than 10 points
prove beneficial in the prescription/selection of the most on the RUN-CAT is deemed a meaningful change in comfort for the runner.
appropriate shoe. Shoe selection has historically been based
on foot posture and rearfoot video gait analysis findings,
with limited evidence to suggest this results in a more Acknowledgments
appropriate shoe. As individuals have their own comfort
preferences, what is comfortable for one may not be com­ CB is currently an ASICS industry funded post-doctoral research fel­
low. ASICS played no role in the design of the study, analysis of
fortable for another (Nigg et al., 2015). If shoes were
results or interpretation of findings. ASICS Oceania donated all foot­
selected based on comfort, the ability to isolate discrepan­ wear used in this study. JBA is currently supported by a National
cies between shoe models relating to one or more particu­ Health & Medical Research Council Early Career Research Fellowship
lar feature may lead to the overall perception of a more (ID: 1120560).
“comfortable” shoe for an individual. Although some shoes
may be rated as extremely comfortable but not appropriate
for running (i.e., too “soft”), the RUN-CAT incorporates scales Disclosure statement
that can define what an individual determines to be “ideal”. No potential conflict of interest was reported by the authors.
This approach can be used to further explore the complex
relationship between comfort, performance and injury.
Further research is also required to determine if shoe selec­ Funding
tion based on comfort results in better outcomes than other
approaches, such as gait analysis. This work was supported by the ASICS Oceania Pty Ltd.
8 C. BISHOP ET AL.

ORCID Mills, K., Blanch, P., & Vicenzino, B. (2010). Identifying clinically meaningful
tools for measuring comfort perception of footwear. Medicine and
Jonathan D. Buckley http://orcid.org/0000-0003-0298-2186 Science in Sports and Exercise, 42(10), 1966–1971. https://doi.org/10.
Adrian E. Esterman http://orcid.org/0000-0001-7324-9171 1249/MSS.0b013e3181dbacc8
John B. Arnold http://orcid.org/0000-0002-1158-8917 Mokkink, L. B., Terwee, C. B., Knol, D. L., Stratford, P. W., Alonso, J.,
Patrick, D. L., Bouter, L. M., & De Vet, H. C. (2010). The COSMIN
checklist for evaluating the methodological quality of studies on
References measurement properties: A clarification of its content. BMC Medical
Research Methodology, 10(1), 22. https://doi.org/10.1186/1471-2288-
Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing 10-22
agreement between two methods of clinical measurement. The Mündermann, A., Nigg, B. M/, Stefanyshyn, D. J., & Humble, R. N. (2002).
Lancet, 1(8476), 307–310. https://doi.org/10.1016/S0140-6736(86) Development of a reliable method to assess footwear comfort during
90837-8 running. Gait & Posture, 16(1), 38–45. https://doi.org/10.1016/S0966-
Dinato, R. C., Ribeiro, A. P., Butugan, M. K., Pereira, I. L., Onodera, A. N., & 6362(01)00197-7
Sacco, I. C. (2015). Biomechanical variables and perception of comfort in Mundermann, A., Stefanyshyn, D. J., & Nigg, B. M. (2001). Relationship
running shoes with different cushioning technologies. Journal of Science between footwear comfort of shoe inserts and anthropometric and
and Medicine in Sport, 18(1), 93–97. https://doi.org/10.1016/j.jsams.2013. sensory factors. Medicine and Science in Sports and Exercise, 33(11),
12.003 1939–1945. https://doi.org/10.1097/00005768-200111000-00021
Hagen, M., Hömme, A.-K., Umlauf, T., & Hennig, E. M. (2010). Effects of Nigg, B., Bahlsen, H., Luethi, S., & Stokes, S. (1987). The influence of running
different shoe-lacing patterns on dorsal pressure distribution during velocity and midsole hardness on external impact forces in heel-toe
running and perceived comfort. Research in Sports Medicine, 18(3), running. Journal of Biomechanics, 20(10), 951–959. https://doi.org/10.
176–187. https://doi.org/10.1080/15438627.2010.490180 1016/0021-9290(87)90324-1
Hirschmüller, A., Baur, H., Müller, S., Helwig, P., Dickhuth, -H.-H., & Nigg, B., Baltich, J., Hoerzer, S., & Enders, H. (2015). Running shoes and
Mayer, F. (2011). Clinical effectiveness of customised sport shoe running injuries: Mythbusting and a proposal for two new paradigm­
orthoses for overuse injuries in runners: A randomised controlled s:‘preferred movement path’ and ‘comfort filter’. British Journal of Sports
study. British Journal of Sports Medicine, 45(12), 959–965. http://dx. Medicine, 49(20), 1290–1294. http://dx.doi.org/10.1136/bjsports-2015-
doi.org/10.1136/bjsm.2008.055830 095054
Hoerzer, S., Trudeau, M. B., Edwards, W. B., & Nigg, B. M. (2016). Intra- Nigg, B. M., Cole, G. K., & Brüggemann, G.-P. (1995). Impact forces during
rater reliability of footwear-related comfort assessments. Footwear heel-toe running. Journal of Applied Biomechanics, 11(4), 407–432.
Science, 8(3), 155–163. https://doi.org/10.1080/19424280.2016. https://doi.org/10.1123/jab.11.4.407
1195451 Portney, L. G., & Watkins, M. P. (2009). Foundations of clinical research:
Kleinbaum, D., Kupper, L., Nizam, A., & Rosenberg, E. (2013). Applied regres­ Applications to practice. Pearson/Prentice Hall.
sion analysis and other multivariable methods. Nelson Education. Schubert, C., Oriwol, D., & Sterzing, T. (2011). Gender and age related
Kong, P. W., Candelaria, N. G., & Smith, D. R. (2009). Running in new and requirements of running shoes: A questionnaire on 4501 runners.
worn shoes: A comparison of three types of cushioning footwear. British Footwear Science, 3(sup1), S148–S150. https://doi.org/10.1080/
Journal of Sports Medicine, 43(10), 745–749. https://doi.org/10.1136/ 19424280.2011.575850
bjsm.2008.047761 Shorten, M., & Mientjes, M. I. (2011). The ‘heel impact’force peak during
Luo, G., Stergiou, P., Worobets, J., Nigg, B., & Stefanyshyn, D. (2009). running is neither ‘heel’nor ‘impact’and does not quantify shoe cushion­
Improved footwear comfort reduces oxygen consumption during ing effects. Footwear Science, 3(1), 41–58. https://doi.org/10.1080/
running. Footwear Science, 1(1), 25–29. https://doi.org/10.1080/ 19424280.2010.542186
19424280902993001 Tay, C. S., Sterzing, T., Lim, C. Y., Ding, R., & Kong, P. W. (2016). Overall
Lynn, M. R. (1986). Determination and quantification of content validity. preference of running shoes can be predicted by suitable perception
Nursing Research, 35(6), 382–386. https://doi.org/10.1097/00006199- factors using a multiple regression model. Human Factors, 59(3), pp.432–
198611000-00017 441. https://doi.org/10.1177/0018720816681147
Mailler-Savage, E. A., & Adams, B. B. (2006). Skin manifestations of running. Wang, L., Xian Li, J., Hong, Y., & He Zhou, J. (2010). Changes in heel
Journal of the American Academy of Dermatology, 55(2), 290–301. https:// cushioning characteristics of running shoes with running mileage.
doi.org/10.1016/j.jaad.2006.02.011 Footwear Science, 2(3), 141–147. https://doi.org/10.1080/19424280.
Miller, J. E., Nigg, B. M., Liu, W., Stefanyshyn, D. J., & Nurse, M. A. (2000). 2010.519348
Influence of foot, leg and shoe characteristics on subjective comfort. Zhang, L., Helander, M. G., & Drury, C. G. (1996). Identifying factors of
Foot and Ankle International, 21(9), 759–767. https://doi.org/10.1177/ comfort and discomfort in sitting. Human Factors, 38(3), 377–389.
107110070002100908 https://doi.org/10.1518/001872096778701962