Statistical Methods in

ISO/IEC 17043 and ISO 13528

Requirements and Guidelines

1
 When is Statistics Relevant
• Planning and Design
• Homogeneity testing
• Stability testing
• Determining Assigned Values
• Determining SD for Proficiency Assessment
• Performance assessment
• Summary statistics

2
 Overview
• Requirements for statistical methods from
ISO/IEC 17043
• Overview of statistical procedures in the major
standards
• Determining the assigned value
• Determining the performance score
• Checking homogeneity and stability
• Graphical methods
• Examples from APLAC and IMEP PT programmes
3
 Documents for PT Statistics
• ISO/IEC 17043: 2010
Conformity Assessment – General
requirements for Proficiency Testing (replacing
Guide 43-1 and 2:1997)
• ISO 13528: 2005
Statistical Methods for use in Proficiency
Testing by Interlaboratory Comparisons

4
 ISO/IEC 17043
• assigned value (ISO/IEC 17043, clause 3.1)
value attributed to a particular property of a
proficiency test item

• assigned value (ISO 13528, clause 3.3)

value attributed to a particular quantity and
accepted, sometimes by convention, as having an
uncertainty appropriate for a given purpose

5
 ISO/IEC 17043
– 3.5 outlier
observation in a set of data that appears to be inconsistent with the
remainder of that set

NOTE An outlier can originate from a different population or be the

result of an incorrect or other gross error

Assumption:
– when a lab’s result is classified as an outlier, its performance is
regarded as unsatisfactory
• But it is not appropriate to call all unsatisfactory results ‘outliers’
– It is a prima facie evidence and the lab needs to investigate its
operation
– There may be other valid reasons not relating to performance causing
its result classified as outlier

6
 ISO 13528
• Written by ISO TC69, SC6
• Approved work item in 1997
• Published in 2005
• Reaffirmed in 2009
• Now under revision
• Still in CD
• CD1 passed ballot, but will re-ballot as CD

7
 Other ISO Documents for PT Statistics
• ISO /TR 13425: 2006 Guidelines for the
selection of statistical methods in
standardization and specification

• ISO 16269-4: 2011

Statistical interpretation of data: Part 4:
Detection and treatment of outliers

8
 ISO 13425
• A descriptive catalogue of the available
TC69 International Standards and Guides
• Assist in selecting most suitable for a
particular purpose
• Gives non-technical abstracts and
technical abstracts

9
 ISO 16269-4
• Provides detailed descriptions of sound
statistical testing procedures and
graphical data analysis methods for
detecting outliers in data
• Primarily designed for the detection and
accommodation of outlier(s)
• Identify and standardize a sound subset
of methods used in the identification and
treatment of outliers
11
 Other Documents for PT Statistics
• The International Harmonized Protocol for
Proficiency Testing of Analytical Chemistry
Laboratories (IUPAC Technical Report) 2006
(http://www.iupac.org/publications/pac/2006/pdf/7801x0145.pdf)

• IUPAC/CITAC Guide: Selection and use of

proficiency testing schemes for a limited number
of participants – chemical analytical laboratories
(IUPAC Technical Report) 2010
(http://iupac.org/publications/pac/pdf/2010/pdf/8205x1099.pdf)

12
 Other Documents for PT Statistics
• Eurachem Guide: Selection, Use and
Interpretation of proficiency testing (2011)
(
http://www.eurachem.org/index.php/publications/guid
es/usingpt
)

13
 ISO/IEC 17043: 2010
Conformity assessment – General
requirements for proficiency testing

14
 ISO/IEC 17043
Requirements relating to statistics

• 4.2 Personnel
– 4.2.4 The PTP shall authorize specific personnel to:
….
h) conduct statistical analysis
….

15
 ISO/IEC 17043
Requirements relating to statistics

 4.4 Design of proficiency testing scheme

 4.4.1 Planning
4.4.1.3 The PTP shall document ...the following
information..
– p) detailed description of the statistical analysis to be used;
– q) the origin, metrological traceability and measurement uncertainty
of any assigned values;
– r) criteria for the evaluation of performance of participants;
– s) a description of the data, interim reports or information to be
returned to participants;
• 4.4.1.4 The PTP shall be access to the necessary technical expertise and
experience in …. statistics
16
 ISO/IEC 17043
• 4.4.3 Homogeneity and stability

4.4.3.2 The procedure for the assessment of

homogeneity and stability shall be documented and
conducted, where applicable, in accordance with
appropriate statistical designs. Where possible, the
PTP shall use a statistically random selection of a
representative number of proficiency testing items
from the whole batch of test material in order to
assess the homogeneity of the material.

17
 ISO/IEC 17043
• 4.4.4 Statistical design
• 4.4.4.1 Statistical designs shall be developed to meet
the objectives of the scheme, based on the nature of
the data (quantitative or qualitative, including ordinal
and categorical), statistical assumptions, the nature
of errors, and the expected number of results

18
 ISO/IEC 17043
• 4.4.4 Statistical design (cont’d)

NOTE 1 Statistical design covers the process of planning,

collection, analysis and reporting of the proficiency testing
scheme data. Statistical designs are often based on objectives
for the proficiency testing scheme, such as detection of certain
types of errors with specified power or determination of
assigned values with specified measurement uncertainty

NOTE 2 Data analysis methods could vary from the very simple
(e.g. descriptive statistics) to complex, using statistical models
with probabilistic assumptions or combinations of results for
difference proficiency test items

19
 ISO/IEC 17043
• 4.4.4 Statistical design (cont’d)

NOTE 3 In cases where the proficiency testing scheme design

is mandated by a specification given by, for example, a
customer, regulatory authority or accreditation body, the
statistical design and data analysis methods can be taken
directly from the specification

NOTE 4 In the absence of reliable information needed to

produce a statistical design, a preliminary interlaboratory
comparison can be used

20
 ISO/IEC 17043
• 4.4.4 Statistical design (cont’d)

• 4.4.4.2 The PTP shall document the statistical design and data
analysis methods to be used to identify the assigned value
and evaluate participant results, and shall provide a
description of the reasons for their selection and assumptions
upon which they are based. The PTP shall be able to
demonstrate that statistical assumptions are reasonable and
that statistical analyses are carried out in accordance with
prescribed procedures

21
 ISO/IEC 17043
• 4.4.4 Statistical design (cont’d)

• 4.4.4.3 In designing a statistical analysis, the PTP shall give

careful consideration to the following:

a) The accuracy (trueness and precision) as well as the measurement

uncertainty required or expected for each measurand or
characteristic in the proficiency testing;
b) The minimum number of participants in the proficiency testing
scheme needed to meet the objectives of the statistical design; in
cases where there is an insufficient number of participants to meet
these objectives or to produce statistically meaningful analysis of
results, the PTP shall document, and provide to participants,
details of the alternative approaches used to assess participant
performance;

22
 ISO/IEC 17043
• 4.4.4 Statistical design (cont’d)

c) The relevance of significant figures to the reported results,

including the number of decimal places;
d) The number of proficiency test items to be tested or measured and
the number of repeat tests, calibrations or measurements to be
conducted on each proficiency test item or for each determination;
e) The procedures used to establish the standard deviation for
proficiency assessment or other evaluation criteria;
f) Procedures to be used to identify or handle outliers, or both;
g) Where relevant, the procedures for the evaluation of values
excluded from statistical analysis; and
h) Where appropriate, the objectives to be met for the design and
the frequency of proficiency testing rounds.

23
 ISO/IEC 17043
• 4.4.5 Assigned values

4.4.5.1 The proficiency testing provider shall document the

procedure for determining the assigned values for the
measurands or characteristics in a particular proficiency testing
scheme. This procedure shall take into account the metrological
traceability and measurement uncertainty required to
demonstrate that the proficiency testing scheme is fit for its
purpose.

4.4.5.4 When a consensus value is used as the assigned value,

the PTP shall document the reason for that selection and shall
estimate the uncertainty of the assigned value as described in
the plan for the proficiency testing scheme

24
 ISO/IEC 17043
• 4.7 Data analysis and evaluation of proficiency
testing scheme results

4.7.1 Data analysis and records

4.7.1.2 Results received from participants shall be recorded and

analysed by appropriate methods. Procedures shall be established
and implemented to check the validity of data entry, data transfer,
statistical analysis, and reporting.

4.7.1.3 Data analysis shall generate summary statistics and

performance statistics, and associated information consistent with
the statistical design of the proficiency testing scheme.

25
 ISO/IEC 17043
• 4.7 Data analysis and evaluation of proficiency
testing scheme results (cont’d)

4.7.1 Data analysis and records

4.7.1.4 The influence of outliers on summary statistics shall be

minimize by the use of robust statistical methods or appropriate
tests to detect statistical outliers.

4.7.1.5 The PTP shall have documented criteria and procedures for
dealing with test results that may be inappropriate for statistical
evaluation, e.g. miscalculations, transpositions and other gross
error.

26
 ISO/IEC 17043
• 4.7 Data analysis and evaluation of proficiency
testing scheme results (cont’d)

4.7.2 Evaluation of performance

4.7.2.1 The PTP shall use valid methods of evaluation which meet
the purpose of the proficiency testing scheme. The methods shall
be documented and include a description of the basis for the
evaluation….

27
 ISO/IEC 17043
• 4.8 Reports

4.8.2 Report shall include the following, unless it is not

applicable or the PTP has valid reasons for not doing so:
….
k) statistical data and summaries, including assigned values
and range of acceptable results and graphical displays:
….
n) procedures used to establish the standard deviation for
proficiency assessment, or other criteria for evaluation;

28
 ISO/IEC 17043
• 4.8 Reports (cont’d)

….
o) assigned values and summary statistics for test
methods/procedures used by each group of participants (if
different methods are used by different groups of participants);
….
r) procedures used to statistically analyse the data;
….

29
 ISO/IEC 17043 Annex B (informative) Statistical methods for
proficiency testing

 The statistical methods used to analyse the results

need to be appropriate for each situation, and so are
too varied to be specified in this International
Standard.
 ISO 13528 describes preferred specific methods for
each of the situations discussed below, but also
states that other methods may be used as long as
they are statistically valid and are fully described to
participants.

30
 ISO/IEC 17043 Annex B (informative) Statistical
methods for proficiency testing
 Some of the methods in ISO 13528, especially for
homogeneity and stability testing, are modified slightly
in the IUPAC Technical Report “The International
Harmonized Protocol for the proficiency testing of
analytical chemistry laboratories”
 These documents also present guidance on design and
visual data analysis.
 Other references may be consulted for specific types
of proficiency testing schemes, e.g. measurement
comparison schemes for calibration

31
 ISO/IEC 17043 Annex B (informative) Statistical
methods for proficiency testing
 Fundamental steps common to nearly all proficiency
testing schemes:
 Determination of the assigned value
 Calculation of performance statistics
 Evaluation of performance
 Preliminary determination of proficiency test item
homogeneity and stability

32
 ISO/IEC 17043 Annex B (informative) Statistical
methods for proficiency testing
Determination of the assigned value and its uncertainty
Procedures available:
• Known values – formulation (e.g. manufacture or dilution)
• Certified reference values – by definitive methods
• Reference values - determined by comparison alongside a
reference material or standard traceable to a national or
international standard
• Consensus value from expert participants (e.g. reference
labs)
• Consensus values from participants

• All these are for quantitative data 33

 ISO/IEC 17043 Annex B (informative) Statistical
methods for proficiency testing
Determination of the assigned value and its uncertainty (cont’d)
• Other considerations:
– If consensus, control outliers
– If consensus, check trueness of process
– Criteria for acceptability on the basis of
uncertainty of the assigned value (for all a.v.,
especially consensus)

34
 ISO/IEC 17043 Annex B (informative) Statistical
methods for proficiency testing
Determination of the assigned value and its uncertainty (cont’d)

 Outliers are statistically treated as described below.

 Obvious blunders, such as those with incorrect unit,
decimal errors, and results for a different proficiency test
item should be removed from the data set and treated
separately. These results should not be subject to outlier
tests or robust statistical methods

35
 ISO/IEC 17043 Annex B (informative) Statistical
methods for proficiency testing
B2 Determination of the assigned value and its uncertainty
(cont’d)

 When participants’ results are used to determine assigned

values, statistical methods should be in place to minimize
the influence of outliers. This can be accomplished with
robust statistical methods or by removing outliers prior to
calculation. In larger or routine proficiency testing
schemes, it may be possible to have automated outlier
screens, if justified by objective evidence of effectiveness

36
 ISO/IEC 17043 Annex B (informative) Statistical
methods for proficiency testing
B2 Determination of the assigned value and its uncertainty
(cont’d)
 If results are removed as outliers, they should be removed
only for calculation of summary statistics. These results
should still be evaluated within the proficiency testing
scheme and be given the appropriate performance
evaluation

NOTE ISO 13528 describes a specific robust method for

determination of the consensus mean and standard
deviation, with the need for outlier removal.

37
 ISO/IEC 17043 Annex B (informative) Statistical
methods for proficiency testing
Determination of the assigned value for qualitative data

 Statistical methods for determining the assigned values for

qualitative data or semi-qualitative values are not discussed in
ISO 13528

 These assigned values need to be determined by expert

judgment or manufacture

38
 ISO/IEC 17043 Annex B (informative) Statistical
methods for proficiency testing
Determination of the assigned value for qualitative data

 Consensus value, as defined by agreement of a

predetermined majority percentage of responses (e.g. 80% or
more)

 Percentage used should be determined based on objectives

for the PT scheme and the level of competence and
experience of the participants

 May use median or mode for ordinal data, not mean

39
 ISO/IEC 17043 Annex B (informative) Statistical
methods for proficiency

• No such thing as standard deviation for ordinal data

• IT IS NOT APPROPRIATE to calculate the mean or SD

of semi-quantitative values.

40
ISO 13528 Statistical methods for use in
proficiency testing by interlaboratory
comparisons

41
 ISO 13528
• Complementary to ISO/IEC Guide 43 providing
detailed guidance that is lacking
• Written as a Standard
• Revision will include changes in Harmonised
Protocol for PT of analytical labs (2006), but is
intended for use with all measurement
methods

42
 ISO 13528
• High interest / some parts are widely used
• Goal is to describe optimal procedures, but to
allow other procedures as long as they are:
– Statistically valid
– Fully described to participants

43
 ISO 13528
• Provides detailed descriptions of sound
statistical methods for organisers to analyse
data obtained from PT schemes
• Can be applied to demonstrate that the
measurement results obtained by labs do not
exhibit evidence of an unacceptable level of bias
• Applicable to quantitative data but not
qualitative data.
– Revision will include procedures for qualitative
data
44
 Statistical guidelines for the design and
interpretation of proficiency test
• Possible conflicts with requirement for
laboratories to treat and report PT same as
for customer

• NO TRUNCATED RESULTS
– “Less than” values not allowed
– Possible resolution:
Restriction only applies to consensus

45
 Reporting considerations: ISO 13528,
section 4.6
• Rounding:
• Independently estimate typical repeatability sr
• Do not round digits by more than sr/2
• Number of replicates
• Concern for getting accurate estimate of bias
• When a method’s repeatability is large, it can
confuse interpretation of scores
• Determine number n of replicates so that:
sr /√n < 0.3sP
46
 Reporting considerations: ISO 13528,
section 4.6 and 5.8
• Missing replicate data
• When the design calls for n=2 or more replicates and with
the mean of the replicates used for grading
• When some participants do not submit all replicates
• Require at least 0.59n replicates
• n=2 replicates, require both
• n=3 replicates, require at least 2
• n=4 replicates, require at least 3
• n=5 replicates, require at least……?

47
 Limiting the effect of repeatability

Example
Say sP=5% and sr = 2%:
Then sr /√n < 0.3sP

So 2%/√n < 0.3(5%) or

2/1.5 < √n  (1.77)2 < n  3.13 < n

Or n=4 replicates

This criterion can lead to large n replicates.

48
Homogeneity and Stability
Demonstration of homogeneity and stability in ISO/IEC 17043

• Ensure sufficient homogeneity so as to not

impact evaluation of performance
• Different needs for determining H&S in PT and
in for Reference Materials (ISO Guides 34 and
35)
– PT (and RM) needs to ensure sufficient
– CRM needs to estimate SD between samples, and
instability as part of uncertainty of assigned value

50
 Homogeneity – ISO 13528
• Homogeneity
– Precision of method: (san / sP) < 0.5
– 10 or more samples, 2 replicates
– SDS for samples (ANOVA or direct calculation)
– SDS < 0.3 sP
– No F test
• Can use experience to reduce testing
When evidence and theory prove homogeneous

51
 Homogeneity – IUPAC (2006)
• Similar to ISO 13528, larger criterion for
acceptance, more complex statistics.
• 10 or more samples, in duplicate
• Sufficient repeatability: σan < 0.5σp
• Cochran test for duplicates
• Visual check for anomalies
– Non-random differences between replicates
– Time trend across manufacture

52
 Homogeneity – IUPAC (2006)
• Calculate variances
– S2an (between replicates)
– S2sam (between samples)
– σ2all = (0.3σp)2

• Calculate acceptance criterion

– Take F1 and F2 from Tables
– c = F1σ 2all +F2s2an
– If S2sam < c then acceptable homogeneity

• Since F1>0 and s2an>0 and σ2all = 13528 criterion, this is always
an easier criterion 53
 Homogeneity - traditional
• F test (allowed, not recommended)
F = (SDS2/sr2)
Sr = repeatability SDS= between samples

Fcrit = F(.05,k-1, s(n-1)) k=# samples n=# replicates

• High Sr insensitive test (large SDS passes)

• Low Sr too sensitive test (small SDS fails)

54
 Stability – ISO 13528
• Stability
– Analysis on or after closing date
– (2-)3 samples, (1-)2 replicates, depending on
experience
– Calculate overall mean
– [Mean(H) – Mean(S)] < 0.3 sP
– No statistical t test
• High Sr insensitive test (big difference passes)
• Low Sr too sensitive test (small difference fails)
55
 Stability - practical
• Can use experience and technical knowledge
(backed by data)
– Same measurand, same manufacture process,
same matrix
– For calibration artefacts, homogeneity and
stability are usually the same thing

56