WHO Protocol for Performance Evaluation of HCV serology assays

WHO PQDX_040 v6 February 2017

WHO PROTOCOL FOR PERFORMANCE

LABORATORY EVALUATION OF HCV
SEROLOGY ASSAYS

WHO/EMP/RHT/PQT/2017.xx Page 1 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017

© World Health Organization 2017

Some rights reserved. This work is available under the Creative Commons Attribution Non-Commercial-
Share Alike 3.0 IGO license (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo).

Under the terms of this license, you may copy, redistribute and adapt the work for noncommercial
purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there
should be no suggestion that WHO endorses any specific organization, products or services. The use of the
WHO logo is not permitted. If you adapt the work, then you must license your work under the same or
equivalent Creative Commons license. If you create a translation of this work, you should add the following
disclaimer along with the suggested citation: “This translation was not created by the World Health
Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original
English edition shall be the binding and authentic edition”.
Any mediation relating to disputes arising under the license shall be conducted in accordance with the
mediation rules of the World Intellectual Property Organization
(http://www.wipo.int/amc/en/mediation/rules).

Cataloguing-in-Publication (CIP) data: CIP data are available at http://apps.who.int/iris. Sales, rights and
licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit requests for
commercial use and queries on rights and licensing, see http://www.who.int/about/licensing.

Third-party materials: If you wish to reuse material from this work that is attributed to a third party, such
as tables, figures or images, it is your responsibility to determine whether permission is needed for that
reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of
any third-party-owned component in the work rests solely with the user.

General disclaimers: The designations employed and the presentation of the material in this publication do
not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any
country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or
boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not
yet be full agreement.

The mention of specific companies or of certain manufacturers’ products does not imply that they are
endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned.
Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters.
All reasonable precautions have been taken by WHO to verify the information contained in this publication.
However, the published material is being distributed without warranty of any kind, either expressed or
implied. The responsibility for the interpretation and use of the material lies with the reader. In no event
shall WHO be liable for damages arising from its use.

WHO and the WHO Prequalification Evaluating Laboratory do not warrant or represent that the evaluations
conducted with the HCV test kits referred to in this document are accurate, complete and/or error-free.
WHO and the WHO Prequalification Evaluating Laboratory disclaim all responsibility for any use made of
the data contained herein, and shall not be liable for any damages incurred as a result of its use. This
document must not be used in conjunction with commercial or promotional purposes.

WHO/EMP/RHT/PQT/2017.xx Page 2 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017
3. Introduction

3.1. Prequalification of in vitro diagnostics

The World Health Organization (WHO) Prequalification of In Vitro Diagnostics is coordinated through the
Prequalification-Diagnostics Assessment team (PQDx). The aim of the WHO Prequalification of In Vitro
Diagnostics Assessment is to promote and facilitate access to safe, appropriate and affordable diagnostics
of good quality in an equitable manner. Focus is placed on products for high burden diseases and their
suitability for use in resource-limited settings.

The WHO prequalification of in vitro diagnostics process includes three main components:

 Review of an pre-submission form and product dossier;

 Performance evaluation of the product;
 Inspection of the manufacturing site(s).

Performance evaluation will be conducted by two different mechanisms as described at:

http://www.who.int/diagnostics_laboratory/evaluations/alternative/en/. Performance evaluation in List 1
laboratories will be coordinated and paid for by WHO. Performance evaluation in List 2 laboratories will be
coordinated and paid for by the manufacturer.

This protocol provides the details of the procedure for evaluation of HCV serology simple and rapid assays
and Enzyme Immunoassays submitted for laboratory evaluation as part of the WHO Prequalification of In-
vitro Diagnostic assays on behalf of World Health Organization at the WHO Prequalification Evaluating
Laboratory.

This protocol shall not replace more extensive IVD manufacturer’s protocols required for validation and
verification studies of their products.

3.2. WHO laboratory evaluation of diagnostics for HCV antibody and/or detection

The performance evaluation determines the accuracy of HCV assays in comparison with established
performance criteria. These characteristics include: sensitivity, specificity, negative and positive predictive
values as well as the accuracy. In addition, a number of operational characteristics are assessed including
the suitability for use in laboratories and/or testing settings with limited infrastructure. Assays for the
detection of HCV antibody and assays for the detection of both HCV antibody and HCV antigen are covered
in this protocol.

All HCV serology assays submitted for performance evaluation are assessed at a WHO Prequalification
Evaluating Laboratory which has been assessed and listed using the WHO Alternative Performance
evaluation Mechanism upon the instruction of WHO/PQDx (List 1 laboratories) or a manufacturer (List 2
Laboratories) who has submitted an IVD for WHO assessment

The laboratory evaluation determines the accuracy of assays in comparison with established performance
criteria. These characteristics include: sensitivity, specificity, negative and positive predictive values as well
as the accuracy. In addition, a number of operational characteristics are assessed including the suitability
for use in laboratories and/or testing settings with limited infrastructure.

The HCV specimen reference panel shall comprise a minimum of 463 anti-HCV positive and anti-HCV
negative serum/plasma specimens from Europe, Africa, Asia and Latin America plus several well-
characterized commercial seroconversion panels and performance panels, a lot-to-lot variation panel, WHO
international biological reference preparations.

WHO/EMP/RHT/PQT/2017.xx Page 3 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017
4. Objectives

4.1. Overall objectives

The overall objectives are:
1. To evaluate currently commercially available HCV assays (including EIA and rapid diagnostic tests)
for detection of HCV antibodies and/or HCV antigen against established performance criteria.

4.2. Specific objectives

The specific objectives of the evaluation are:
1. To determine the sensitivity, specificity of currently available HCV assays (EIA's and rapid diagnostic
tests and other formats) for the detection of HCV antibodies and/or HCV antigen as compared to a
reference result (two EIA's and one line immunoassay)

2. To evaluate the operational characteristics of HCV assays, e.g. ease of performance, utility of
specimen type, inter-reader variability, reaction endpoint stability, rate of invalid runs/devices,
suitability for use in extreme climates (high/low temperatures, high humidity), and suitability for
use in countries with limited infrastructure (no/limited electricity, no/limited clean water, and
inadequate means of biosafety disposal).

5. Study Design

5.1. WHO Prequalification Evaluating Laboratory

The WHO Prequalification Evaluating Laboratory shall be one that has undergone assessment using the
WHO Alternative Laboratory Evaluation Mechanism which include submission of an EoI, Stage 1 audit
(Assessment of EoI and specific quality management system (QMS) documentation), Stage 2 audit which
include On-site audit of the laboratory to assess compliance with WHO requirements and lastly listed as
WHO Prequalification Evaluating Laboratories.
http://www.who.int/diagnostics_laboratory/evaluations/alternative/en/.

The laboratory shall hold the following certification for quality management within the laboratory:
ISO17025 (General requirements for the competence of testing and calibration laboratories), ISO15189
(Medical laboratories: Particular requirements for quality and competence) or equivalent.

The person(s) listed in the EoI letter to WHO will act as the Principal Investigator (PI) for the work
performed by the WHO Prequalification Evaluating Laboratory.

5.2. Training, performance evaluation and supervision

The following issues are key to minimizing error and maximizing the value of this performance evaluation:
 The PI will be responsible for training the laboratory technicians on the details of the evaluation
protocol and on the performance of each assay undergoing evaluation;
 Only those personnel who have received specific training for this evaluation will be employed in the
evaluation;
 Accurate record keeping is crucial to the success of the evaluation and the PI will be responsible for
ensuring that all data required for the evaluation are recorded on the agreed data collection sheets,
and are accurate and up to date;
 It is important to plan work in advance and follow standard operating procedures as prepared and
controlled by the WHO Prequalification Evaluating Laboratory;
 To reduce the risk of adding an incorrect specimen to a test device/well, before starting the test
run, the operator will prepare worksheets and label all tubes, dilution vessels, test devices or plates
with the specimen’s unique number;
 Because objective, machine-generated, permanent results for simple/rapid diagnostic tests are not
feasible, it is essential that the PI emphasizes to the operator performing the tests the need for
accurate recording of results and recordkeeping;

WHO/EMP/RHT/PQT/2017.xx Page 4 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017
 To minimize the risk of error, it is recommended that the results are read and recorded
independently by three trained staff members;
 To allow immediate correction of erroneous recording of results (rather than differences in visual
interpretation), the PI or designee should assess the results as soon as possible to allow her/him to
return to the original test device to investigate apparently discordant readings;
 For the performance evaluations performed at the WHO Prequalification Evaluating Laboratory, at
least one representative result from both HCV positive and negative specimens will also be
recorded by taking electronic images. Unexpected test results will also be digitally recorded as well
an image of the instructions for use.

5.3. Safety
HIV, hepatitis B and hepatitis C and other viruses are transmissible by blood and body fluids. Therefore, all
types of specimens (including venous and capillary whole blood, serum/plasma, oral fluid, etc.) must be
handled as potentially infectious. Appropriate precautions to minimize infectious hazards must be taken at
all stages from the collection of specimens to the disposal of used materials from the laboratory. The WHO
Guidelines on HIV Safety Precautions, and Guidelines for the Safe Transport of Specimens (WHO/EMC/97.3)
and the HPE guidelines on laboratory safety should be carefully followed by the laboratory staff.

5.4. Storage of assays

All reagents must be stored as indicated in the instructions for use. Some assays may not need
refrigeration. Calibrated thermometers are placed at each location where reagents and specimens are
stored, i.e. ambient, refrigerator and freezer. Temperatures are recorded daily on the WHO
Prequalification Evaluating Laboratory forms. The lot numbers of the test kits received/used and their
expiry dates are recoded on the individual run worksheets.

Two separate production lots (with different lot numbers and different expiry dates) will be requested for
evaluation, according to the following definition1 of a lot: “The amount of material that is uniform in its
properties and has been produced in one process or series of processes. The material can be either starting
material, intermediate material or finished product.” Furthermore, lots must be sourced from a
representative production run and not produced especially for the purpose of this evaluation. WHO will
verify this information before the product assessment has been finalized.

6. Specimens

6.1. HCV specimen reference panel

6.1.1.Collection of specimens for the HCV specimen reference panel

Specimens are collected as serum or plasma (with EDTA as an anti-coagulant). Panel specimens are
assigned a unique identification number at the collection site and sent as large volumes to the WHO
Prequalification Evaluating Laboratory where they are assigned the specimen identification number. Once
the specimens have been processed and labelled they are frozen immediately at -20°C. At the WHO
Prequalification Evaluating Laboratory, the specimens are aliquotted into working volumes of 250µl and the
remainder into 2ml volumes and stored at -20oC until testing commences. During the period of testing the
specimens are stored at 2 - 8 °C and this time period does not exceed one week. After the completion of
testing, they are again stored at -20 oC. Each aliquot does not undergo more than two freeze/thaw cycles.

6.1.2.Characterization of the WHO HCV specimen reference panel

The panel consists of minimum of 480 serum/plasma specimens of European, African, Latin American and
Asian origin. There are 150 anti-HCV positive specimens, and 310 anti-HCV negative specimens.

1
ISO 18113-1:2009 In vitro diagnostic medical devices -Information supplied by the manufacturer (labelling) – Part 1: Terms, definitions
and general requirements
WHO/EMP/RHT/PQT/2017.xx Page 5 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017
Table 1 - HCV specimen reference panel
HCV positive specimens HCV negative specimens Total number

160 320 480

The HCV specimen reference panel has been characterized using a standardized combination of assays i.e. a
testing algorithm. These results are used to determine the HCV status of each specimen for the purpose of
this evaluation see Figure 1. Use of any other combination of assays for characterization of the HCV core
Ag specimen evaluation panel shall be communicated, discussed and agreed with WHO beforehand.

Initially, each specimen is tested on the Murex anti HCV EIA (version 4.0) (DiaSorin S.A. Italy) and Monolisa
Anti-HCV Plus version 2.0 (Bio-Rad Laboratories) in parallel.
Specimens that are non-reactive on both EIAs are not further tested and are assigned as anti-HCV negative.

Specimens with discrepant EIA results AND with dually reactive EIA results are tested on the RIBA HCV 3.0
Strip Immunoassasy (Chiron) or the HCV Blot 3.0 WB (MP Biomedicals).

Specimens that are positive by line immunoassay/WB are assigned as anti-HCV positive.

Specimens that are indeterminate or negative by line immunoassay / WB are excluded from the HCV
specimen reference panel.

Specimens that are indeterminate by line immunoassay are assigned as anti-HCV indeterminate and
excluded from the HCV specimen reference panel.

Specimens that are negative by line immunoassay are assigned as anti-HCV negative.

WHO/EMP/RHT/PQT/2017.xx Page 6 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017

EIA 1 DiaSorin Murex Anti-HCV EIA (version 4.0)

EIA 2 Bio-Rad Monolisa Anti-HCV Plus
in parallel

+ + +- / -+ - -

Chiron RIBA HCV 3.0 Strip Immunoassay

/ MP Diagnostics HCV Blot 3.0

Positive Indeterminate Negative

Indeterminate –
Excluded from the panel

Anti-HCV Positive Anti-HCV Negative

Figure 1 - Testing Algorithm for characterization of the WHO HCV Specimen Reference Panel
(Serum/Plasma Specimens)

6.2. Lot-to-lot variation panel

Lot-to-lot variation is assessed on two separate production lots by testing the same dilution series
(comprised of 2-fold dilutions of 10 stock HCV positive specimens in commercially available normal human
serum) in the same testing session.

6.3. HCV seroconversion panels

A seroconversion panel is a series of specimens, sequentially collected over a period of time, from an
individual developing antibody in response to acute infection. The following commercial seroconversion
panels: PHV913, PHV919, PHV 920 and PHV 922 [sourced from SeraCare, Life Sciences Inc] are tested using
the assay under evaluation in singular on one lot. These panels consist of a total of 26 specimens collected
from four individuals during seroconversion.

WHO/EMP/RHT/PQT/2017.xx Page 7 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017
6.4. HCV performance panels
One HCV Mixed Titre performance panel containing 16 members, 0810-0175 [sourced from SeraCare, Life
Sciences Inc] and one anti-HCV low titre performance panel containing 11 members, 0810-0192 [sourced
from SeraCare, Life Sciences Inc] are tested using the assay under evaluation in singular on one lot.

6.5. External quality control specimen

See section 6 for further details

Table 2 - Overview of specimen panels used in laboratory evaluation

Panel name Number of specimens
WHO HCV specimen reference panel 163 HCV positive, 320 HCV negative

Lot-to-lot variation panel 16 member dilution series of 10 specimens, (160 in total)

Commercial HCV seroconversion panels 4 panels comprising 26 specimens in total

Commercial HCV performance panels 2 panels comprising 27 specimens in total

7. Laboratory testing

7.1. Sequence of testing

Each product under evaluation is used strictly in accordance with the instructions for use (IFU) issued by the
manufacturer. The WHO Prequalification Evaluating Laboratory will send a hard or electronic copy of the
IFU to WHO/PQDx upon delivery of the test kits and prior to the commencement of the laboratory
evaluation. The IFU must be reviewed against the IFU submitted to WHO/PQT as part of the dossier
assessment for the prequalification assessment. If the IFU has been updated since dossier submission, a
letter from the manufacturer detailing changes made must be sent to WHO/PQDx prior to the laboratory
evaluation commencing.

The specimen reference panel is run in order that approximately one half of the specimen panel will be run
with the one lot and the other half of the panel with the other lot. The specimens of the WHO HCV
specimen reference panel should initially be tested in singular and in a blinded manner

Lot-to-lot variation is assessed on two separate production lots by testing the same set of dilution series
(comprised 2-fold dilutions of 10 stock HCV positive specimens).

The seroconversion panels, and performance panels are then tested in singular on a single lot.

For the purpose of evaluating the rapid diagnostic tests, a ‘test run’ is defined as a consecutive run of
simple/rapid diagnostic tests of the same production lot performed during the same ‘session’. A ‘testing
session’ might be considered to be a morning or afternoon.

7.2. Recording test results

Scoring index Intensity reading scale

0 Negative

1 Uncertain Reactivity, Indeterminate

2 Very Weak, but Definitely Reactive

WHO/EMP/RHT/PQT/2017.xx Page 8 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017

3 Medium to Strong Reactivity

Invalid (including no control line/band/dot/spot visible, or obviously defective test device,

4 no flow, debris present)

All test results are recorded electronically directly from the plate reader and then entered in a Microsoft
Excel spreadsheet for further data analysis. For subjectively read assays such as rapid diagnostic tests or
line immunoassay, the intensity of band/line/spot is additionally entered into the data collection sheet.
The intensity rating system reads as described in Table 3

Table 3 - Results legend for data collection sheets for subjectively read assays

Visual interpretation of results of subjectively read assays is made independently by three readers (without
the knowledge of the other two sets of results and blinded to the reference result for the specimen) and
entered onto the data collection sheets. These results are compared by the operator carrying out the assay
so that any mistakes may be identified and rectified immediately. Should recording errors be identified,
both the original and corrected result are recorded and initialed by the reader. When the three readers
interpret the results differently from each other, the consensus is recorded as that interpretation which
occurs two out of three times. In cases where all three interpretations are different, the result is recorded
as indeterminate.

A technician's appraisal is made of each assay under evaluation and is completed by the operator
performing the testing. It includes questions about the ease of the procedure, reading of results, clarity of
IFU, as well as room to record any specific difficulties encountered during the evaluation.

8. Quality control and interpretation of test results

8.1. Test kit controls

Manufacturer-supplied positive and negative test kit controls will be run as indicated in the IFU for all test
formats included in each test run for EIAs and at the commencement of each testing session for rapid
diagnostic tests and other formats. Where positive and negative test kit controls are not supplied by the
manufacturer, as will be the case for many rapid diagnostic tests, the external quality control specimen will
act at the control specimen, see section 9.3

8.2. Internal control lines for rapid diagnostic tests

Generally, rapid diagnostic tests contain a control band, line or spot to determine that the test device is
operating correctly. Most control bands/lines/spots will become visible with the addition of reagent (i.e.
buffer). However, some rapid diagnostic tests will contain a control band/line/spot that becomes visible
with addition of specimen (i.e. presence of IgG). It is imperative that the exact nature of the control
band/line/spot is ascertained and recorded in the report. An experiment is performed to verify this point, if
not explicitly mentioned in the IFU.

8.3. External quality control specimen

The WHO prequalification evaluating laboratory supplies an external quality control (QC) specimen which is
tested in singular at the beginning of each test session for rapid diagnostic tests and other formats and in
triplicate for each test run for EIAs. See definition of testing session in section 5.1. The QC specimen
represents a lowly reactive HCV positive specimen, and thus may be different for different assays and
different assay formats. The QC specimen is supplied by the WHO prequalification evaluating laboratory or
acquired commercially, depending on the assay under evaluation.

WHO/EMP/RHT/PQT/2017.xx Page 9 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017
8.4. Proficiency panels
A proficiency panel must be run successfully for each assay by each operator before the evaluation
commences. This may be the same panel as that used at the time of assay demonstration by the
manufacturer or for training purposes.

8.5. Limits of acceptability

All results on test kits controls and QC specimens are entered on the data collection sheets. Should the test
kit controls or the QC specimen not give results within the expected ranges, evaluation testing on that
assay is suspended until the cause has been identified and a satisfactory solution identified. Such problems
must be communicated immediately to WHO and recorded on the data sheets. The PI is responsible for
carefully checking all data entry forms for legibility, accuracy and completeness.

8.6. Interpretation of results

The interpretation of results for each assay under evaluation is made strictly according to the
manufacturers’ instructions within the IFU. Invalid test results are recorded on the data collection sheets
including where the control line does not appear or in any other way the test result is invalid as defined by
the IFU. For test results that are indeterminate according to the IFU, the results are recorded on data
collection sheets.

9. Analysis of data

9.1. Invalid runs/devices

The number of invalid test runs (if EIA) is recorded as the number of invalid runs as a percentage of the
total number of runs performed for clinical specimens only (excluding the commercially acquired panels, lot
variation panels, and WHO reference preparation).

The number of invalid devices (if rapid diagnostic test or other format) is recorded as the number of invalid
test devices as a percentage of the total number of devices used for the entire evaluation.

Invalid results may mean invalid test results as defined by the IFU where the control line/band/spot does
not appear or invalid due to obviously defective test device or defective transfer pipette.

9.2. Inter-reader variability

The inter-reader variability is calculated when assay readings must be performed without any objective
reading instruments i.e. rapid diagnostic tests. Three persons independently interpret each test result. The
inter-reader variability is expressed as the percentage of specimens for which initial test results are
differently interpreted (i.e. reactive or non-reactive or indeterminate) by the independent readers.

9.3. Performance characteristics from WHO HCV specimen reference panel

The following strategies are used to calculate the performance characteristics for each assay under
evaluation and is closely linked to the reference testing results gained during specimen panel
characterization.

Table 4 - 2 x 2 table for calculation of performance characteristics

Reference testing results

Reactive Non-reactive Total
Reactive a b a+b
Results of assay under evaluation (true positives) (false positives)

WHO/EMP/RHT/PQT/2017.xx Page 10 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017
Non-reactive c d c+d
(false negatives) (true negatives)
Total a+c b+d a+b+c+d

9.3.1.Sensitivity
Sensitivity is the ability of the assay under evaluation to detect correctly specimens that contain HCV
antibodies and/or HCV antigen (reference results positive). Thus sensitivity is the number of true positive
specimens identified by the assay under evaluation as positive (a), divided by the number of specimens
identified by the reference assays as positive (a+c), expressed as a percentage.
Sensitivity = a
ac

9.3.2.Specificity
Specificity is the ability of the assay under evaluation to detect correctly specimens that do not contain HCV
antibodies and/or HCV antigen (reference results negative). Thus specificity is the number of true negative
specimens identified by the assay under evaluation as negative (d), divided by the number of specimens
identified by the reference assays as negative (b+d), expressed as a percentage.
Specificity = d
bd

9.3.3.Confidence intervals
The 95% confidence intervals are calculated for both sensitivity and specificity in order to assess the level of
uncertainty introduced by sample size, etc. Exact 95% confidence intervals for binomial proportions were
calculated from the F-distribution. [Armitage, 2002; Kirkwood, 2003]

9.3.4.Positive predictive value (PPV)

The probability that when the test is reactive that the specimen does contain HCV antibodies and/or HCV
antigen. PPVs were calculated using the formula.

PPV =  prevalencesensitivity 
 prevalencesensitivity   1  prevalence1  specificit y 
9.3.5.Negative predictive value (NPV)
The probability that when the test is negative that a specimen does not have contain HCV antibodies
and/or HCV antigen. NPVs were calculated using the formula.

NPV = 1  prevalencespecificit y 
1  prevalencespecificit y    prevalence1  sensitivity 

The probability that a test result will accurately determine the true infection status of a person being tested
varies with the prevalence of HCV infection in the population from which the person comes. In general, the
higher the prevalence of HCV infection in the population, the greater the probability that a person testing
positive is truly infected (i.e., the greater the positive predictive value [PPV]). Thus, with increasing
prevalence, the proportion of individuals testing false-positive decreases; conversely, the likelihood that a
person whose test result is negative is truly uninfected (i.e., the negative predictive value [NPV]), decreases
as prevalence increases. Therefore, as prevalence increases, so does the proportion of individuals testing
false-negative.

The PPV and NPV are calculated at prevalences of 0.1%, 1% and 5%.

WHO/EMP/RHT/PQT/2017.xx Page 11 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017
9.4. Indeterminate results
For the WHO specimen reference panel only: specimens which are found to be indeterminate (grey zone)
by the criteria stated in the instructions for use should be retested in duplicate on the same lot number of
assay. In the case that the testing result cannot be resolved after all testing, the specimen is to be called
indeterminate and included in sensitivity/specificity calculations. A value for initial sensitivity and specificity
are calculated based on the results obtained for the assay under evaluation on the first lot. The final
sensitivity and specificity values are calculated taking into consideration the repeat testing performed on a
same lot and further testing second lot of the assay under evaluation.

9.5. Discrepant results

Those specimens with results that are consistent with the reference results i.e. the characterized specimen
results undergo no further testing. Those specimens with results discrepant from the reference result are
retested in duplicate using the same lot number by the same operator. The results that occur two out of
three times are recorded as the final result. If the result is again discrepant, the specimen is retested on a
second lot number, if available. If the result on the second lot is concordant with the reference result, no
further testing is required. If the result is still discrepant from the reference results, the result is recorded
as is.

Should a negative specimen from the HCV specimen reference panel give a repeatedly reactive result in the
assay under evaluation it will be further characterized with additional serology (antigen only; antibody only)
and molecular techniques. This additional testing will be conducted to rule out the remote possibility that
the reactivity in the assay under evaluation is true, and is more sensitive than the assays with which the
panel specimens were characterized.

An initial sensitivity and specificity are calculated based on the initial results obtained for the assay under
evaluation on the first lot. The final sensitivity and specificity are calculated taking into consideration the
results of repeat testing performed on a second lot of the assay under evaluation i.e. if found to be
concordant on the second lot will be recorded as such, and if found to be discrepant on the second lot will
be recorded as such.

9.6. Interpretation of results from lot-to-lot variation panel

The results of the lot-to-lot panel for the two production lots are compared and a variation of +/- 1 dilution
series is considered acceptable.

9.7. Interpretation of seroconversion sensitivity

The results obtained from seroconversion panels using the assay under evaluation are compared with those
obtained using DiaSorin Murex Anti-HCV EIA (version 4.0) EIA; the assay arbitrarily designated the reference
for determination of relative sensitivity in these panels. For each seroconversion series (panel) the first
specimen in the sequence to become reactive with DiaSorin Murex Anti-HCV EIA (version 4.0) EIA is
assigned the value “0”. Results from the assay under evaluation are compared with DiaSorin Murex Anti-
HCV EIA (version 4.0) EIA by determining the difference between the specimen assigned value “0” and the
relative position in the sequence of the first specimen showing a reactive result for the assay under
evaluation. For example, if an assay becomes reactive two specimens earlier in a series than DiaSorin
Murex Anti-HCV EIA (version 4.0) EIA , the value assigned for that series in that assay would be -2. Similarly,
if an assay becomes reactive one specimen later than DiaSorin Murex Anti-HCV EIA (version 4.0) EIA, the
value assigned would be +1. The assigned values over the four seroconversion panels (series) are averaged
to determine a mean relative seroconversion sensitivity index for each assay and the 95% confidence
interval is determined.

9.8. Interpretation of results from performance panels

The number of specimens correctly identified by the assay under evaluation in the HCV performance panels
is determined by comparison with DiaSorin Murex Anti-HCV EIA (version 4.0) EIA.

WHO/EMP/RHT/PQT/2017.xx Page 12 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017
9.9. Interpretation of results from WHO international reference preparations
The number of specimens identified by the assay under evaluation for the WHO international reference
preparation are determined by comparison with the combined reference results generated by NRLHPA:
DiaSorin Murex Anti-HCV EIA (version 4.0) EIA HCV ELISA Test System 3.0 with Enhanced SAVe [short
incubation] (Ortho-Clinical Diagnostics), Monolisa Anti-HCV Plus version 2.0 (Bio-Rad Laboratories), and
CHIRON RIBA HCV 3.0 (Ortho-Clinical Diagnostics).

9.10. Technician's appraisal

The technical aspects of the assays under evaluation is assessed by the technician who performed the
testing. These assessments, along with other selected assay characteristics, contribute to an overall
appraisal of each assay’s suitability for use in small laboratories. To enable comparison between assays, a
scoring system is used to rate specified operational characteristics.

9.11. Report preparation

The data analysis and report drafting is carried out by the WHO prequalification evaluating laboratory and
sent to WHO for List 1 laboratories. In case of list two laboratories the data and the report will be sent
simultaneously to WHO and the Manufacturer in a timely manner. WHO verifies the draft report from List
one laboratories and sends to the authorized contact designated by the manufacturer for comment. The
company has one month right of reply. The final report is prepared after one month has elapsed. The
WHO scientist ensures that the comments of the authorized contact are reviewed and any outstanding
issues are resolved before final publication. The final report is prepared and disseminated by WHO. A copy
of the final report is sent to the authorized contact designated by the manufacturer and to WHO
prequalification evaluating laboratory.

10. Materials and supplies

10.1. Data collection sheets

All data will be reported to WHO/PQ Diagnostics assessment on the following forms:

A. Data collection Microsoft Excel spreadsheet for the rapid diagnostic tests
B. Data collection Microsoft Excel spreadsheet for EIAs (print-outs)
C. Technician's appraisal worksheet
D. Findings of supplementary testing on discrepant specimens

10.2. Supplies
The manufacturers of products will provide the products and any equipment necessary for the evaluation
free of charge.

11. Roles and responsibilities

11.1. Responsibilities of the WHO Prequalification Evaluating Laboratory

1. Act as repository for the HCV specimen reference panel, lot-to-lot variation, seroconversion
and performance panels;
2. Conducting the laboratory evaluation according to the agreed protocol;
3. Preparation of QC specimens and proficiency panels;
4. Preparation of draft report on laboratory evaluation;
5. Advising WHO on operational characteristics of assays evaluated.

All source data, data analysis records and all correspondence are retained and archived for a period of at
least ten years.

11.2. Responsibilities of WHO/PQ Diagnostics assessment

1. Technical advice to the PI;
2. Technical and administrative management of the laboratory evaluation;
WHO/EMP/RHT/PQT/2017.xx Page 13 of 14
WHO Protocol for Performance Evaluation of HCV serology assays
WHO PQDX_040 v6 February 2017
3. Verification of the draft report, seeking of comments from manufacturer;
4. Preparation and dissemination of the final report;
5. Formal contacts with authorized contacts of the manufacturers.

Any publication by WHO of the results of these evaluations and the WHO recommendations derived
therefrom will, however, be accompanied by the following disclaimer:

The mention of specific companies or of certain manufacturers’ products does not imply that they are
endorsed or recommended by the World Health Organization in preference to others of a similar nature that
are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by
initial capital letters.

WHO and the WHO Prequalification Evaluating Laboratory, do not warrant or represent that the
evaluations conducted with the HCV test kits referred to in this document are accurate, complete and/or
error-free. WHO and the WHO Prequalification Evaluating Laboratory disclaim all responsibility for any use
made of the data contained herein, and shall not be liable for any damages incurred as a result of its use.
This document must not be used in conjunction with commercial or promotional purposes.

12. References
Armitage P, Berry G, Matthews, JNS, Statistical Methods in Medical Research, 4th Edition. Blackwell
Scientific Publications, Oxford, 2002

Kirkwood B, Stern J, Essential Medical Statistics 2nd edition, Blackwell Science Ltd. 2003

World Health Organization Guidelines on HIV Safety Precautions, and Guidelines for the Safe Transport of
Specimens (WHO/EMC/97.3)

International Standards
EN 13612:2002 Performance evaluation of in vitro diagnostic medical devices

ISO 17025 (General requirements for the competence of testing and calibration laboratories)

ISO15189 (Medical laboratories — Particular requirements for quality and competence)

13. Other documents required

Work Instructions
SOP_PQDx_072 PQT work instruction for laboratory testing at the WHO Prequalification Evaluating
Laboratory
SOP_PQDx_074 PQT work instruction for data entry and analysis
SOP_PQDx_075 PQT work instruction for report preparation and dissemination

Master Templates
SOP_PQDx_136 PQDx report template for 3rd generation HCV EIAs including excel spreadsheets of results
SOP_PQDx_137 PQDx report template for 3rd generation HCV simple/rapid assays for non-discriminatory
detection including excel spreadsheets of results

WHO/EMP/RHT/PQT/2017.xx Page 14 of 14