Quality Analysis Checklist for Primary Research

Ref 1 Ref 2 Ref 3 Ref 4 Ref 5 Ref 6 Tally

RELEVANCE QUESTIONS
1. Would implementing the studied intervention or procedure (if found successful) result in improved
outcomes for the patients/clients/population group? (NA for some Epi studies) Yes Yes Yes Yes Yes Y6s
6
2. Did the authors study an outcome (dependent variable) or topic that the patients/clients/population
group would care about? Yes Yes Yes Yes Yes Yes
6
3. Is the focus of the intervention or procedure (independent variable) or topic of study a common issue of
concern to dietetics practice? Yes Yes Yes Yes Yes Yes
6
4. Is the intervention or procedure feasible? (NA for some epidemiological studies) Yes Yes Yes Yes Yes Yes 6
If the answers to all of the above relevance questions are “Yes,” the report is eligible for designation with
a plus (+) on the Evidence Quality Worksheet, depending on answers to the following validity questions.

VALIDITY QUESTIONS
1. Was the research question clearly stated?
1.1 Was the specific intervention(s) or procedure (independent variable(s)) identified?
1.2 Was the outcome(s) (dependent variable(s)) clearly indicated? Yes Yes Yes Yes Yes Yes
1.3 Were the target population and setting specified?
6
2. Was the selection of study subjects/patients free from bias?
2.1 Were inclusion/exclusion criteria specified (e.g., risk, point in disease progression, diagnostic or
prognosis criteria), and with sufficient detail and without omitting criteria critical to the study?
2.2 Were criteria applied equally to all study groups?
2.3 Were health, demographics, and other characteristics of subjects described? Yes Yes Yes Yes Yes Yes
2.4 Were the subjects/patients a representative sample of the relevant population?

6
3. Were study groups comparable?
3.1 Was the method of assigning subjects/patients to groups described and unbiased? (Method of
randomization identified if Randomized Controlled Trial (RCT))
3.2 Were distribution of disease status, prognostic factors, and other factors (e.g., demographics) similar
across study groups at baseline?
3.3 Were concurrent controls used? (Concurrent preferred over historical controls.)
3.4 If cohort study or cross-sectional study, were groups comparable on important confounding factors
and/or were preexisting differences accounted for by using appropriate adjustments in statistical analysis?
3.5 If case control study, were potential confounding factors comparable for cases and controls? (If case
series or trial with subjects serving as own control, this criterion is not applicable. Criterion may not be Yes Yes Yes Yes Yes Yes
applicable in some cross-sectional studies.)
3.6 If diagnostic test, was there an independent blind comparison with an appropriate reference standard
(e.g., “gold standard”)?

6
 Ref 1 Ref 2 Ref 3 Ref 4 Ref 5 Ref 6 Tally
4. Was method of handling withdrawals described?
4.1 Were follow up methods described and the same for all groups?
4.2 Was the number, characteristics of withdrawals (i.e., dropouts, lost to follow up, attrition rate) and/or
response rate (cross-sectional studies) described for each group? (Follow up goal for a strong study is 80%.)
4.3 Were all enrolled subjects/patients (in the original sample) accounted for?
4.4 Were reasons for withdrawals similar across groups? Yes Yes Yes Yes Yes Yes
4.5 If diagnostic test, was decision to perform reference test not dependent on results of test under
study?

6
5. Was blinding used to prevent introduction of bias?
5.1 In intervention study, were subjects, clinicians/practitioners, and investigators blinded to treatment
group, as appropriate?
5.2 Were data collectors blinded for outcomes assessment? (If outcome is measured using an objective
test, such as a lab value, this criterion is assumed to be met.)
5.3 In cohort study or cross-sectional study, were measurements of outcomes and risk factors blinded?
5.4 In case control study, was case definition explicit and case ascertainment not influenced by exposure Yes Yes Yes Yes Yes Yes
status?
5.5 In diagnostic study, were test results blinded to patient history and other test results?

6
6. Were intervention/therapeutic regimens/exposure factor or procedure and any comparison(s)
described in detail? Were intervening factors described?
6.1 In RCT or other intervention trial, were protocols described for all regimens studied?
6.2 n observational study, were interventions, study settings, and clinicians/provider described?
6.3 Was the intensity and duration of the intervention or exposure factor sufficient to produce a
meaningful effect?
6.4 Was the amount of exposure and, if relevant, subject/patient compliance measured?
6.5 Were co-interventions (e.g., ancillary treatments, other therapies) described?
6.6 Were extra or unplanned treatments described? Yes Yes Yes Yes Yes Yes
6.7 Was the information for 6.4, 6.5, and 6.6 assessed the same way for all groups?
6.8 In diagnostic study, were details of test administration and replication sufficient?

6
7. Were outcomes clearly defined and the measurements valid and reliable?
7.1 Were primary and secondary endpoints described and relevant to the question?
7.2 Were nutrition measures appropriate to question and outcomes of concern?
7.3 Was the period of follow-up long enough for important outcome(s) to occur?
7.4 Were the observations and measurements based on standard, valid, and reliable data collection
instruments/tests/procedures?
Yes Yes Yes Yes Yes Yes
7.5 Was the measurement of effect at an appropriate level of precision?
7.6 Were other factors accounted for (measured) that could affect outcomes?
7.7 Were the measurements conducted consistently across groups?

6
 Ref 1 Ref 2 Ref 3 Ref 4 Ref 5 Ref 6 Tally
8. Was the statistical analysis appropriate for the study design and type of outcome indicators?
8.1 Were statistical analyses adequately described the results reported appropriately?
8.2 Were correct statistical tests used and assumptions of test not violated?
8.3 Were statistics reported with levels of significance and/or confidence intervals?
8.4 Was “intent to treat” analysis of outcomes done (and as appropriate, was there an analysis of
outcomes for those maximally exposed or a dose-response analysis)?
8.5 Were adequate adjustments made for effects of confounding factors that might have affected the Yes Yes Yes Yes Yes Yes
outcomes (e.g., multivariate analyses)?
8.6 Was clinical significance as well as statistical significance reported?
8.7 If negative findings, was a power calculation reported to address type 2 error?

6
9. Are conclusions supported by results with biases and limitations taken into consideration?
9.1 Is there a discussion of findings? Yes Yes Yes Yes Yes Yes
9.2 Are biases and study limitations identified and discussed?
6
10. Is bias due to study’s funding or sponsorship unlikely?
10.1 Were sources of funding and investigators’ affiliations described? Yes Yes Yes yes Yes Yes
10.2 Was there no apparent conflict of interest?
6
MINUS/NEGATIVE (-):
If most (six or more) of the answers to the above validity questions are “No,” the report should be
designated with a minus (-) symbol on the Evidence Worksheet.

NEUTRAL (∅):
If the answers to validity criteria questions 2, 3, 6, and 7 do not indicate that the study is exceptionally
strong, the report should be designated with a neutral ( ∅) symbol on the Evidence Worksheet.

PLUS/POSITIVE (+):
If most of the answers to the above validity questions are “Yes” (including criteria 2, 3, 6, 7 and at least one
additional “Yes”), the report should be designated with a plus symbol (+) on the Evidence Worksheet.
 Ref 1 Ref 2 Ref 3 Ref 4 Ref 5 Ref 6 Tally
When a validity criteria question is NA:
If any of the ten validity questions are NA, the report requires a majority of “Yes” answers (including 2, 3, 6,
7, as applicable) for a plus (+), or a majority or “No” answers for a minus (-) rating.

Estimate of Relevance Questions 10 10 10 10 10 10 60

Estimate of Validity Questions 10 10 10 10 10 10 60
Overall Summary 10 10 10 10 10 10 60

Appendix 17: Grade Definitions: Strength of the Evidence for a Conclusion/Recommendation

Instructions: Compile Evidence Worksheets of all studies and reports relevant to each key question
addressed by the clinical recommendation, practice guideline or position statement. The expert panel
makes a considered judgment to formulate each conclusion statement using its knowledge of the evidence
and methods used to generate it. Then a grade is assigned to indicate the strength of the evidence
supporting the conclusion statement.

Grade I: Good—The evidence consists of results from studies of strong design for answering the question
addressed. The results are both clinically important and consistent with minor exceptions at most. The
results are free of serious doubts about generalizability, bias, and flaws in research design. Studies with
negative results have sufficiently large sample sizes to have adequate statistical power.

Grade II: Fair—The evidence consists of results from studies of strong design answering the question
addressed, but there is uncertainty attached to the conclusion because of inconsistencies among the results
from different studies or because of doubts about generalizability, bias, research design flaws, or adequacy
of sample size. Alternatively, the evidence consists solely of results from weaker designs for the questions
addressed, but the results have been confirmed in separate studies and are consistent with minor
exceptions at most.

Grade III: Limited—The evidence consists of results from a limited number of studies of weak design for
answering the questions addressed. Evidence from studies of strong design is either unavailable because no
studies of strong design have been done or because the studies that have been done are inconclusive due
to lack of generalizability, bias, design flaws, or inadequate sample sizes.

Grade IV: Expert Opinion Only—The support of the conclusion consists solely of the statement of informed
medical commentators based on their clinical experience, unsubstantiated by the results of any research
studies.

Grade V: Not Assignable*— There is no evidence available that directly supports or refutes the conclusion.
 Grades
I II III IV V
Strength of Evidence Elements Good Fair Limited Expert Opinion Grade Not Assignable
Look at: Quality - Studies of strong design for - Studies of strong design for - Studies of weak design for - No studies available No evidence that pertains to
Quality Checklist - scientific rigor/validity question question with minor answering the question - Conclusion based on usual question being addressed
Questions: - considers design & execution - Free from design flaws, bias & methodological concerns OR practice, expert consensus, clinical
Validity Questions execution problems OR - Inconclusive findings due to experience, opinion, or
- Only studies of weaker study design flaws, bias or execution extrapolation from basic research
design for question problems

Look at: Consistency of findings across studies Findings generally consistent, with - Inconsistency among results of - Unexplained inconsistency Conclusion supported solely by NA
Results from studies minor exceptions at most, in: studies with strong design among results from different statements of informed nutrition
you reviewed for - direction OR studies or medical commentators
direction of results, i.e. - size of effect or degree of - Consistency with minor OR
treatment is effective association exceptions across studies of - Single study unconfirmed by
or not - statistical significance weaker design other studies

Look at: Quantity - One to several good quality - Several studies by independent - Limited number of studies Unsubstantiated by published Relevant studies have not been
- number of studies - number of studies studies investigators - Low number of subjects studied research studies done
resulting from your - number of subjects in studies - Large number of subjects - Doubts about adequacy of and/or inadequate sample size
search & exclusion studied sample size to avoid Type I and within studies
- sample size in studies - Studies with negative results Type II error
- results of Quality have sufficiently large sample size
rating above for adequate statistical power

Look at: Clinical Impact - Studied outcome relates directly Some doubt about the statistical - Studied outcome is an Objective data unavailable Indicates area for future research
Quality Checklist - importance of studied outcomes to the question or clinical significance of the effect intermediate outcome or
Questions: Relevance - magnitude of effect - Size of effect is clinically surrogate for the true outcome of
Questions meaningful interest
- Significant (statistical) difference OR
is large - Size of effect is small or lacks
statistical and/or clinical
significance

Look at: Generalizability to population of Studied population, intervention Minor doubts about Serious doubts about Generalizability limited to scope of NA
Quality Checklist interest and outcomes are free from generalizability generalizability due to narrow or experience
Questions: Relevance serious doubts about different study population,
Questions generalizability intervention or outcomes studies