IE-472 Design of Experiments

Fall-21

Course Instructor: Prof. Dr. Misbah Ullah

Assistant Teacher: Engr. Muhammad Nauman

University of Engineering & Technology, Peshawar
Department of Industrial Engineering
 What If There Are More Than Two Factor Levels?
• Previously, we discussed methods for comparing two conditions or
treatments. For example, the Portland cement tension bond
experiment involved two different mortar formulations.
• Another way to describe this experiment is as a single-factor
experiment with two levels of the factor, where the factor is mortar
formulation and the two levels are the two different formulation
methods.
• Many experiments of this type involve more than two levels of the
factor.
• This session focuses on methods for the design and analysis of single
factor experiments with an arbitrary number a levels of the factor (or
a treatments).
 What If There Are More Than Two Factor Levels?
• The t-test does not directly apply
• There are lots of practical situations where there are either more than
two levels of interest, or there are several factors of simultaneous
interest
• The analysis of variance (ANOVA) is the appropriate analysis
“engine” for these types of experiments.
• The ANOVA was developed by Fisher in the early 1920s, and initially
applied to agricultural experiments
• Used extensively today for industrial experiments
ANOVA calculations are usually done via computer. Text exhibits sample
calculations from two very popular software packages, Design-Expert and
Minitab. Text discusses some of the summary statistics provided by these
packages.
where yij is the ijth observation, µi is the mean of the ith factor level or
treatment, and Єij is a random error component that incorporates all
other sources of variability in the experiment including measurement,
variability arising from uncontrolled factors, differences between the
experimental units (such as test material, etc.) to which the treatments
are applied, and the general background noise in the process (such as
variability over time, effects of environmental variables, and so forth).
This model is called the Means Model.
Analysis of Variance
• Suppose we run an experiment and observe the response of “y” at “a”
settings of a variable or factor. At each setting, we take multiple
readings called replication.
• In general, there will be a levels of the factor, or a treatments, and n
replicates of the experiment, run in random order… a completely
randomized design (CRD)
• N = an total runs
• Objective is to test hypotheses about the equality of the a treatment
means
Grand Summation Grand Mean
 The Analysis of Variance

Partitioned difference from the grand mean

The difference of each observation, Y, from the grand
mean can be partitioned into two parts:
i. The difference between the individual
observation and the group mean.
ii. The difference between the group mean and the
grand mean.

𝑌 − 𝑌 = 𝑌 − 𝑌𝑗 + (𝑌𝑗 − 𝑌)

Grand Mean Group Mean Grand Mean

This partitioned relationship is also true for the sums of the squared
differences which are called ‘sums of squares’. In the formula below
Yij refers to an individual observation i in group j.

(𝑦𝑖𝑗 − 𝑦.. )2 = (𝑦𝑖𝑗 − 𝑦𝑖. )2 + (𝑦𝑖. − 𝑦.. )2

In words, the total sum of squares (SST) is equal to the Error Sum of squares (SSE) plus
the sum of squares among groups (SSTreatments).
The identity above expresses how between-treatment and within-treatment variation
add to the total sum of squares.
SST
The total corrected sum of squares measures the total variability in the data.

SSTreatments
It is sum of squares of the differences between the treatment averages and the grand
average. The difference between the observed treatment averages and the grand
average is a measure of the differences between treatment means.

SSE
It is sum of squares of the differences of observations within treatments from the
treatment average. The differences of observations within a treatment from the
treatment average can be due to only random error. MSE is a pooled estimate of the
common variance within each of the a treatments.
If the null hypothesis is true, the between treatment variation (numerator) will not
considerably exceed the residual or error variation (denominator) and the F statistic will
be small. If the null hypothesis is false, then the F statistic will be large.
The Analysis of Variance
The Analysis of Variance
 The Analysis of Variance

a= treatments level, n= replications, N= a*n

𝑦 2
𝑎 𝑛 𝑎 𝑛 2
𝑆𝑆𝑇 = 𝑖=1 𝑗 =1(𝑦𝑖𝑗 − 𝑦.. )2 = 𝑖=1 𝑗 =1 𝑦𝑖𝑗
− ..
𝑁
• Take square all the observations and add them. Then subtract
(square of grand summation/N) from this calculated value.
𝑎 2
𝑎 𝑦
𝑖=1 𝑖𝑗 𝑦 ..2
𝑆𝑆𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 = 𝑛 𝑖=1(𝑦𝑖. − 𝑦.. )2 = −
𝑛 𝑁
• Take square of all the levels/treatments averages, add all the
treatments averages and divide it by “n”. Then subtract (square of
grand summation/N) from this calculated value.

𝑆𝑆𝐸 = 𝑆𝑆𝑇 − 𝑆𝑆𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡

 The Analysis of Variance

• The reference distribution for F0 is the Fa-1, a(n-1)

distribution
• Reject the null hypothesis (equal treatment means) if

F0  F , a −1, a ( n −1)
The Analysis of Variance is Summarized in a
Table
a =5, n=5, a(n-1) = 20
Fα,a-1, a(n-1) = F0.05, 4, 20 = 2.87
❑If the model assumptions are reasonable, then the residuals
• Should appear to be normally distributed
• Should be close to statistically independent
• Have a constant variance; the residual variance should not differ for
different treatment groups

❑Residual Plots:
• Normal QQ Plot: Residuals should be scattered about a straight line.
• Run Chart: There should be no systematic pattern.
• Plot of Residuals Versus Fitted Values 𝑦(𝑖. ): There should be no systematic
pattern.

❑Remedial Measures:
•If the residuals are not close to normal, a data transformation might help
•A pattern on the run chart may be indicating that the measurements are not
independent. Run order may be an important factor – it should be included in the
experimental design.
 𝜀𝑖𝑗 = 𝑦𝑖 randomized
For the completely ,𝑗 − 𝜇𝑖 design,𝑒𝑖the
,𝑗 = 𝑦𝑖 ,𝑗 −
residuals are 𝑦𝑖 .
𝑒𝑖𝑗 = 𝑦𝑖 ,𝑗 − 𝑦𝑖 . For i= 1, 2,…a j= 1, 2,…n

Residual i,j= ijth value of observation – average of that corresponding treatment

Residuals are the deviations of the observations from the sample mean. It is an
estimate of the random errors.
Random errors are the deviations of the observations from the population mean.
Plot of Residuals in Time Sequence
Plotting the residuals in time order of data collection is helpful in detecting
strong correlation between the residuals. A tendency to have runs of positive
and negative residuals indicates positive correlation. This would imply that
the independence assumption on the errors has been violated. This is a
potentially serious problem and one that is difficult to correct, so it is
important to prevent the problem if possible when the data are collected.
Proper randomization of the experiment is an important step in obtaining
independence. Sometimes the skill of the experimenter (or the subjects) may
change as the experiment progresses, or the process being studied may “drift”
or become more erratic.
This will often result in a
change in the error variance
over time. This condition
often leads to a plot of
residuals versus time that
exhibits more spread at one
end than at the other.
 Plot of Residuals in Time Sequence

Systematic patterns on the run charts are not evident. Hence, the
independence assumption is not violated.
Plot of Residuals Versus Fitted Values

If the model is correct and the assumptions are satisfied, the residuals
should be structureless; in particular, they should be unrelated to any
other variable including the predicted response.
A defect that occasionally shows up on this plot is nonconstant
variance. Sometimes the variance of the observations increases as the
magnitude of the observation increases. This would be the case if the
error or background noise in the experiment was a constant percentage of
the size of the observation. (This commonly happens with many
measuring instruments—error is a percentage of the scale reading.)
ANOVA table for Tensile Strength Example
Optional
 An Example

• An engineer is interested in investigating the relationship between the

RF power setting and the etch rate for this tool. The objective of an
experiment like this is to model the relationship between etch rate and
RF power, and to specify the power setting that will give a desired
target etch rate.
• The response variable is etch rate.
• She is interested in a particular gas (C2F6) and gap (0.80 cm), and
wants to test four levels of RF power: 160W, 180W, 200W, and 220W.
She decided to test five wafers at each level of RF power.
• The experimenter chooses 4 levels of RF power 160W, 180W, 200W,
and 220W
• The experiment is replicated 5 times – runs made in random order
 An Example

• Does changing the

power change the
mean etch rate?
• Is there an optimum
level for power?
 The Analysis of Variance

• In general, there will be a levels of the factor, or a treatments, and n

replicates of the experiment, run in random order…a completely
randomized design (CRD)
• N = an total runs
• We consider the fixed effects case…the random effects case will be
discussed later
• Objective is to test hypotheses about the equality of the a treatment
means
 The Analysis of Variance
• The name “analysis of variance” stems from a
partitioning of the total variability in the response
variable into components that are consistent with a
model for the experiment
• The basic single-factor ANOVA model is

 i = 1, 2,..., a
yij =  +  i +  ij , 
 j = 1, 2,..., n

 = an overall mean,  i = ith treatment effect,

 ij = experimental error, NID(0,  2 )
 Models for the Data

There are several ways to write a model for

the data:
yij =  +  i +  ij is called the effects model
Let i =  +  i , then
yij = i +  ij is called the means model
Regression models can also be employed
 The Analysis of Variance
• Total Variability is measured by the total sum of
squares:
a n
SST =  ( yij − y.. ) 2

i =1 j =1

• The basic ANOVA partitioning is:

a n a n

 ij ..  i. .. ij i.
( y − y
i =1 j =1
) = [( y −2
y ) + ( y − y
i =1 j =1
)]2

a a n
= n ( yi. − y.. ) 2 +  ( yij − yi. ) 2
i =1 i =1 j =1

SST = SSTreatments + SS E
 The Analysis of Variance

SST = SSTreatments + SSE

• A large value of SSTreatments reflects large differences in
treatment means
• A small value of SSTreatments likely indicates no differences in
treatment means
• Formal statistical hypotheses are:

H 0 : 1 = 2 = = a
H1 : At least one mean is different
 The Analysis of Variance
• While sums of squares cannot be directly compared to test
the hypothesis of equal means, mean squares can be
compared.
• A mean square is a sum of squares divided by its degrees
of freedom:

dfTotal = dfTreatments + df Error

an − 1 = a − 1 + a(n − 1)
SSTreatments SS E
MSTreatments = , MS E =
a −1 a(n − 1)
• If the treatment means are equal, the treatment and error
mean squares will be (theoretically) equal.
• If treatment means differ, the treatment mean square will
be larger than the error mean square.
 The Analysis of Variance is Summarized in a
Table

• Computing…see text, pp 66-70

• The reference distribution for F0 is the Fa-1, a(n-1) distribution
• Reject the null hypothesis (equal treatment means) if
F0  F ,a −1,a ( n−1)
ANOVA Table
Example 3-1
The Reference Distribution:
 Model Adequacy Checking in the ANOVA

• Checking assumptions is important

• Normality
• Constant variance
• Independence
• Have we fit the right model?
• Later we will talk about what to do if some
of these assumptions are violated
 Model Adequacy Checking in the ANOVA
• Examination of residuals
(see text, Sec. 3-4, pg. 75)
eij = yij − yˆij
= yij − yi.

• Design-Expert generates
the residuals
• Residual plots are very
useful
• Normal probability plot
of residuals
Other Important Residual Plots
 Post-ANOVA Comparison of Means

• The analysis of variance tests the hypothesis of equal

treatment means
• Assume that residual analysis is satisfactory
• If that hypothesis is rejected, we don’t know which specific
means are different
• Determining which specific means differ following an
ANOVA is called the multiple comparisons problem
• There are lots of ways to do this…see text
• We will use pairwise t-tests on means…sometimes called
Fisher’s Least Significant Difference (or Fisher’s LSD)
Method
Design-Expert Output
Graphical Comparison of Means
The Regression Model
 Why Does the ANOVA Work?
We are sampling from normal populations, so
SSTreamtents SS E
 2
a −1 if H is true, and  2
a ( n −1)
 
2 0 2

Cochran's theorem gives the independence of

these two chi-square random variables
SSTreatments /(a − 1)  a2−1 /(a − 1)
So F0 = Fa −1,a ( n −1)
SS E /[a(n − 1)]  2
a ( n −1) /[a(n − 1)]
n
n  i2
Finally, E ( MSTreatments ) =  2 + i =1
and E ( MS E ) =  2
a −1
Therefore an upper-tail F test is appropriate.
 Sample Size Determination

• FAQ in designed experiments

• Answer depends on lots of things; including what
type of experiment is being contemplated, how it
will be conducted, resources, and desired
sensitivity
• Sensitivity refers to the difference in means that
the experimenter wishes to detect
• Generally, increasing the number of replications
increases the sensitivity or it makes it easier to
detect small differences in means
 Sample Size Determination
Fixed Effects Case

• Can choose the sample size to detect a specific

difference in means and achieve desired values of
type I and type II errors
• Type I error – reject H0 when it is true ( )
• Type II error – fail to reject H0 when it is false (  )
• Power = 1 - 
• Operating characteristic curves plot  against a
parameter  where
a
n  2 i
2 = i =1
a 2
 Sample Size Determination
Fixed Effects Case---use of OC Curves
• The OC curves for the fixed effects model are in the
Appendix, Table V, pg. 613
• A very common way to use these charts is to define a
difference in two means D of interest, then the
minimum value of 2 is
2
nD
2 =
2a 2
• Typically work in term of the ratio of D /  and try
values of n until the desired power is achieved
• Minitab will perform power and sample size
calculations – see page 103
• There are some other methods discussed in the text
Power and Sample Size Calculations from Minitab (Page 103)
ANOVA Mechanics
ANOVA Mechanics
 Why ANOVA is one-tailed test
If Fo value is close to 1, we don’t want to reject (i.e., MSTreatments and MSE
are close to one another, hence null hypothesis is true).

If you get a large F you want to reject and if you get a small F you don't
want to reject. The large values of F are those values in the upper tail while
small values of F are those values in the lower tail. You only want to reject
when the values are large ... i.e., in the upper tail, but not the lower tail.
The rejection happens only when the F ratio is on the right side & never
when the F ratio is on the left side. The level of significance is the measure
of error due to statistical limitations. As the rejection happens only on the
right the entire level of significance (error risk of mis-conclusion) is kept in
the right. So that is why, you put all your significance on the upper tail.
How F-Distribution tells about Population Means
How F-Distribution tells about Population Means
How F-Distribution tells about Population Means