Hypothesis Testing

Hypothesis
•
Null and Alternate Hypotheses
•
 α: What we set before a

Significance Level (α) and p-value statistical test

p-value: What we calculate
after the test

• α = Significance level = Threshold or acceptable probability of rejecting the

null hypothesis when it is actually true
• Usually set to 5% (or 0.05)
• Meaning: We are okay with a 5% chance of rejecting the null hypothesis
incorrectly
• Why not keep it at 0%? => We would never reject H0!
• p-value = Result of an actual test (Area of rejection)
• Low p-value: Reject H0, High p-value: Fail to reject H0
• So, we may start with α = 0.05, run a test, and calculate p-value
• If p-value = 0.02, Reject H0
• If p-value = 0.10, Do not reject H0 If p-value <= α … Reject H0

If p-value > α … Do not reject H0

Significance Level (α) and p-value
• Example: Tossing a coin with H0 = Coin is fair and H1: Coin is not fair
• We toss it 5 times and each time we get a head! (See next slide for a bar plot)
Toss Number Result Current Probability formula Theoretical Probability
Situation (p-value)
1 H H 0.50 (50%)

2 H HH 0.25 (25%)

3 H HHH 0.125 (12.50%)

4 H HHHH 0.06 (6%)

5 H HHHHH 0.03 (3%)

• At what point we conclude that the coin is not fair and reject H0?
• That is our significance level (α), usually set at 0.05 (i.e. 5%) or 0.10 (0r 10%) – Our Lakshman
Rekha! Our p-value must be > α for us not to reject H0
• If we have set it at 5%, if we get a heads a fifth time, the p-value < α and we reject H0
Visualizing p-value
Significance Level (α) – More Details
• α: The probability of rejecting the null hypothesis even though it is
true (Type I Error)
• Why? What if the coin was indeed fair and showed HHHHH? – Very
rare but possible!
• We ended up rejecting H0 when we should not have! => Type I error
• Another Example: A course head thinks average student score is 70%
• H0: Population mean score (μ) = 70%, H1: Population mean score (μ) ≠
70%
• So, scores distribution should look like this
Significance Level
•
 Critical values define the
boundaries of the non-rejection
Significance Level zone for Z-scores … If our
Z-score crosses this zone on
either side, we reject H0 …

• If Z-score is in the middle part, we do not reject H0

• But if it goes beyond the dashed red lines, we reject H0 – Why?
• Those are rejection regions (Critical zone)
• Area of rejection regions depends on
significance level (alpha)
• If alpha = 0.05, we have 0.0250 on both sides
• From Z-table, when alpha = 0.0250, Non-Rejectio
n zone
critical value = 1.96 on RHS and -1.96 on LHS
• So, if Z-score < -1.96 or > 1.96, reject H0
Rejection Rejection
zone zone
Significance Level and Confidence Level

Confidence level = 90% Confidence level = 95% Confidence level = 99%

Significance level (α) = 10% Significance level (α) = 5% Significance level (α) = 1%
Critical Z-Value = ±1.64 Critical Z-Value = ±1.96 Critical Z-Value = ±2.58

• As the confidence level increases, Critical Z-value increases

• Range of acceptance level increases
How to Get Critical Value from Z-Table
• Look for our probability value
• That is 0.025
• Critical value = Row + Column
• Critical value = 1.9 + .06
• Critical value = 1.96

• Note: If we look in the positive Z-table (the second table on the same page,
we will need to look for 1 – 0.025, i.e. 0.975 and we will get the same
result, i.e. 1.96)
Type I, II Errors, Alpha Beta
H0 is True H0 is False
Reject H0 Type I Error, Alpha Correct
Do not Reject H0 Correct Type II Error, Beta
• Why can this happen? Bad sample data …
• Example: We got a bad sample where Average sample score = 80
• We would reject H0 because in our sample it seemed so, although in
general it is true, considering all the other samples
• Note: Try to reduce Type I errors => Try not to reject H0 (See table above)
• But it automatically means a scope for increase in Type II errors (See table
above)
 According to all the
other samples, we will
Visualizing Type I Errors Meaning: If by
chance, the only
fail to reject H0, which is
correct
sample we tested
was x̄5, we would
commit Type I
error

Type I error
 According to all the
other samples, we will
Similarly, Type II Error Now reject H0, which is
correct

Meaning: If by
chance, the only
sample we tested
was x̄4, we would
commit Type II
error
Type I and Type II Errors – Example
• H0: John’s used car is safe to drive
• Possibilities
a) John thinks that his car may be safe when in fact it is not safe - Error
b) John thinks that his car may be safe when in fact it is safe - Ok
c) John thinks that his car may not be safe when in fact it is not safe - Ok
d) John thinks that his car may be not be safe when in fact it is safe – Error
• Which of these is Type I and which is Type II Error?
Type I and Type II Errors – Example
• H0: John’s used car is safe to drive • Remember:
• Possibilities • Type I Error occurs when H0 is true
a) John thinks that his car may be safe • Type II Error occurs when H0 is false
when in fact it is not safe - Error • In this example: H0: John’s used car
b) John thinks that his car may be safe is safe to drive
when in fact it is safe - Ok
c) John thinks that his car may not be • So:
safe when in fact it is not safe - Ok • (d) is Type I error
d) John thinks that his car may be not • (a) is Type II error
be safe when in fact it is safe – • Which error is more serious?
Error
• Type II
• Which of these is Type I and which
is Type II Error?
Type I and Type II Errors – Another Example
• In a criminal court case, H0: The defendant is innocent
• Possibilities:
a. The jury believes that the defendant is guilty when she is innocent - Error
b. The jury believes that the defendant is guilty when she is not innocent - Ok
c. The jury believes that the defendant is innocent when she is not innocent -
Error
d. The jury believes that the defendant is innocent when she is innocent – Ok

• What are Type I and Type II errors here, and which one is more
dangerous?
Type I and Type II Errors – Another Example
• In a criminal court case, H0: The defendant • Remember:
is innocent • Type I Error occurs when H0 is true
• Possibilities: • Type II Error occurs when H0 is false
a. The jury believes that the defendant is not • In this example: H0: Defendant is innocent
innocent when she is innocent - Error • (c) Type II error
b. The jury believes that the defendant is not • (a) Type I error
innocent when she is not innocent - Ok
c. The jury believes that the defendant is • Which one is more serious?
innocent when she is not innocent - Error • Perhaps (c), but then someone can also
d. The jury believes that the defendant is argue it is (a), depending on the situation
innocent when she is innocent – Ok and the crime
• What are Type I and Type II errors here,
and which one is more dangerous?
One-Tailed and Two-Tailed Tests
• Two-tailed test: Rejection zone at both sides
• One-tailed test: Rejection zone only in one side
• Example: A new medicine is being tested with current lifespan = 70 years
• Two-tailed Test
• H0: Lifespan = 70
• H1: Lifespan ≠ 70
• One-tailed Test
• H0: Lifespan = 70
• H1: Lifespan > 70
• C:\code\Data Analytics\One_tailed_Two_tailed.py,
one_tailed_two_tailed_tips.py
Two-Tailed and One-Tailed Tests
(Visualization: Next slide)
• Two-tailed Test • One-Tailed Test (Left-Tailed)
• H0: The average weight • H₀: The average weight of chips in a packet is
of chips in a packet is equal to or greater than 50 grams (μ ≥ 50)
equal to 50 grams (μ = • Ha: The average weight of chips in a packet is
50) less than 50 grams (μ < 50)
• Ha: The average weight • Rejection zone: Towards the left side of 50
of chips in a packet is not
equal to 50 grams (μ ≠ • One-Tailed Test (Right-Tailed):
50) • H₀: The average weight of chips in a packet is
• Rejection zone: Both to equal to or less than 50 grams (μ ≤ 50)
the left and to the right • Ha: The average weight of chips in a packet is
of 50 greater than 50 grams (μ > 50)
• Rejection zone: Towards the right side of 50
Two-Tailed and One-Tailed Tests
Rejection/Non-Rejection Criteria
Test Type α p-value
Two-tailed Consider α/2 to find critical value Compare p-value with α/2
One-tailed Consider α to find critical value Compare p-value with α

Test Type Condition Decision

Two-tailed |Test-statistic| > |Critical Value| Reject H0
|Test-statistic| <= |Critical Value| Do not reject H0
Left-tailed Test-statistic < Critical value Reject H0
Test-statistic >= Critical value Do not reject H0
Right-tailed Test-statistic > Critical value Reject H0
Test-statistic <= Critical value Do not reject H0
Calculate Test Statistic
•
P-value Calculation Example
• We have Z-Score = 2.82 for a two-tailed test
For Z-Score = 2.82, using
Z-table, area under the
curve = 0.9976

So, area to the right of Z

= 2.82 = 1 - 0.9976 =
0.0024

Because it is a two-tailed
test, P-value = 2 x 0.0024
= 0.0048

Since P-value < α, we do

not reject H0
P-value
• P-value = Area of rejection
• Low p-value: Reject H0, High p-value: Fail to reject H0
• Example: A pizza outlet claims they deliver pizza in <= 30 minutes on
average
• H0: Mean delivery time = 30 minutes
• Ha: Mean delivery time <> 30 minutes
• We test a few sample delivery times and get a p-value = 0.001
• Since 0.001 is much less than 0.05, we reject H0
Critical Values for Different Levels of α
Approximate Critical Values
α Left-Tailed Test Right-Tailed Test Two-Tailed Test
0.1 -1.28 +1.28 -1.64 and + 1.64
0.05 -1.64 +1.64 -1.96 and + 1.96
0.01 -2.33 +2.33 -2.58 and + 2.58
Margin of Error
•
Confidence Interval * Calculate MoE and CI as shown on
the previous slide, for each sample
Confidence Level
• Interval Estimate = 114.4 to 147.2
• Confidence interval = Calculate for H0: Average number of friends for a user = 114.4
to 147.2
each sample*
• Confidence level = Confidence Sample Sample
…
Sample Now if we take
about the confidence interval 1 2 100 100 more
samples
• 95% confidence level means: If we
take 100 samples, calculate sample Calculate CI for
mean, calculate sample CI* around each sample
the sample mean, 95% of such mean
sample CIs would contain the We expect 95%
true/hypothesized value of the of these CIs to
population mean (μ) contain μ (true
✔ ✔ X or hypothesized)
 Null Hypothesis – How to State?
Test Null Hypothesis
One-sample T-Test There is no significant difference between the sample mean
and the hypothesized population mean
Two-sample T-Test The population means of the two groups are equal
Paired T-Test The means before and after of the same group are equal
ANOVA The means of more than two groups are equal
Chi-Square Test There is no association between the two categorical variables
(they are independent)
Linear regression The coefficients of the independent variables are equal to zero
(no relationship)
Logistic regression There is no association between the independent variables and
the binary outcome
t-tests
• t-test: Also called student’s t-test
• Determines if there is a significant difference between the means of
two samples
• Difference from Z-test: Here, the sample size is expected to be <= 30
Types
•Type of the t-test Purpose
One sample t-test Compare the mean of a sample with that of the
population
Independent sample Compare the mean of two different populations
t-test
Paired sample t-test Compare the mean of the same group at different
times
One-Sample T-test
One-Sample T-test
•
One-Sample T-test : Example
•
One-Sample T-test : Example
• T-table: At
https://www.sjsu.edu/faculty/gerstman/StatPrimer/t-table.pdf, look
for DF 19 and two-tails value of 0.05
• Why 0.05? Because confidence interval is 95%, so significance level (alpha) is
0.05
• Calculated t-statistic (2.08) < Critical t-value (2.093)
• Conclusion: Fail to reject the null hypothesis
• Conclusion: We do not have enough evidence to conclude that the
average amount of coffee in a medium-sized cup served by the
company is different from 12 units