Scribd
Upload
63 views

Classification Example

The threshold is moved to favor classifying fish as salmon rather than sea bass. This reduces the riskier second type of error (classifying salmon as sea bass) at the cost of increasing the first type of error (classifying sea bass as salmon). Overall risk is minimized by tuning the decision boundary based on the different costs of the two error types.

Uploaded by

Faraz Bacha
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
63 views

Classification Example

The threshold is moved to favor classifying fish as salmon rather than sea bass. This reduces the riskier second type of error (classifying salmon as sea bass) at the cost of increasing the first type of error (classifying sea bass as salmon). Overall risk is minimized by tuning the decision boundary based on the different costs of the two error types.

Uploaded by

Faraz Bacha
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 12

Bayesian Decision Theory

Bayes Rule
We now pose the following question: Given that the event A has
occurred. What is the probability that any single one of the event
B’s occur?

i i i
i

This is known as the Bayes rule


Posterior Probability
Suppose, we know P(w1), P(w2), p(x|w1) and p(x|w2) and that we have observed
the value of the feature (a random variable) x
– How would you decide on the “state of nature”–type of fish, based on this
info?
– Bayes theory allows us to compute the posterior probabilities from prior and
class-conditional probabilities
Likelihood: The (class-conditional) probability of Prior Probability:The total
observing a feature value of x, given that the correct class is probability of correct class
wj. All things being equal, the category with higher class being class wj determined
conditional probability is more “likely”to be the correct based on prior experience
class.

Posterior Probability: The j


(conditional) probability of Evidence:The total
correct class being ùj, given probability of observing
that feature value x has been the feature value as x
observed
The sea bass/salmon example
State of nature: w1: Sea bass, w2: Salmon

Assume that we know the probabilities of observing sea bass and


salmons, P(w1) and P(w2) (prior probabilities), for a particular
location of fishing and time of year

Based on this information, how would you guess that the type of
the next fish to be caught?
• Decide w1 if P(w1) > P(w2) otherwise decide w2
A reasonable decision rule?

If the catch of salmon and sea bass is equi-probable


P(w1) = P(w2) (uniform priors)
P(w1) + P( w2) = 1 (exclusivity and exhaustivity)

This rule will not be applicable


Class Conditional Probabilities
p(x|w1) and p(x|w2) describe the difference in lightness between
populations of sea bass and salmon
Selection Criterion
Select the length of the fish as a possible feature for discrimination
Selection Criterion
The length is a poor feature alone!
Select the lightness as a possible feature.

Length or Lightness (weight), which one is a better feature?


No value of either feature will “classify” all fish correctly
Selection Criterion and Decision Boundary
Adopt the lightness and add the width of the fish
Fish xT = [x1, x2]

Lightness Width
Posterior Probabilities
Bayes rule allows us to compute the posterior probability (difficult to
determine) from prior probabilities, likelihood and the evidence (easier to
determine).

Posterior probabilities for


priors P(w1) = 2/3 and P(w2)
= 1/3. For example, given
that a pattern is measured
to have feature value x =14,
the probability it is in
category w2 is roughly 0.08,
and that it is in w1 is 0.92. At
every x, the posteriors sum
to 1.0.

Which class would you choose now?

What is probability of making an error with this decision?


Bayes Decision Rule
Decision given the posterior probabilities

X is an observation for which:


if P(w1 | x) > P(w2 | x) True state of nature = w1
if P(w1 | x) < P(w2 | x) True state of nature = w2

That is Choose the class that has the larger posterior probability !
If there are multiple features, x={x1, x2,…, xd} and multiple classes
Choose wi if P(wi| x) > P(wj| x) for all i = 1, 2, …, c

Error:
whenever we observe a particular x, the probability of error is :
P(error | x) = P(w1 | x) if we decide w2
P(error | x) = P(w2 | x) if we decide w1
Therefore:
P(error | x) = min [P(w1 | x), P(w2 | x)]
(Bayes decision)
Adjustment of decision boundary
A classifier, intuitively, is designed to minimize classification error,
the total number of instances (fish) classified incorrectly.

– Is this the best objective (cost) function to minimize? What kinds


of error can be made? Are they all equally bad? What is the real
cost of making an error?

• Sea bass misclassified as salmon: Pleasant surprise for the


consumer, tastier fish
• Salmon misclassified as sea bass: Customer upset, paid too
much for inferior fish

– We may want to adjust our decision boundary to minimize overall


risk –in this case, second type error is more costly, so we may
want to minimize this error.
Adjustment of decision boundary
What happens when the error is minimized by moving the
threshold to the left?

You might also like