Chapter 2

Probabilities
Outline

Outline of the chapter

1. Definitions
2. Frequentist approach
3. Elementary probability laws
4. Conditional probability
5. Independence
6. Bayes Theorem
7. Appendix: counting techniques

2
1. Definitions

 The calculus of probabilities is the mathematical base

necessary for the statistical analysis of data.
 The knowledge of probabilities will help us to take decisions in
the presence of random factors.

Example
What is the probability that a new satellite will function without problem for at
least ten years?
Once calculated, this probability will help us in deciding whether it’s worth
launching the latter.

3
1. Definitions

Before entering into the details of probalistic theory, let’s wonder at what one
would intuitively expect to hold for the calculus of probabilities.

The context: we face a

 Random Experiment, i.e. an operation for which the exact outcome is not known
beforehand with certitude even if one repeats the operation with the same
conditions (e.g. toss of a die).

With respect to this random experiment, we define the

 Sample space: the set of all possible results of the random experiment, denoted
either by S or Ω (e.g. Ω = {1, 2, 3, 4, 5, 6} for a toss of a die).
 Event: subset A of a sample space, A ⊂ S (e.g. ‘obtain either a 5 or a 6 from the
toss’: A = {5, 6}).

4
1. Definitions

Our goal is to evaluate the ‘likelihood’ of some events, and attribute a

numerical value to the latter on a (arbitrary) scale of 0 to 1.

For events A and B, it seems reasonable to

 Set a probability of 0 if the event is impossible, i.e. A = ∅ (e.g. ‘obtain a 7 from

the toss’).
 Set a probability of 1 if the event is certain, i.e. A = Ω (e.g. ‘obtain a result
smaller than 7’).
 If A and B cannot happen at the same time, the the event ‘A happens or B
happens’ should have a probability equal to the sum of the probabilities of A and
B (e.g. A=’obtain a 3’, B=’obtain a 6’).

5
1. Definitions

We may now formalize a bit in a set of axioms.

For every event of the sample space Ω we associate a number measuring the
degree of uncertainty with respect to the occurrence of the event. This
number is called the probability of the event A, denoted by P(A).

Axiomatic definition of probability (Kolmogorov, 1933)

 P(Ω) = 1
 For every event A, P(A) ≥ 0.
 If Ai for i ∈ N are mutually exclusive events (i.e. their intersection is
empty, Ai ∩ Aj = ∅, ∀i 6= j), then
X
P(∪i Ai ) = P(Ai ),
i

where ∪ denotes the ’union’.

6
1. Definitions

Remark
The theory of probability does not aim at strictly defining the notion of
randomness.
For instance, at a microscopic level, knowing all the external factors and
taking into account the laws of physics, is the result of the toss of a die still
truly random?

Instead, as every mathematical theory, the probabilistic theory only aims at

drawing conclusions one can obtain from a series of axioms that seem
relevant and appropriate.

7
2. Frequentist approach

 The axiomatic definition of probability lays the ground for probability

theory, but it doesn’t tell us how to evaluate probabilities of events nor
how to interpret these.
 The third axiom however already tells us that we can calculate the
probability of any event by summing the probabilities of the simple
events contained in it.
 Let ω1 , ω2 , . . . be the possible outcomes (countable) of a random
experiment, i.e. Ω = {ω1 , ω2 , . . .}. Let pi = P({ωi }) ≥ 0 such that
P P
i pi = 1. Then, P(A) = i:ωi ∈ A pi , where A ⊆ Ω.

Hence, determining a probability of an event requires the evaluation of the

probabilities of simple events.

8
2. Frequentist approach

So, what would we intuitively do in practice?

Equal probability case (Pascal, 1654)
Let us consider an experiment with |Ω| = N < ∞ possible outcomes which all
have the same chances of occurrence, i.e. P({ωi }) = 1/N, ∀i = 1, . . . , N.

The probability of an event A which may happen in n different ways is

n #favorable cases
P(A) = = .
N #all possible cases

9
2. Frequentist approach

Example
Two dice are rolled, one at time:
S = {(1, 1), (1, 2), . . . , (1, 6), (2, 1), (2, 2), . . . , (6, 6)}.

• The sum is equal to six: A = {(1, 5), (2, 4), (3, 3), (4, 2), (5, 1)}.

• The sum is pair: B = {(1, 1), (1, 3), (1, 5), . . . , (6, 2), (6, 4), (6, 6)}

Calculate P(A) and P(B).

5
P(A) = .
36
1
P(B) = .
2

10
2. Frequentist approach

 What is the (frequentist) interpretation of a probability?

 What does it mean to say “the probability of obtaining an even
number from the toss of a die is 1/2”?

A simple way to interpret is by saying that the probability of an event

is the relative frequency to which the event would happen if we
repeated, in the same conditions, our random experiment a very big
number of times.

In other words, if after N repetitions of the experiment, where N is

very big, the event A has been observed n times, then

n(N)
P(A) = lim .
N→∞ N

11
2. Frequentist approach

Illustration of frequentist interpretation

We throw a coin 100 times. Every time we recalculate the frequency :
#number of heads
#number of throws

12
2. Frequentist approach

Combinatorial analysis To evaluate empirically the probability of an

event A using a frequentist approach, one uses
n #cases favorable for A
P(A) = = .
N #possible cases
Hence

 To find N and n, one may list all the outcomes.

 Sometimes this is almost impossible due to the huge number of
outcomes.
 In this case one may use counting techniques to find N and n
and avoid listing all the outcomes.

A more detailed summary of classical counting techniques is

presented at the end of this chapter.

13
3. Properties of a probability function

Proposition
Following the axiomatic definition of a probability function:

1. P(∅) = 0.
2. P(A) = 1 − P(A), where A = Ω\A.
3. A ⊆ B : P(A) ≤ P(B).
4. P(A ∪ B) = P(A) + P(B) − P(A ∩ B) (Additive Law).
5. P(A ∪ B) = P(A ∩ B) and P(A ∩ B) = P(A ∪ B).

14
3. Properties of a probability function

Definition
If the events B1 , . . . ..., Bn are such that

 Bi ∩ Bj = ∅, ∀i 6= j,
Sn
 i=1 Bi = Ω,

then, B1 , . . . ..., Bn is called a partition of Ω.

Proposition
Law of total probabilities
If B1 , . . . ..., Bn is a partition of Ω, then
n
X
P(A) = P(A ∩ Bi ).
i=1

15
3. Properties of a probability function

Illustration 1
One card is taken from a shuffled deck of 52 cards. What is the
probability that this card will be either a king or an ace (either an ace
or a heart)?

♣ : A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K
♦ : A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K
♥ : A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K
♠ : A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K

Calculate P(A ∪ K ) and P(A ∪ ♥)

8
P(A ∪ K ) = = P(A) + P(K )
52
16
P(A ∪ ♥) = = P(A) + P(♥) − P(A ∩ ♥)
52

16
3. Properties of a probability function

Illustration 2
Let A and B be two events in the sample space Ω. One knows that
P(A) = 0.45, P(B) = 0.6 and P(A ∩ B) = 0.3. Calculate P(A ∪ B),
P(A ∩ B) and P((A ∩ B) ∪ (A ∩ B)).

•
P(A ∪ B) = 0.7

•
P(A ∩ B) = 0.25

•
P((A ∩ B) ∪ (A ∩ B)) = 0.45

17
4. Conditional probability

Motivation

What is the impact of the occurrence of an event on the probability of

occurrence of another event?

Example: one card is taken from a non-standard shuffled deck of 52 cards.

What is the probability that this card will be an ace if one knows that is a
heart?

♣ : A, A, A, A, 2, 3, 4, 5, 6, 7, 8, 9, 10
♦ : A, A, A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J
♥ : A, A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q
♠ : A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K

The information on the first card drawn transforms the sample space to:
♥ : A, A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q

18
4. Conditional probability

♥ : A, A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q

P(select an ace IF a heart was selected)

=: P(A|♥)
2 10
= 6= P(A) =
13 52

Definition
Let B ⊆ Ω be an event such that P(B) > 0. Then, for every A ⊆ Ω, the
conditional probability of A given that the event B happened, denoted by
P(A|B), is defined by
P(A ∩ B)
P(A|B) = .
P(B)

19
4. Conditional probability

P(A ∩ ♥) 2/52 2
P(A|♥) = = = .
P(♥) 13/52 13

Remark
 P(B|B) = 1.
 If B ⊂ A, A ∩ B = B and hence P(A|B) = 1.
 One may verify that the conditional probability P(·|B) verifies the
axioms of a probability function.
20
4. Conditional probability

Multiplicative law
As a consequence of the definition of conditional probabilities, we have

P(A ∩ B) = P(A|B)P(B),

P(A ∩ B) = P(B|A)P(A).
Example
From a well shuffled deck of 52 cards two cards are drawn successively
(without replacement). What is the probability that the second card will be an
ace?

21
4. Conditional probability

Example continued
A1 =“The first card is an ace".
A2 =“The second card is an ace".

P(A2 ) = P(A2 ∩ A1 ) + P(A2 ∩ A1 )

= P(A2 |A1 )P(A1 ) + P(A2 |A1 )P(A1 )
3 4 4 48
= × + ×
51 52 51 52
4 1
= = .
52 13
And if the first card is replaced back into the deck?

22
5. Independence

What happens if the occurrence of event B does not influence the probability
of A happening?

Definition
Two events A and B in Ω are independent, denoted as A ⊥
⊥ B, iff

P(A ∩ B) = P(A)P(B).

Note that, if P(B) > 0, this is equivalent to P(A|B) = P(A), and similarly when
inverting A and B.
Remark

 Independence: the occurrence of event B does not influence the

probability of event A occurring.
 Mutual exclusive: the occurrence of event B prevents A from occurring.

23
5. Independence

How do we know that two events are independent?

• Sometimes the independence follows directly from the description of the

random experiment.
For example: A coin is tossed two times

 A=“the first toss results in a head".

 B =“the second toss results in a head".

The fact of obtaining head or tail in the first toss does not influence the
fact of obtaining head or tail in the second toss, so the events A and B
are independent.

• Sometimes we need to calculate the probabilities explicitly.

24
5. Independence

Example
We throw a die. Let us define the events:

 A=“the result is less than 5" = {1, 2, 3, 4}.

 B=“the result is even" = {2, 4, 6}.

Are A and B independent?

P(A) = 32 , P(B) = 1
2
and P(A ∩ B) = 13 .

1/3 1
⇒ P(B|A) = = = P(B)
2/3 2

So A and B are independent.

Another way of showing the independence is to verify that in this case
P(A ∩ B) = P(A)P(B).

25
6. Bayes Theorem

Exercise
A binary communication channel can transmit messages using two signals: 0
and 1.
One knows that: in 40% of cases 1 is transmitted and the probability that the
transmission of 1 (0) will be correctly interpreted is 0.95 (0.9). Find:

 The probability that 1 will be received.

 The probability that 1 was transmitted, given that 1 was received.

First we define the events:

1E=“1 transmitted", 0E=“0 transmitted"

1R=“1 received", 0R=“0 received"

26
6. Bayes Theorem

P(1R) = P(1R ∩ 1E) + P(1R ∩ 0E) = 0.44

P(1R ∩ 1E)
P(1E|1R) = ≈ 0.864
P(1R)

27
6. Bayes Theorem

Second example
Vous venez de passer un test pour le dépistage du cancer. Le médecin vous
convoque pour vous annoncer le résultat : mauvaise nouvelle, il est positif.
Pas de chance, alors que ce type de cancer ne touche que 0.1% de la
population.
Vous demandez alors au praticien si le test est fiable. Sa réponse est sans
appel : “ Si vous avez le cancer, le test sera positif dans 90% des cas ; alors
que si vous ne l’avez pas, il sera négatif dans 97% des cas ”.
Is it panicking time?

28
6. Bayes Theorem

Proposition
Let A and B be two events in Ω such that P(A) > 0 and P(B) > 0. We then
have
P(A|B)P(B)
P(B|A) = .
P(A|B)P(B) + P(A|B)P(B)

Proof
P(A ∩ B)
P(B|A) =
P(A)
P(A|B)P(B)
=
P(A ∩ B) + P(A ∩ B)
P(A|B)P(B)
= .
P(A|B)P(B) + P(A|B)P(B)

In other words, Bayes’ formula allows us to get from P(B|A) to P(A|B).

29
6. Bayes Theorem

More generally, if B1 , . . . , Bn is a partition of Ω with P(Bi ) > 0 ∀i = 1, . . . , n,

then, for every event A ⊆ Ω such that P(A) > 0,

P(Bi ∩ A) P(A|Bi )P(Bi )

P(Bi |A) = = Pn
i=1 P(A|Bi )P(Bi )
P(A)

30
Application

A barometer is used to forecast the weather. However the prediction

may be erroneous. It is observed that in 20 cases over 200 rainy days
the barometer has predicted good weather, and in 20 cases over 100
good days it has predicted rain. The local tourist guide says that 10%
of the days in a year are rainy. What is the probability that it will rain if
the barometer predicts rain?

Answer : ≈ 0.333 ( verify !)

31
Appendix:
Counting techniques (WMS Ch. 2.6)

 Permutations

 Arrangements

 Combinations

32
Appendix: Example 1

One tosses a die three times. We denote the obtained result by

(X , Y , Z ).

X = 1, 2, 3, 4, 5, 6.
Y = 1, 2, 3, 4, 5, 6.
Z = 1, 2, 3, 4, 5, 6.

We define the event

A = “X ≤ 4 and Y ≤ 4 and Z ≤ 4 and X 6= Y 6= Z ”

Find P(A).

33
Appendix: Ordered or non-ordered

It is necessary to distinguish two cases:

• Order is important: (3, 2, 1) 6= (1, 2, 3).

• Order is not important: (3, 2, 1) = (1, 2, 3).

The first case corresponds to an arrangement and the second case

to a combination.

34
Appendix: Permutation

Definition
A permutation of n different objects is an ordered sequence of all
these n objects.
There are 4 objects A, B, C, D, the possible permutations are:

In total there 4 × 3 × 2 × 1 = 24 possibilities.

In general this number is equal to

n × (n − 1) × (n − 2) × . . . × 2 × 1

35
Appendix: Factorial of n

n! = n × (n − 1) × (n − 2) × . . . × 2 × 1

5! = 5 × 4 × 3 × 2 × 1
= 5 × 4!
= 5 × 24 = 120.

• n! = n × (n − 1)!

• 0! = 1

5! 5 × 4 × 3!
= = 20.
3! 3!

36
Appendix: Arrangements

Definition
An ordered sequence of r objects chosen among n different objects
is called an arrangement of r objects from n.
The possible arrangements of 2 objects from A, B, C, D, are the
following:

In total there are 4 × 3 = 12 possibilities.

In general this number is equal to

n × (n − 1) × (n − 2) × . . . × (n − r + 1)

37
Appendix: Arrangements (in WMS called Prn )

Arn = n × (n − 1) × (n − 2) × . . . × (n − r + 1)
n!
=
(n − r )!
Remarks

• Ann = n!

• If the repetitions are allowed (experiment with replacement) then

the number of possible arrangements is equal to nr .

38
Appendix: Combinations

Definition
The set of r objects taken from the set of n different objects without
paying attention to the order is called a combination of r objects
from n.
The possible combinations of 2 objects among A, B, C, D, are the
following:

In total there are 6 possibilities.

In general this number is equal to
Arn n!
Cnr = = (= Crn in WMS !)
r! r !(n − r )!

39
Appendix: Example 1 continued

The die is thrown three times. (X , Y , Z ) and X , Y , Z = 1, 2, 3, 4, 5, 6.

A = “X ≤ 4 and Y ≤ 4 and Z ≤ 4 and X 6= Y 6= Z ”

Find P(A)

With order
1
P(A) =
9
Without order
4
P(A) =
56

40
Appendix: Example 2

To do an experiment in the lab a researcher should choose three

electric conductors. There are six conductors of type A and four
conductors of type B. Find the probability that all the three chosen
conductors will be of type A given that the choice is random.

1
P(3 × A) = .
6

41
Appendix: Example 3

18 persons have donated their blood. Among them there are 11

persons with blood type O, 4 persons with A, 2 persons with B and 1
person with AB. After the collection 3 containers have been chosen at
random. Find the probability of the following events:

a. the blood types in the 3 containers are the same,

b. at least one of the 3 containers contains the blood of type A,

c. the blood types in the 3 containers are pairwise different.

42
Appendix: Example 3 continued

a. the blood types in the 3 containers are the same : (O,O,O) or

(A,A,A)
3
C11 + C43
3
≈ 0.207
C18

b. at least one of the 3 containers contains the blood of type A:

(A,A,A) or (A,A,X) or (A,X,X)

C43 + C42 C14

1
+ C41 C14
2

3
≈ 0.554
C18

43
Appendix: Example 3 continued

c. the blood types in the 3 containers are pairwise different: (O,A,B)

or (O,A,AB) or (O,B,AB) or (A,B,AB)

11 × 4 × 2 + 11 × 4 × 1 + 11 × 2 × 1 + 4 × 2 × 1
3
C18
≈ 0.199

44