Ambo University

Institute of Technology
Department of Electrical & Computer Engineering

Probability and Statistics

Chapter 1: Basic Concepts of Probability Theory

 Basic Concepts of Probability Theory
Outline
 Introduction
 Sample Space and Events
 Basic Set Operations
 Axioms and Properties of Probability
 Conditional Probability
 Independence of Events

2
Introduction
 Events can in general be classified as:
• Random-the event may or may not occur
• Certain- the occurrence of the event is inevitable (Incapable of being
avoided or prevented)
• Impossible-the event will never occur

 Probability is the study of randomness and uncertainty.

 Probability theory provides powerful tools to explain, model
and design real world physical systems with some degree of
uncertainty.
 Some application areas of probability theory include:
• Data communication systems
• Wireless communication systems
3
• Control systems, etc…
Sample Space and Events
i. Random Experiment
 A random experiment is an experiment in which the outcome
varies in an unpredictable manner when the experiment is
repeated under the same conditions.
Examples:
• Tossing a coin
• Rolling a die

ii. Sample Space

 The sample space is the set of all possible outcomes of a
random experiment.

4
Sample Space and Events Cont’d….

 The sample space is denoted by Ω and the possible outcomes

are represented by  i
  {1 , 2 , ......n }
iii. Event
 An event is any subset of the sample of the sample space, Ω
 Events can be represented by A, B, C, ……
Example-1:
Consider a random experiment of rolling a die once.
i. Sample Space
  {1, 2, 3, 4, 5, 6}

5
Sample Space and Events Cont’d….
ii. Some possible events
 An event of obtaining even numbers
A  {2, 4, 6}

 An event of obtaining numbers less than 4

B  {1, 2, 3}

Example-2:
Consider a random experiment of flipping a fair coin twice.
i. Sample space
  {HH , HT, TH, TT }

6
Sample Space and Events Cont’d….

ii. Some possible events

 An event of getting exactly one head
A  {HT , TH }

 An event of getting at least one tail

B  {HT , TH, TT }

 An event of getting at most one tail

C  {HH , HT , TH }

7
 Basic Set Operations
 We can combine events using set operations to obtain other
events.
1. Union
The union of two events A and B is defined as the set of outcomes
that are either in A or B or both and is denoted by A  B.

A  B  { :   A or   B}  A  B  { :   A    B}
Ω

A B

E F

AB

8
 Basic Set Operations Cont’d…..
2. Intersection
The intersection of two events A and B is defined as the set of
outcomes that are common to both A and B and is denoted by
A  B.

A  B  { :   A and   B}  A  B  { :   A    B}

A B

AB

9
Basic Set Operations Cont’d…..
3. Complement
The complement of an event A is defined as the set of all
outcomes that are not in A and is denoted by A.

A  { :    and   A}  A  { :       A}
Ω

EA A

10
 Basic Set Operations Cont’d…..
4. Mutually Exclusive (Disjoint) Events
Two events A and B are said to be mutually exclusive or disjoint
if A and B have no elements in common, i.e., A B  

A B

A B  

5. Equal Events
Two events A and B are said to equal if they contain the same
outcomes and is denoted by A=B.

11
 Some Properties of Set Operations
1. Elementary Properties
i.    v. A  A  
ii.    vi. A  A  
iii.   A   vii. A  A
iv.   A  A
2. Commutative Properties
A B  B  A
A B  B  A
3. Associative Properties
A  ( B  C )  ( A  B)  C
A  ( B  C )  ( A  B)  C

12
Some Properties of Set Operations Cont’d…..
4. Distributive Properties
A  ( B  C )  ( A  B)  ( A  C )
A  ( B  C )  ( A  B)  ( A  C )
5. DeMorgan’s Rules
( A  B)  A  B
( A  B)  A  B

 The union and intersection operations can be repeated for an

arbitrary number of events as follows.
n

A i  A1  A2  ....  An
i 1
n

A i  A1  A2  ...  An
i 1

13
Axioms and Properties of Probability
 Probability is a rule that assigns a number to each event A in the
sample space, Ω.
 In short , the probability of any event A is given by
n( A)
P( A) 
n ( )
where
n( A) - is the number of elementsin the event A
n() - is the number of elementsin the sample space 

14
 Axioms and Properties of Probability Cont’d…..

 The probability of an event A is a real number which satisfies the

following axioms.
1. Probability is a non-negative number, i.e.,
P( A)  0
2. Probability of the whole set is unity, i.e.,
P ()  1
From axioms (1) and (2), we obtain
0  P( A)  1

15
Axioms and Properties of Probability Cont’d…..

3. Probability of the union of two mutually exclusive (disjoint)

events is the sum of the probability of the events, i.e.,
If A  B  , then P ( A  B )  P ( A)  P ( B )

 We can generalize axiom (3) for n pairwise mutually exclusive

(disjoint) events.
 If A1, A2, A3, …, An is a sequence of n pairwise mutually
exclusive (disjoint) events in the sample space Ω such that
Ai  Aj  , for i  j, then

 n  n
P  Ai    P( Ai )
 i 1  i 1
16
Axioms and Properties of Probability Cont’d…..
 By using the above probability axioms, other useful
properties of probability can be obtained.

1. P( A)  1  P( A)
Proof:

A  A    P( A  A)  P( A)  P( A), but A  A  
 P()  P( A)  P( A), P()  P( A  A)
 1  P( A)  P( A), P()  1
 P( A)  1  P( A)

17
Axioms and Properties of Probability Cont’d…..
 We can decompose the events A, B and AUB as unions of
mutually exclusive (disjoint) events as follows.

A B

A B A B A B

We can see that A  B, A  B and A  B are disjoint events.

18
Axioms and Properties of Probability Cont’d…..

 From the above Venn diagram, we can write the following

relations.
i. A  ( A  B)  ( A  B) ii. B  ( A  B)  ( A  B)

 P( A)  P( A  B)  P( A  B)  P( B)  P( A  B)  P( A  B)
 P( A  B)  P( B)  P( A  B)
 P( A  B)  P( A)  P( A  B)

iv. A  B  B  ( A  B)
iii. A  B  A  ( A  B)
 P( A  B)  P( B)  P( A  B)
 P ( A  B )  P ( A)  P ( A  B )

v. A  B  ( A  B)  ( A  B)  ( A  B)
 P( A  B)  P( A  B)  P( A  B)  P( A  B)

19
Axioms and Properties of Probability Cont’d…..

2. P ( A  B )  P ( A)  P ( B )  P ( A  B )

Proof:
P( A  B)  P( A)  P( A  B)
But, P( A  B)  P( B)  P( A  B)
 P( A  B)  P( A)  P( B)  P( A  B)

 We can generalize the above property for three events A, B and C

as follows.
P( A  B  C )  P( A)  P( B)  P(C )  P( A  B)  P( A  C )  P( B  C )
 P( A  B  C )

20
Axioms and Properties of Probability Cont’d…..
3. P( A  B)  P ( A)  P( B)

Proof:
P( A  B)  P( A)  P( B)  P( A  B)
But, P( A  B)  0
 P( A  B)  P( A)  P( B)

21
Axioms and Properties of Probability Cont’d…..
Example-1:
A box contains 10 identical balls numbered 0, 1, 2,…,9. A single
ball is selected from the box at random. Consider the following
events.
A: number of ball selected is odd
B: number of ball selected is multiple of 3
C: number of ball selected is less than 5
Find the following probabilities.
a. P( A) d. P( A  B)
b. P( B) e. P( A  B  C )
c. P(C )
22
Axioms and Properties of Probability Cont’d…..
Solution:
 The sample space and the events are given by:

  {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} C  {0, 1, 2, 3, 4}
A  {1, 3, 5, 7, 9} A  B  {3, 9}
B  {3, 6, 9} A  B  C  {0, 1, 2, 3, 4, 5, 6, 7, 9}

 The number of elements in the sample space and events are:

n()  10 n(C )  5
n( A)  5 n( A  B )  2
n( B )  3 n( A  B  C )  9

23
Axioms and Properties of Probability Cont’d…..
 Thus, probabilities of the given events are given by:
n( A) 5 1 n( A  B ) 2 1
a. P( A)    d . P( A  B)   
n() 10 2 n ( ) 10 5
n( B ) 3 n( A  B  C ) 9
b. P( B)   e. P( A  B  C )  
n() 10 n () 10
n(C ) 5 1
c. P(C )   
n() 10 2
Example-2:
Given P( A)  0.9, P( B)  0.8 and P( A  B)  0.75, find :
a. P( A  B) c. P( A  B) e. P( A  B)
b. P( A  B) d . P( A  B) f . P( B)

24
Axioms and Properties of Probability Cont’d…..
Solution:
a. P( A  B)  P( A)  P( B)  P( A  B) d . P( A  B)  P( A  B)  1  P( A  B)
 P( A  B)  0.9  0.8  0.75  P( A  B)  1  0.75
 P( A  B)  0.95  P( A  B)  0.25

b. P( A  B)  P( A)  P( A  B) e. P( A  B)  1  P( A)  P( A  B)
 P( A  B)  0.9  0.75  P( A  B)  1  0.9  0.75
 P( A  B)  0.15  P( A  B)  0.85

c. P( A  B)  P( A  B)  1  P( A  B) f . P( B)  1  P( B)
 P( A  B)  1  0.95  P( B)  1  0.8
 P( A  B)  0.05  P( B)  0.2

25
Conditional Probability
 The conditional probability of an event A given B, denoted by
P(A/B), is defined as:
P( A  B)
P( A / B)  , P( B)  0 (1)
P( B)

 Similarly, the conditional probability of an event B given A,

denoted by P(B/A), is given by
P( A  B)
P( B / A)  , P( A)  0 (2)
P( A)

 From equations (1) and (2), we will get

P( A  B )  P( A / B) P( B)  P ( B / A) P( A) (3)

26
Conditional Probability Cont’d…….
 Then using equation (3), we will get

P( B / A) P( A) P( A / B) P( B)
P( A / B)  OR P( B / A)  (4)
P( B) P( A)

 We know that
P( B)  P( A  B)  P( A  B)
 P( B)  P( B / A) P( A)  P( B / A) P( A) (5)

 Substituting equation (5) into equation (4), we will get

P( B / A) P( A)
P( A / B)  (6)
P( B / A) P( A)  P( B / A) P( A)

27
Conditional Probability Cont’d…….
 Similarly,
P( A / B) P( B)
P( B / A)  (7)
P( A / B) P ( B )  P( A / B) P( B)

 Equations (6) and (7) are known as Baye’s Rule.

 Baye’s Rule can be extended for n events as follows.
 Let events A1, A2, A3, …, An be pairwise mutually exclusive
(disjoint ) events and their union be the sample space Ω, i.e.
n
Ai  A j   and A  i
i 1

 n  n
 P  Ai    P( Ai )
 i 1  i 1
28
Conditional Probability Cont’d…….
 Let B be any event in Ω as shown below.

A2 .....
A1 An 1

B
A3 ..... An

B  B  ( A1  A2  ....  An )

 B  ( B  A1 )  ( B  A2 )  ...  ( B  An )

But, Ai  A j    ( B  Ai )  ( B  A j )  

29
Conditional Probability Cont’d…….
 The events B  Ai and B  Aj are mutually exclusive events.
 P( B)  P( B  A1 )  P( B  A2 )  ...  P( B  An )

 P( B)  P( B / A1 ) P( A1 )  P( B / A2 ) P( A2 )  ...  P( B / An ) P( An ) (8)
 In short,
n n
P( B)   P( B  Ai )   P( B / Ai ) P( Ai ) (9)
i 1 i 1

 Then using equation (4), we will obtain

P( B / Ai ) P( Ai )
P( Ai / B)  n
(10)
 P( B / A ) P( A )
i 1
i i

30
Conditional Probability Cont’d…..
Example-1:
Show that P( A / B)  1  P( A / B)

Solution:
P( B)  P( A  B)  P( A  B)

 P( B)  P( A / B) P( B)  P( A / B) P( B)
Dividing both sides by P( B), we obtain

1  P( A / B)  P( A/B)

 P( A/B)  1  P( A / B)

31
Conditional Probability Cont’d…..
Example-2:
Let A and B be two events such that P(A)=x, P(B)=y and
P(B/A)=z. Find the following probabilities in terms of x, y and z.
a. P ( A / B )
b. P ( A  B )
c. P ( A / B )
Solution:
P( A  B)  P( B / A) P( A)  xz
P( A  B) xz
a. P( A / B)  
P( B) y
b. P( A  B)  P( A  B)  1  P( A  B)  1  xz
P( A  B) P( B)  P( A  B) xz
c. P( A / B)    1
P( B) P( B) y
32
Conditional Probability Cont’d…..
Example-3:

A box contains two black and three white balls. Two balls are
selected at random from the box without replacement. Find the
probability that

a. both balls are black

b. the second ball is white

Solution:

First let us define the events as follows:

B1 : the outcome in the first selectionis a black ball
33
Conditional Probability Cont’d…..

B2 : the outcome in the second selectionis a black ball

W1 : the outcome in the first selectionis a white ball
W2 : the outcome in the second selectionis a white ball
P( B1 )  2 / 5 P( B2 / B1 )  1 / 4 P(W2 / B1 )  3 / 4
P(W1 )  3 / 5 P( B2 / W1 )  2 / 4 P(W2 / W1 )  2 / 4

a. P ( B1  B2 )  P ( B2 / B1 ) P ( B1 )  (1 / 4)(2 / 5)
 P ( B1  B2 )  1 / 10
b. P (W2 )  P (W2  B1 )  P (W2  W1 )
 P (W2 / B1 ) P ( B1 )  P (W2 / W1 ) P (W1 )
 (3 / 4)(2 / 5)  (2 / 4)(3 / 5)
 P (W2 )  3 / 5
34
Conditional Probability Cont’d…..
Example-4:

Box A contains 100 bulbs of which 10% are defective. Box B

contains 200 bulbs of which 5% are defective. A bulb is
picked from a randomly selected box.

a. Find the probability that the bulb is defective

b. Assuming that the bulb is defective, find the probability

that it came from box A.

35
Conditional Probability Cont’d…..

Solution:
First let us define the events as follows.
A : Box A is selected

P ( A)  P ( B )  1 / 2
B : Box B is selected P ( D / A)  1 / 10
D : Bulb is defective P ( D / B )  1 / 20

a. P ( D)  P ( D / A) P ( A)  P ( D / B ) P ( B )
 (1 / 10)(1 / 2)  (1 / 20)(1 / 20)
 P ( D)  3 / 40
P ( D / A) P( A) 1 / 20
b. P ( A / D)    (1 / 20)(40 / 3)
P( D) 3 / 40
 P( A / D)  2 / 3

36
Conditional Probability Cont’d…..
Example-5:

One bag contains 4 white and 3 black balls and a second bag
contains 3 white and 5 black balls. One ball is drawn from the
first bag and placed in the second bag unseen and then one ball
is drawn from the second bag. What is the probability that it is
a black ball?

Solution:

First let us define the events as follows.

B1 : black ball is drawn from the first bag
W1 : white ball is drawn from the first bag
37
Conditional Probability Cont’d…..
B2 : black ball is drawn from the second bag
W2 : white ball is drawn from the second bag

Then, we will have:

P( B1 )  3 / 7 P( B2 / B1 )  6 / 9 P( B2 / W1 )  5 / 9

P(W1 )  4 / 7 P(W2 / B1 )  3 / 9 P(W2 / W1 )  4 / 9

P ( B2 )  P( B2  B1 )  P( B2  W1 )

 P( B2 )  P( B2 / B1 ) P( B1 )  P( B2 / W1 ) P (W1 )

 P( B2 )  (6 / 9)(3 / 7)  (5 / 9)(4 / 7)

 P ( B2 )  28 / 63

38
Independence of Events
 Two events A and B are said to be statistically independent if
and only if
P ( A  B )  P ( A) P ( B )

 Similarly, three events A, B and C are said to be statistically

independent if and only if
P ( A  B  C )  P ( A) P ( B ) P (C )

 Generally, if A1, A2, …, An are a sequence of independent

events, then
 n  n
P  Ai    P( Ai )
 i 1  i 1

39
Independence of Events Cont’d……
 If A and B are independent, then we have
P( A  B) P( A) P( B)
i. P( A / B)    P( A)
P( B) P( B)
 P( A / B)  P( A)
P( A  B) P( A) P( B)
ii. P( B / A)    P( B)
P( A) P( A)
 P( B / A)  P( B)

Example-1:

If A and B are independent, then show that A and B are also

independent.

40
Independence of Events Cont’d……
Solution:
P( A)  P( A  B)  P( A  B)
 P( A  B)  P( A)  P( A  B)  P( A)  P( A) P( B)
 P( A  B)  P( A)[1  P( B)]  P( A) P( B)
 By the definition of independent events, A and B are independent.

41