Chapter 2 - YU, Chi Wai PDF

Preview

Summary

MATH2411: Applied Statistics Dr. YU, Chi Wai Chapter 2: PROBABILITY 1 WHAT IS PROBABILITY? Probability is the study of randomness. A probability of an uncertain is the chance (or likelihood) that it will occur. The probability must be between 0 and 1, inclusively. Our goal: We would like to develop ...

Description

MATH2411: Applied Statistics | Dr. YU, Chi Wai

Chapter 2: PROBABILITY HAT BAB ILIT 1 WHA T IS PPR ROBA BILI TY?

Probability is the study of randomness. A probability of an uncertain outcome/event is the chance (or likelihood) that it will occur.



The probability must be between 0 and 1, inclusively.

Our goal: We would like to develop terms and results that form the basis of probability theory, which will allow us to quantify the likelihood or chance of outcomes of interest.

ASIC BABI BILI LITY ERMS 2 BAS IC PPR ROBA BI LI TY TER MS  Random experiment o Is a process which generates observations. o The outcome of the experiment cannot be known BEFORE hand and is only known AFTER the experiment is performed.  Typical examples: Toss a coin, roll a die, etc.

 Sample Space o Is the set of all possible outcomes for a given random experiment, and is often denoted by S.  For instance, S = {H, T}, S = {1, 2, 3, 4, 5, 6} = {integers from 1 to 6}, etc.

 Event o Is a subset/portion of outcomes from the sample space that we are interested in, often denoted by capital letters.  For instance, A = {H} = {Head}, B = {Roll a 1, 2 or 3}, etc.

~1~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

PERA RATI TION EVE BA HEOR ORYY) 3 OPE RA TI ON ON EV ENTS (OR B ASIC SSE ET TTHE HE OR 3.1 UNI NION ON

 The union of two events A and B, denoted by 𝐴 ∪ 𝐵, is the event that either A or B AND both occur. o If the union of some events which are in the same sample space S is exactly the same as S, then those events are said to be exhaustive. o 𝐴 ∪ 𝐵 = 𝐵 if 𝐴 is in 𝐵.

3.2 INTER TERSSEC ECTI TI TION ON

 The intersection of two events A and B, denoted by 𝐴 ∩ 𝐵, is the event that both A and B occur. o If two events have no common element, then their intersection is an empty set 𝜙 (a set without element) and the events are said to be disjoint. o If two or more events are considered and they are pairwise disjoint, then all of these events are said to be mutually exclusive. o 𝐴 ∩ 𝐵 = 𝐴 if 𝐴 is in 𝐵.

3.3 COM OMPL PL PLE EME MENT NT

 For any event A in the sample space S, the complement of A, denoted by 𝐴𝐶 , is the event that contains all points in S but not in A. That is, it is the event that A does not occur. o 𝑆 𝐶 = 𝜙 and 𝜙 𝐶 = 𝑆.

3.4 SYM YMME ME METR TR TRIIC DI DIFFE FFE FFERE RE REN NCE

 The symmetric difference of two events A and B, denoted by, 𝐴∆𝐵, is the event that either A or B (BUT not both) occurs. More simply, it is the union without the intersection. o 𝐴∆𝜙 = 𝐴 and 𝐴∆𝐴 = 𝜙.

~2~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

ENN DIIAG AGRA RAM 4 VEN ND RA M Venn diagrams are the graphs which can illustrate the relationships among events. Note that in Venn diagram, a rectangle is often used to represent the sample space S, and a circle (or other patterns) inside the rectangle represents a particular event in the sample space.

Usually, the relationship between two sets A and B can be drawn as

The two circles overlap, indicating that some, but not all, of the elements of A is also elements of B.

In particular, if all elements of B are elements of A, but some of elements of A are not elements of B (That is, B is strictly in A), then we have the following Venn diagram.

~3~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

UNION

INTERSECTION

EXHAUSTIVE

MUTUALLY EXCLUSIVE

EXHAUSTIVE AND MUTUALLY EXCLUSIVE --- PARTITION

COMPLEMENT

SYMMETRIC DIFFERENCE

~4~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

ROPE PER TIESS OF EEVE VENTS 5 PRO PE RTIE VE NTS  𝐴 ∩ 𝐵 = 𝐵 ∩ 𝐴, and 𝐴 ∪ 𝐵 = 𝐵 ∪ 𝐴  (Associate laws) 𝐴 ∩ (𝐵 ∩ 𝐷) = (𝐴 ∩ 𝐵) ∩ 𝐷, and 𝐴 ∪ (𝐵 ∪ 𝐷) = (𝐴 ∪ 𝐵 ) ∪ 𝐷  (Distributive laws) 𝐴 ∩ (𝐵 ∪ 𝐷) = (𝐴 ∩ 𝐵) ∪ (𝐴 ∩ 𝐷), and 𝐴 ∪ (𝐵 ∩ 𝐷) = (𝐴 ∪ 𝐵) ∩ (𝐴 ∪ 𝐷)  (DeMorgan’s laws) (𝐴 ∩ 𝐵)𝐶 = 𝐴𝐶 ∪ 𝐵𝐶 , and (𝐴 ∪ 𝐵)𝐶 = 𝐴𝐶 ∩ 𝐵𝐶

6 KOLM OG OROV’S A XIOM SYS TEM LMOG OGOROV AXIOM SYSTEM In 1933, a Russian mathematician named Andrei Nikolayevich Kolmogorov published the axiomatic structure of probability theory. The following is the definition of a probability under Kolmogorov’s system.



Definition: A probability on the sample space S is an assignment of a value, say 𝑃(𝐴), to an event A in S via a function 𝑃(⋅) such that:

1. 𝑃(𝑆) = 1. 2. 𝑃(𝐸) ≥ 0, for any event 𝐸 in 𝑆. 3. For any sequence of mutually exclusive events 𝐸1 , 𝐸2 , …, ∞

∞

𝑖=1

𝑖=1

𝑃 (⋃ 𝐸𝑖 ) = ∑ 𝑃(𝐸𝑖 ) .

~5~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

QUESTI ESTION ON Consider a sample space S of all outcomes of a fair die and events A = {1, 3} and B = {2, 4}. Use Kolmogorov’s system to find the probabilities of A, B and their union.

7 PROPE RTIES OF PPROB ROB ABIL ITY ROPERTIES ROBABIL ABILITY  𝑃(𝐴) = 1 − 𝑃(𝐴𝐶 )  𝑃(𝜙) = 0

 𝑃(𝐴) ≤ 𝑃(𝐵), if 𝐴 is in 𝐵.

 𝑃(𝐴) = 𝑃(𝐵), if 𝐴 = 𝐵 (i.e. 𝐴 is in 𝐵 and 𝐵 is in 𝐴)

 𝑃(𝐴 ∪ 𝐵) = 𝑃(𝐴) + 𝑃(𝐵) − 𝑃(𝐴 ∩ 𝐵)

~6~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

~7~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

ROBABI BABILI LITY AN EVENT AMPLE WITH FINITE EQUAL UALLY LIKELY 8 PRO BABI LI TY OF A N EVEN T IN SSAMP AMP LE SSP PACE W ITH FI NITE EQ UAL LY LIK ELY OUTCOM TCOMES OU TCOM ES One simple scenario in probability is to consider the sample space with finite equally likely outcomes. Outcomes are said to be equally likely if they have equal chances to occur. (Theorem) Consider a sample space 𝑆 = {𝑤1 , 𝑤2 , … , 𝑤𝑁 } with N equally likely outcomes, where N is a finite positive integer. Let 𝐸 be any event in 𝑆. Then 𝑃(𝐸) =

𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠 𝑖𝑛 𝐸 . 𝑁

EXA XAMP MP MPLE LE Flip a fair coin three times. We have the sample space S = { HHH, HHT, HTH, THH, TTH, THT, HTT, TTT }. Denote by E the event that “heads” occurs at the first time. Thus, 𝐸 = {HHH, HHT, HTH, HTT},

and its probability can be found by 𝑃(𝐸) =

4 1 = . 8 2

ONDITIO DITIONAL ROBAB BABILI ILITY 9 CON DITIO NAL PPRO RO BAB ILI TY Conditional probability is the probability of one event given that another event (conditional event) has already occurred. The conditional probability of A given that an event B has already occurred or known is denoted by 𝑃(𝐴|𝐵). A typical example: Toss a fair die once. P(getting odd number) = 1/2, and P(getting odd number | number is known to be greater than 1) = 2/5. Note that the sample space is reduced to {2, 3, 4, 5, 6} if the conditional event occurs.

~8~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

This example tells us that sometimes the knowledge/information that an event B has occurred influences the probability that A will occur. We can “update” the probability of an event A happening after obtaining some “additional” information from B.



Definition: Let A and B be two events such that 𝑃(𝐵)

> 0 . Then the conditional probability that A will occur given that B has occurred is defined to be 𝑃(𝐴|𝐵) =

𝑃(𝐴 ∩ 𝐵) . 𝑃(𝐵)

Referring to the typical example, we re-do the question of finding the conditional probability by the above definition. Let A be the event “getting odd number” and B be the event “getting number greater than 1”. Thus, 𝑃(𝐴|𝐵) =

𝑃(𝐴 ∩ 𝐵) 2/6 2 = = . 𝑃(𝐵) 5/6 5

Note that the formula of the conditional probability implies that 𝑃(𝐴 ∩ 𝐵 ) = 𝑃(𝐴|𝐵)𝑃(𝐵), known as the multiplication rule, which can be easily generalized to a case with 3 or more events. For instance, Multiplication rule for three events: 𝑃(𝐴 ∩ 𝐵 ∩ 𝐶 ) = 𝑃(𝐶|(𝐴 ∩ 𝐵))𝑃(𝐵|𝐴)𝑃(𝐴) . Note that the above result requires one condition. What is it? Discuss it in class.

Example: A box of fuses contains 20 fuses, of which 5 are defective. If three of the fuses are selected randomly and removed from the box in succession without replacement, the probability that all three fuses are defective can be found in the following way: Let A be the event that the first selected fuse is defective. Similarly, let B and C be the respective events that the second and third selected fuses are defective. Obviously, 𝑃(𝐴) = 5/20, 𝑃(𝐵|𝐴) = 4/19 and 𝑃(𝐶|(𝐴 ∩ 𝐵)) = 3/18 . Thus, the required probability is 4

5

1

𝑃(𝐴 ∩ 𝐵 ∩ 𝐶 ) = 𝑃(𝐶|(𝐴 ∩ 𝐵 ))𝑃(𝐵|𝐴)𝑃(𝐴) =18 × 19 × 20 = 114 . 3

~9~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

OPERTIES COND NDITION ITIONAL ROBABI BABILITY 10 PROPER TIES OF CO ND ITION AL PPRO RO BABI LITY  𝑃(𝐴|𝐵) = 1 − 𝑃(𝐴𝐶 |𝐵)  𝑃( (𝐴 ∪ 𝐵)|𝐷) = 𝑃(𝐴|𝐷) + 𝑃 (𝐵|𝐷 ) − 𝑃((𝐴 ∩ 𝐵)|𝐷)  𝑃((𝐴 ∩ 𝐵)|𝐷) = 𝑃(𝐴|(𝐵 ∩ 𝐷))𝑃(𝐵|𝐷)

11 IND EPE NDEN CE NDEPE EPENDEN NDENCE Note that it is possible that the knowledge/information that an event B has occurred DOES NOT INFLUENCE the probability that A will occur. That is, 𝑃(𝐴|𝐵) = 𝑃(𝐴). In this case, we would say that the events A and B are independent. By the above multiplication rule, we can notice that events A and B are independent if and only if 𝑷(𝑨 ∩ 𝑩) = 𝑷(𝑨)𝑷(𝑩) .

If the events are NOT independent, then they are said to be dependent.

QUESTI ESTION ON A fair coin is tossed three times. Consider the following events: A: the event that a head occurs on each of the first two tosses, B: the event that a tail occurs on the third toss, and C: the event that exactly two tails occur in the three tosses. Are A and B independent? Are B and C independent?

~ 10 ~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

The following definition is for the independence of three events: 

Definition: Events 𝐴1 , 𝐴2 and 𝐴3 are independent if and only if the following conditions are ALL true:

TOTALL PR PROBA OBABILITY 12 LAW OF TOTA OBA BILITY By the Venn diagram, we can notice that (𝐴 ∩ 𝐵) and (𝐴 ∩ 𝐵𝐶 ) are disjoint. Therefore, we can write the event A as 𝐴 = (𝐴 ∩ 𝐵) ∪ (𝐴 ∩ 𝐵𝐶 ) and then have the result often known as “LAW OF TOTAL PROBABILITY”

For 0 < 𝑃(𝐵) < 1,

𝑃(𝐴) = 𝑃(𝐴 ∩ 𝐵 ) + 𝑃(𝐴 ∩ 𝐵𝐶 ) 𝑃(𝐴) = 𝑃(𝐴|𝐵)𝑃(𝐵) + 𝑃(𝐴|𝐵𝐶 )𝑃(𝐵𝐶 )

More generally, (LAW OF TOTAL PROBABILITY --- GENERAL VERSION) If 𝐵1 , 𝐵2 , , … , 𝐵𝑘 are mutually exclusive and exhaustive events with probabilities between 0 and 1 exclusively, then for any event A, we have 𝑘

𝑃(𝐴) = ∑ 𝑃(𝐴|𝐵𝑖 )𝑃(𝐵𝑖 ) . 𝑖=1

~ 11 ~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

QUESTI ESTION ON In a certain assembly plant, there are only three machines, I, II and III, which make 30%, 45%, and 25%, respectively, of the products. It is known from past experience that 2%, 3%, and 2% of the products made by each machine, respectively, are defective. Now, suppose that a finished product is randomly selected. What is the probability that it is defective?

~ 12 ~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

YESS’ THEO THEOREM 13 BAYE REM Bayes’ theorem is a famous theorem in probability related the conditional probability 𝑃(𝐴|𝐵) to its “inverse” counterpart 𝑃(𝐵|𝐴). One typical application of using this theorem in practice is for a diagnostic test --- determine the probability of a person infected by a particular virus, given that the test on this person shows a positive result.

Thomas Bayes (1701 - 1761)

(Bayes’ theorem --- 1st form) If 𝑃(𝐴) > 0 and 𝑃(𝐵) > 0, then we have

𝑃(𝐵|𝐴) =



 

𝑃(𝐵) 𝑃(𝐴|𝐵) . 𝑃(𝐴)

𝑃(𝐴|𝐵) and 𝑃(𝐵|𝐴) if and only if 𝑃(𝐴) = 𝑃(𝐵). In general, 𝑃(𝐴|𝐵) and 𝑃(𝐵|𝐴) are different. Thus, please don’t jump to conclude that smokers will have large chance to have lung cancer if we observed that a large proportion of lung cancer patients are smokers. The unconditional probability 𝑃(𝐵) is also called the prior probability of 𝐵 in the sense that it does not take any information about 𝐴 into account. The conditional probability of 𝐵 given 𝐴, 𝑃(𝐵|𝐴), is also called the posterior probability in contrast to the prior probability 𝑃(𝐵). It is “posterior” in the sense that some additional information (occurrence of 𝐴) has been taken into account.

Together with the law of total probability, we have the following 2nd form: (Bayes’ theorem --- 2nd form) If 𝐵1 , 𝐵2 , , … , 𝐵𝑘 are mutually exclusive and exhaustive events with probabilities between 0 and 1 exclusively, then for any event 𝐵𝑗 , we have 𝑃(𝐵𝑗 |𝐴) =

∑𝑘𝑖=1

𝑃(𝐵𝑗 ) 𝑃(𝐴|𝐵𝑗 ) . 𝑃(𝐴|𝐵𝑖 )𝑃(𝐵𝑖 )

~ 13 ~

MATH2411: Applied Statistics | Dr. YU, Chi Wai

QUESTI ESTION ON Given 𝑃(𝐵) = 0.03, 𝑃(𝐴𝐶 |𝐵) = 0.01, and 𝑃(𝐴|𝐵 𝐶 ) = 0.02, where 𝐴 is the event that a test shows a positive result as carrier of a particular virus and 𝐵 is the event that a person is infected by this virus. Then, what is the probability of a person infected by the virus, given that the test on this person shows a positive result?

Huh? Positive?!?!?!?!

~ 14 ~...