Complete Lecture Notes PDF

Preview

Summary

Statistics 1 Notesby Tyler Wrightgithub/Fluxanoia fluxanoia.coThese notes are not necessarily correct, consistent, representative of the course as it stands today, or rigorous. Any result of the above is not the author’s fault.These notes are marked as unsupported, they were supported up until June ...

Description

Statistics 1 Notes by Tyler Wright github.com/Fluxanoia

fluxanoia.co.uk

These notes are not necessarily correct, consistent, representative of the course as it stands today, or rigorous. Any result of the above is not the author’s fault. These notes are marked as unsupported, they were supported up until June 2019.

1

Contents 1 The 1.1 1.2 1.3

Basics of Data Analysis Samples . . . . . . . . . . . . . . Probability Density Functions . . Measures of Central Tendency . . 1.3.1 Sample Median . . . . . . 1.3.2 Sample Mean . . . . . . . 1.3.3 Trimmed Sample Mean . . 1.4 Measures of Spread . . . . . . . . 1.4.1 Sample Variance . . . . . 1.4.2 Hinges . . . . . . . . . . . 1.4.3 Quartiles . . . . . . . . . . 1.4.4 Interquartile Range (IQR) 1.4.5 Skewness . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

7 7 7 7 7 7 8 8 8 8 8 9 9

2 Assessing Fit 9 2.1 Quantiles of a Distribution . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Quantile-Quantile (Q-Q) Plots . . . . . . . . . . . . . . . . . . . . . . 10 2.3 Probability Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3 Estimation 10 3.1 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Distribution Quantities . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4 Method of Moments Estimation 11 4.1 Definition of a Moment . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4.2 The Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4.3 Method of Moments on the Exponential . . . . . . . . . . . . . . . . 11 5 Maximum Likelihood Estimation 5.1 The Process . . . . . . . . . . . . 5.2 Optimisation of the Method . . . 5.3 Multiple Parameters . . . . . . . 5.4 Non-regular density . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

12 12 12 13 13

6 The Performance of Estimators 13 6.1 Variation of Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . 13 6.2 Method of Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2

7 Central Limit Theorem 14 7.1 Definition of the Central Limit Theorem . . . . . . . . . . . . . . . . 14 7.2 Continuity Correction . . . . . . . . . . . . . . . . . . . . . . . . . . 14 8 A Reminder on Moment Generating Functions 8.1 Definition of a Moment Generating Function (MGF) . . . 8.2 Properties of a Moment Generating Function . . . . . . . . 8.2.1 Standard examples of moment generating functions 8.2.2 Joint moment generating functions . . . . . . . . . 8.2.3 Independence of moment generating functions . . . 8.2.4 Uniqueness of moment generating functions . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

14 14 15 15 15 15 15

9 The Normal Distribution 16 9.1 Transformation and Addition of the Normal . . . . . . . . . . . . . . 16 9.2 Independence of the Sample Mean and the Sum of Squared Difference 16 10 Sampling Distributions related to the Normal 10.1 The χ2 Distribution . . . . . . . . . . . . . . . . . 10.1.1 Definition of the χ2 distribution . . . . . . 10.1.2 Properties of the χ2 distribution . . . . . . 10.2 The t Distribution . . . . . . . . . . . . . . . . . 10.2.1 Definition of the t distribution . . . . . . . 10.2.2 Properties of the t distribution . . . . . . . 10.2.3 Samples from the Normal with σ unknown

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

11 Confidence Intervals 11.1 Definition of a Confidence Interval . . . . . . . . . . . . . . . 11.2 Examples of Confidence Intervals . . . . . . . . . . . . . . . 11.2.1 N (µ, σ2 ) : Confidence interval for µ with σ 2 known . 11.2.2 N (µ, σ2 ) : Confidence interval for µ with σ 2 unknown 11.2.3 N (µ, σ2 ) : Confidence interval for σ 2 with µ unknown 11.2.4 U (0, θ) : Confidence interval for θ . . . . . . . . . . . 11.2.5 Confidence Intervals by Simulation . . . . . . . . . . 12 Notes on Hypothesis Testing: 12.1 Test Statistics . . . . . . . . 12.2 p-values . . . . . . . . . . . 12.3 Error . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

16 16 16 17 18 18 18 18

. . . . . . .

19 19 19 19 19 20 20 20

Population Means 21 . . . . . . . . . . . . . . . . . . . . . . . 21 . . . . . . . . . . . . . . . . . . . . . . . 21 . . . . . . . . . . . . . . . . . . . . . . . 21 3

13 Hypothesis Tests: Comparision 13.1 Normal Hypothesis Test . . . 13.1.1 Assumptions . . . . . . 13.1.2 Hypotheses . . . . . . 13.1.3 Test Statistic . . . . . 13.1.4 Critical Region . . . . 13.2 One Sample t-test . . . . . . . 13.2.1 Assumptions . . . . . . 13.2.2 Hypotheses . . . . . . 13.2.3 Test Statistic . . . . . 13.2.4 Critical Region . . . . 13.3 Pooled Two Sample t-test . . 13.3.1 Assumptions . . . . . . 13.3.2 Hypotheses . . . . . . 13.3.3 Test Statistic . . . . . 13.3.4 Critical Region . . . . 13.4 Welch Two Sample t-test . . . 13.4.1 Assumptions . . . . . . 13.4.2 Hypotheses . . . . . . 13.4.3 Test Statistic . . . . . 13.4.4 Critical Region . . . . 13.5 Paired t-test . . . . . . . . . . 13.5.1 Assumptions . . . . . . 13.5.2 Hypotheses . . . . . . 13.5.3 Test Statistic . . . . . 13.5.4 Critical Region . . . .

of Population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14 Linear Regression 14.1 Model Assumptions . . . . . . . 14.2 Errors . . . . . . . . . . . . . . 14.3 Summary Statistics . . . . . . . 14.4 Finding α ˆ and βˆ . . . . . . . . 14.5 Residual Sum of Squares and σˆ 2

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

Means . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

22 22 22 22 22 22 23 23 23 23 23 24 24 24 24 24 25 25 25 25 25 26 26 26 26 26

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

27 27 27 27 28 28

. . . . .

. . . . .

. . . . .

. . . . .

15 Notes on Hypothesis Testing: Regression 28 15.1 Assumption of Normality . . . . . . . . . . . . . . . . . . . . . . . . . 28 ˆ and σˆ 2 . . . . . . . . . . . . . . . . . . . . . 28 15.2 The Distribution of α, ˆ β, 16 Confidence Intervals for α and β 29 16.1 Confidence Interval for α . . . . . . . . . . . . . . . . . . . . . . . . . 29 16.2 Confidence Interval for β . . . . . . . . . . . . . . . . . . . . . . . . . 29 4

17 Hypothesis Tests: Linear 17.1 Hypothesis Test for β . 17.1.1 Assumptions . . 17.1.2 Hypotheses . . 17.1.3 Test Statistic . 17.1.4 Critical Region 17.2 Hypothesis Test for α . 17.2.1 Assumptions . . 17.2.2 Hypotheses . . 17.2.3 Test Statistic . 17.2.4 Critical Region

Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

18 A List of Important Formulae 18.1 Measures of Central Tendency . . . . . . . . . . . . . . . . . . 18.1.1 Sample Mean . . . . . . . . . . . . . . . . . . . . . . . 18.1.2 Sample Median . . . . . . . . . . . . . . . . . . . . . . 18.2 Measures of Spread . . . . . . . . . . . . . . . . . . . . . . . . 18.2.1 Sample Variance . . . . . . . . . . . . . . . . . . . . . 18.2.2 Hinges . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.2.3 Quartiles . . . . . . . . . . . . . . . . . . . . . . . . . . 18.2.4 Outliers . . . . . . . . . . . . . . . . . . . . . . . . . . 18.3 Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.3.1 Maximum Likelihood . . . . . . . . . . . . . . . . . . . 18.3.2 Performance of Estimators . . . . . . . . . . . . . . . . 18.4 Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.4.1 Distributions related to the normal . . . . . . . . . . . 18.4.2 χ2 distribution . . . . . . . . . . . . . . . . . . . . . . 18.4.3 t-distribution . . . . . . . . . . . . . . . . . . . . . . . 18.5 Sum of Squared Difference . . . . . . . . . . . . . . . . . . . . 18.6 Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . 18.6.1 α ˆ and βˆ . . . . . . . . . . . . . . . . . . . . . . . . . . 18.6.2 Sum of squared errors . . . . . . . . . . . . . . . . . . 18.6.3 Estimated variance of the errors . . . . . . . . . . . . . 18.6.4 The distribution of α, ˆ βˆ and, σˆ 2 . . . . . . . . . . . . . 18.6.5 Sample variance of α ˆ and βˆ . . . . . . . . . . . . . . . 18.7 Confidence Intervals . . . . . . . . . . . . . . . . . . . . . . . 18.7.1 N (µ, σ2 ) : Confidence interval for µ with σ 2 known . . 18.7.2 N (µ, σ2 ) : Confidence interval for µ with σ 2 unknown . 18.7.3 N (µ, σ2 ) : Confidence interval for σ 2 with µ unknown . 18.7.4 U (0, θ) : Confidence interval for θ . . . . . . . . . . . . 18.7.5 Confidence interval for α . . . . . . . . . . . . . . . . . 5

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

30 30 30 30 30 30 31 31 31 31 31

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

32 32 32 32 32 32 32 32 32 33 33 33 33 33 33 33 34 34 34 34 34 34 35 35 35 35 35 35 35

18.7.6 Confidence interval for β . 18.8 Test Statistics . . . . . . . . . . . 18.8.1 Normal Hypothesis Test . 18.8.2 One Sample t-test . . . . . 18.8.3 Pooled Two Sample t-test 18.8.4 Welch Two Sample t-test . 18.8.5 Paired t-test . . . . . . . . 18.8.6 Test for α . . . . . . . . . 18.8.7 Test for β . . . . . . . . .

6

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

36 36 36 36 36 36 36 37 37

1 1.1

The Basics of Data Analysis Samples

A sample is a set of values observed from a simple random sample of some size n from a population where each sample member is chosen independently of each other and each population member is equally likely to be selected. Samples are usually written as {x1 , x2 , . . . , xn } where each xi represents an observed value. If the data is ordered, the data is written as {x(1) , x(2) , . . . , x(n) } (for numerical values, this is ascending order). So, in this case, x(1) is always the minimum, x(n) is always the maximum.

1.2

Probability Density Functions

For a sample {x1 , x2 , . . . , xn }, we can imagine each datum as being distributed with some population distribution X. As each datum is independent of all other observed values, we can write the probability density of this sample as follows: fX (x1 , x2 , . . . , xn ) =

n Y

fX (xi ).

i=1

1.3 1.3.1

Measures of Central Tendency Sample Median

For a sample X = {x1 , x2 , . . . , xn }, we define the sample median M as follows: ( x(m+1) for n = 2m + 1 M(X) = x(m) +x(m+1) for n = 2m. 2 Essentially, it equals the middle value or the average of the middle values. Also, it’s important to note that the median is not sensitive to extreme values. 1.3.2

Sample Mean

For a sample X = {x1 , x2 , . . . , xn }, we define the sample mean X as follows: ! n 1 X xi . X= n i=1

This is easy to calculate even when combining samples. However, it is sensitive to extreme values. 7

1.3.3

Trimmed Sample Mean

For a sample X = {x1 , x2 , . . . , xn }, we define the trimmed sample mean X∆ for some percentage ∆% as follows:   ∆ Let k = n 100 ˜ = {x(k+1), x(k+2), . . . , x(n−k) } Let X ˜ X∆ = X˜ (the sample mean of X).

Basically, you remove the first and last ∆% of values and take the sample mean of the remaining values.

1.4 1.4.1

Measures of Spread Sample Variance

For a sample X = {x1 , x2 , . . . , xn }, we define the sample variance s2 as follows: Pn (xi − x ¯2 ) 2 s = i=1 n−1 ! n X 1 = (xi2 ) − n¯ x2 . n−1 i=1

This measures how much the data varies. 1.4.2

Hinges

There are two hinge measures, lower (H1 ) and upper (H3 ): H1 = median of {data values ≤ the median}

H3 = median of {data values ≥ the median}. 1.4.3

Quartiles

For a sample X = {x1 , x2 , . . . , xn }, there are two quartile measures, lower (Q1 ) and upper (Q3 ). The formulae are long and overly complicated so for Q1 : • Calculate k =

n+1 4

• If k ∈ Z, Q1 = x(k) • Otherwise, do linear interpolation between x(⌊k⌋) and x(⌊k+1⌋) 8

And similarly for Q3 : • Calculate k = 3

 n+1 4

• If k ∈ Z, Q3 = x(k)

• Otherwise, do linear interpolation between x(⌊k⌋) and x(⌊k+1⌋) For large samples, the quartiles and hinges tend to be close to each other. 1.4.4

Interquartile Range (IQR)

The IQR is the difference between Q3 and Q1 (Q3 − Q1 ). In this course, outliers are defined as more than 23 (IQR) (or approx. 23(H3 − H1 )) from the median. 1.4.5

Skewness

We measure skewness by the distance of the hinges from the median. If H3 is further from the median than H1 , we have a longer right tail. If the converse is true, we have a longer left tail.

2

Assessing Fit

2.1

Quantiles of a Distribution

For a distribution X with cumulative distribution function FX , the quantiles of the distribution are defined as the set of values:   n 1 −1 ,..., FX . n+1 n+1 We use n + 1 on the denominator as F −1 X (1) can be ∞. The ordered sample is called the set of sample quantiles.

9

2.2

Quantile-Quantile (Q-Q) Plots

These are the steps for constructing a Q-Q plot of a sample {x1 , x2 , . . . , xn } with cumulative distribution function FX : • Generate an estimate for the parameter(s) (θˆ1 , θˆ2 , . . .) • Compute the quantiles (the expected quantiles if the hypothesised model is correct) k ˆ x(k) ). • Plot each expected quantile against the sample quantile (FX−1( n+1 ; θ),

What we would expect, if our hypothesis is correct, is that the plotted points lie close to the line y = x. This is saying our sample and expected quantiles are close together.

2.3

Probability Plots

These are similar to the Q-Q plots but plot the sample cumulative probability against k ). expected probability (FX (x(k) ), n+1

3

Estimation

We have that a population is distributed with some distribution X with a probability density function (PDF) fX , cumulative distribution function (CDF) FX , and some parameters {θ1 , . . .}. We can make guesses at the distribution of a sample and use tests to verify that. But, to do these tests we need a valu for the parameters. It’s not practical to guess these, so we need to estimate them.

3.1

Parameters

We say θˆ is an estimator for θ and define it as a function of a sample {x1 , x2 , . . . , xn }: ˆθ(x1 , x2 , . . . , xn ).

3.2

Distribution Quantities

From our estimated value of the distribution parameters, we can calculate estimated values for distribution quantities like the mean and variance. We consider τ a function of the parameter that gives a distribution quantity: • True quantity: τ (θ) where θ is the true distribution parameter

ˆ where θˆ is our estimated parameter. • Estimated quantity: τˆ = τ (θ) 10

4

Method of Moments Estimation

4.1

Definition of a Moment

The kth moment of a probability distribution X is defined as follows: Z ∞ k xk fX (x)dx. E(X ) := −∞

Setting k = 1 gives us the familiar expectation of X : Z ∞ E(X) = xf (x)dx. −∞

In the discrete case, the integral is a sum. We define the kth sample moment mk as follows: Pn k x mk = i=1 i . n Or rather, the kth moment is the average value of xk in the sample.

4.2

The Process

By considering the a probability distribution X with parameter θ, we can find functions for the moments of X in terms of θ. These can be rearranged to give functions for θ in terms of the moments. We can then use the sample moments to generate an estimate for θ (θˆmom ).

4.3

Method of Moments on the Exponen...