Lecture 15: The full rank model, ANOVA PDF

Preview

Summary

Lecture 15: The full rank model (hypothesis), ANOVA (SSres, SSreg, SStotal, MSreg, MSres, ANOVA Table)...

Description

Lecture 15 The full rank model - The first thing we want to test is model relevance: does our model contribute anything at all? - If none of the x variables have any relevance for predicting y , then all the parameters β will be 0. - We test for this using the null hypothesis H 0 : β=0

- Alternatively, if at least some of the x variables are relevant to predicting y , then the corresponding parameters will be nonzero. So our alternative hypothesis is H1: β ≠ 0

- To test these hypotheses, we assume throughout this section that the error ε are multivariate normal.

ANOVA - The method used to test the hypotheses is analysis of variance (ANOVA). - If β=0 , then y=ε consists entirely of errors. In the case, y T y , the sum of squares of the errors, measures the variability of the errors. - However, if β ≠ 0 , then y= Xβ + ε . In this case, some of y T y will come from errors, but some will come from the model predictions. - By separating y T y into these two parts, we can compare them to see how well the model is doing. - More precisely, the sum of squares of the residuals (variation attributed to error) is

which means that

- We call the regression sum of squares and denote it by S S Reg . It reflects the variation in the response variable that is explained by the model. - We call the total variation in the response variable S S Total = y T y . We have divided it into: S S Total =S S Reg + S S Res

- To create a formal test of β=0 , we compare S S Reg against S S Res . If S S Reg is large compared to S S Res , then we have evidence that β≠0 . - To know exactly how large, we must first derive the distributions of S S Reg and S S Res . - Theorem 5.1 o In the full rank general linear model y= Xβ + ε ,

S S Res σ2

has a

distribution with n− p degrees of freedom. - Theorem 5.2 ❑

2

o In the full rank general linear model y= Xβ + ε ,

S S Reg σ2

has a

noncentral ❑2 distribution with p degrees of freedom and noncentrality parameter ¿

1 2σ

2

βT X T Xβ

- Theorem 5.3 o In the full rank general linear model y= Xβ + ε , S S Res and S S Reg are independent. o This can be proved by observing that they are both quadratic forms in y and applying Theorem 3.11. o Alternatively, we can write S S Reg =bT X T Xb and observe that b and s 2 are independent. o How to test β=0 ? o Observe that if this is true, the noncentrality parameter for S S Reg σ2

must be 0.

o Thus, under H 0 :

has an F distribution with p and n-p degrees of freedom. o What happens if H 0 is not true? The expected values of M S Reg is

o The expected value of the denominator M S Reg is

o So if β=0 , E

[ ]

S S Reg 2 = σ and the statistic should be close to 1. p

o But if β ≠ 0 , since X T X is positive definite, we get

[ ]

S S Reg 2 > σ and the statistic should be bigger than 1. p o Therefore, we should use a one-tailed test and reject H 0 is the E

statistic is large. o To lay out all the calculations, we use a familiar ANOVA table....