Chapter 4 Lack of Fit Test notes PDF

Preview

Summary

lecture 4...

Description

1

• We shall discuss the pure error sum of squares, SSPE • and the Lack of fit sum of squares, SSLF • The above 2 sum of squares are the decomposition of the Error sum of squares. That is, SSE = SSLF + SSPE • We shall discuss “Repeated measurements” which are required in the lack of fit test • The Lack of fit test is to check if the proposed model has room for improvement • Finally we shall talk about how to use software to do the lack of fit test

2

Introduction • F-test for the significance of the model only tests if a model with at least one predictors is better than a model without any predictors. (Test H0: 𝛽 = 0 for all 𝑖 = 1, ⋯ , 𝑝 against H1: At least one of the 𝛽 ’s are different from 0) • While the partial F-test only test if some of the predictors contributing to the model that has already included other predictors • Neither of these 2 tests tells us whether the proposed regression model is appropriate or not (or whether we have room for improvement for the given model)

3

• We use an example to illustrate the significance of a model does not necessary imply the model is good. There may be a better model to fit a given data set. • Let us consider the following data set • There are altogether 12 pairs of observations. • The values of 𝑥 range from 0 to 6, while the values of 𝑦 range from 18 to 42. • The scatter plot of the observations is as shown. • Notice that there are 2 observations with the 𝑥 -value being 1, another 2 observations with the 𝑥 -value being 2 and so on..

4

• We fit a simple regression model 𝑦 = 𝛽 + 𝛽 𝑥 + 𝜖 to the data. • The results are given as follows • The fitted model is 𝑦 = 19.67 + 4𝑥 • The sum of squares error SSE equals 310.67 with a 10 degrees of freedom • The observed F-value equals 18.03 and the corresponding p-value equals 0.002 • Hence the simple regression model is significant. • As we can see from the plot, the fitted line suggests that the slope of the regression model is likely to be different from 0.

5

• We have the following questions • Is the simple regression model appropriate? • Is it possible to get a better model? • In order to answer these questions, we first try to fit a second order polynomial model to the data. • As seen from the scatter plot, the quadratic polynomial model 𝑦 = 𝛽 + 𝛽 𝑥 + 𝛽 𝑥  + 𝜖 seems fitting the data better.

6

• The fitted model is 𝑦 = 1 + 18𝑥 − 2𝑥  . • The sum of squares error SSE equals 12 with a 9 degrees of freedom • The observed F-value equals 322 and the corresponding p-value is 4.23(10)-9 • Hence the quadratic polynomial model is significant. • Remark: Though both models have significant models, the unexplained variation (SSE) drops from 310.67 for the model with 𝑥 only to 12 for the model with 𝑥 and 𝑥  after adding the predictor 𝑥  to the model with 𝑥.

7

• Questions • Is the quadratic polynomial model appropriate? • Is it possible to get a better model? • By a better model, we mean it is a model which is significant and has a significant improvement over the quadratic polynomial model. • The answer: Perform a lack of fit test if there are repeated measurements

8

Section 4.2 Repeated measurements • Suppose there are m groups of repeated measurements and each of these groups has 𝑛 observations, where 𝑗 denotes the 𝑗-th group and 𝑗 = 1, 2, ⋯ , 𝑚. • Repeated measurements are the measurements taken at the same combination of levels of 𝑥1, and up to 𝑥 on individual subjects • Let us look at the following example • The data are given in the table • Both the first and the second observations have 𝑥 = 0 and 𝑥 = 20. These two observations are considered as “repeated measurements”. • Similarly, the third and the fourth observations are considered as repeated measurements since they both have 𝑥 = 5 and 𝑥 = 5. • The fifth and the sixth observations are “repeated measurements” as well.

9

Section 4.3 Pure error sum of squares and lack of fit sum of squares • The part of the variation that cannot be explained by the model, that is, SSE, can be decomposed into 2 components, sum of squares pure error, SSPE, and sum of squares due to lack of fit, SSLF. • SSPE is the part of SSE that can never be explained by any model. • SSLF is the part that we can improve on it by fitting a better model. • The relationship can be explained in the following diagram • To begin with, we have a rectangle to represent the total sum of squares SST • Part of the Total sum of squares, SST, is the error sum of squares, SSE, which is represented by the blue rectangle. • The error of sum of squares, SSE, is divided into two parts, the pure error sum of squares, SSPE, represented by the purple rectangle • and the lack of fit sum of squares, SSLF, represented by the shaded part of the blue rectangle.

10

• SSE measures the variability of 𝑦 which cannot be explained by the given model • The pure error component, SSPE measures the inherent variability of 𝑦 which cannot be explained by ANY model. • The lack of fit component, SSLF, represents the variability of y that cannot be explained by the given model and may be reduced if a “better” model is used. • That is, SSE = SSPE + SSLF.

11

• SSPE is defined as the sum over j from 1 to m (where m is the number of groups of   repeated measurements) of ∑ 𝑦 − 𝑦 (where 𝑛 is the number of observations in the j-th group). • where 𝑦 is the mean of the y’s in the j-th group with the combination of levels of 𝑥 , 𝑥 , ⋯ , 𝑥 . • The SSPE can be interpreted as follows • For each group of repeated measurements, we compute the square of the deviation of each observations, 𝑦 from the group mean 𝑦 • and then sum all these deviations within the group. • Finally, we sum over all the m groups of the sum of the squares of the deviations • Since each sum of the squares of deviation has 𝑛 − 1 degrees of freedom, therefore the pure error sum of squares has a degrees of freedom which is sum over degrees of freedom of all the m groups • This is given by sum (𝑛 – 1) over j from 1 to m.. (i.e. ∑  (𝑛 − 1))

12

• Let us look at Example 1 • There are 3 groups of repeated measurements • The first group is indexed by 𝑥 , 𝑥 = (0, 20). There are 2 observations in this group. The sum of the squares of the deviation of the observations from the group mean is 1.0658. This sum of squares contributes 2 – 1 degree of freedom towards the degrees of freedom for the SSPE. • Similarly, the sum of squares of the deviation of 𝑦’s from its group mean in the second group is 3.125 with a degree of freedom 1 • And sum of squares of the deviation in the third group is 0.18605 with 1 degree of freedom. • Hence the SSPE is the sum of all the sum of squares of deviation in these groups of repeated measurements and it equals 4.37665 with 3 degrees of freedom.

13

Lack of Fit Sum of Squares, SSLF • Definition: Sum of squares lack of fit equals the difference between the error sum of squares and the pure error sum of squares • That is, SSLF for a model equals SSE for the model minus SSPE. Note that SSPE does not depend on the given model. • Hence the degrees of freedom for lack of fit sum of squares is 𝑛 − 𝑝 + 1 − ∑  𝑛 − 1 . where 𝑛 − (𝑝 + 1) is the degrees of freedom for SSE and ∑  𝑛 − 1 is the degrees of freedom for the SSPE

14

• We use the data in Example 1 to show how to obtain the SS Lack of Fit • It can be shown that 𝑆𝑆𝐸 𝑥 , 𝑥 = 4.4162 with 4 degrees of freedom. • Hence SSLF for the model with 𝑥 and 𝑥 equals SSE for the model with 𝑥 and 𝑥 – SSPE = 4.4162 – 4.3768 and it equals 0.0394 with a degrees of freedom 4 – 3 and it equals 1. • Question: Is the SSLF for the model with 𝑥 and 𝑥 large? • We compare the SSLF with 𝑥 and 𝑥 with the SSPE after each of them has been adjusted by its degrees of freedom. • That is, we consider the ratio 𝑆𝑆𝐿𝐹(𝑥1, 𝑥2) divided by 𝑑𝑓 and SSPE divided by 𝑑𝑓 .

15

Section 4.4 Lack of fit test • We want to test the null hypothesis H0 that there is no lack of fit in the model • against the alternative hypothesis that there is lack of fit • Consider the test statistic • F-test for the lack of fit which is the mean squared lack of fit, MSLF, over the mean square pure error, MSPE. • where MSLF equals the lack of fit sum of squares SSLF divided by the corresponding degrees of freedom, 𝑛 – (𝑝 + 1) – ∑  𝑛 − 1 • and MSPE equals the pure error sum of squares SSPE divided by the corresponding degrees of freedom, ∑  𝑛 − 1 .

16

• It can be shown that under the null hypothesis, H0 • F-test for the lack of fit (𝐹 ) defined in the previous slide follows an Fdistribution with 𝑛 − (𝑝 + 1) – 𝑎 and 𝑎 degrees of freedom. where 𝑎 = ∑  𝑛 − 1 and it is the degrees of freedom for SSPE. • We reject the null hypothesis at the α level of significance if • the observed 𝐹 value is greater than the critical value 𝐹 (𝑛 − 𝑝 + 1 −  ∑  𝑛 − 1 , ∑ 𝑛 − 1 ).

17

• Let us look at Example 1 again • Follow from Slide 15, we have 𝑆𝑆𝐿𝐹 = 0.03935 with 1 d.f. and 𝑆𝑆𝑃𝐸 = 4.3769 with 3 d.f. • the F-test for the Lack of Fit equals MSLF divided by MSPE and it equals 0.267 • Since the observed F-value for the Lack of Fit Test equals 0.267 which is less than the critical value 𝐹. (1, 3) which is 10.13, therefore we do not reject H0 and conclude that there is no significant evidence of any lack of fit in the given multiple regression model 𝑦 = 𝛽 + 𝛽 𝑥 + 𝛽 𝑥 + 𝜖.

18

Some remarks • If the lack of fit test is significant, then we should look for an alternate model for the relationship between the response y and the predictors 𝑥 ’s. It is because the difference between the SSE for the given model and the SSE for the “best model” (i.e. SSPE is treated as the SSE for the “best” model) is considered as significantly large. Therefore, we have a large room for improvement for the given model. • If the lack of fit test is not significant, then it is not necessary to find a more complicated model. It is because the difference between the SSE for the given model and the SSE for the “best” model (i.e. SSPE) is not significantly large. Therefore, we consider the given model is not very different from the ”best” model in terms of their SSEs. • However, no lack of fit does not ensure that the given model is an useful model for the purpose of prediction. • Let us know at the example in the next slide.

19

No lack of fit does not mean the model must be significant. • Let us look at the following example with 6 pairs of repeated measurements • The scatter plot of these observations are shown on the right. • The observed F-value for the significance of the straight line model 𝐸 𝑦 = 𝛽 + 𝛽 𝑥 is 0.004 which is smaller than the critical value 𝐹. (1,10) or 4.96. Hence the straight line model is insignificant. That is we do not reject the null hypothesis that the slope 𝛽 is 0. • On the other hand, the observed F-value for the lack of fit test is 1.326 which is less than the critical value 𝐹. (4,6) or 4.53. Hence there is no lack of fit for the straight line model. In other words, we cannot find a model better than the straight line model which has a zero slope and it means that 𝑦 and 𝑥 are not related. Note: • 𝑆𝑆𝐸 = 30.4028 with 10 degrees of freedom • 𝑆𝑆𝑅 = 0.0119 with 1 degree of freedom • 𝑆𝑆𝑃𝐸 = 16.138 with 4 degrees of freedom • Hence 𝑆𝑆𝐿𝐹 = 14.265 with 6 degrees of freedom

20

Section 4.5 Example 2 • The marketing department for a large manufacturer of electronic games would like to measure the effectiveness of different types of advertising media in promotion of its products. • specifically, two types of media are to be considered: radio and television advertising, and newspaper advertising (including the cost of discount coupons).

21

• A sample of 22 cities with approximately equal populations is selected for the study during a test period of 1 month. Each city is to allocate a specific expenditure level for both types of advertising. • The sales for electronic games during the test month are recorded in the table in next slide.

22

• The table shows the sales 𝑦, the TV/ radio advertising expenditure 𝑥 and the newspaper advertisement expenditure 𝑥 . • Note the first 2 observations are considered as repeated measurements as both observations have the same 𝑥 values and 𝑥 values. • Notice that there are altogether 11 groups of repeated measurements in this data set.

23

• The table shows how the pure error sum of squares is computed 

 • Sum of squares in each subgroup equals ∑ 𝑦 − 𝑦    ∑ 𝑦 − 𝑛 𝑦 .



which is the same as

• For the first group of repeated measurements with 𝑥 = 0 and 𝑥 = 20, the sum of the squares of the deviations of each observations and the group mean is 1.0658. • since there are 2 observations in this group, therefore the degrees of freedom is 2 – 1 = 1. • Similarly, we compute the sum of squares of deviations for other groups • Hence the pure error sum of squares equals sum of all the sum of squares of deviations for all subgroups and it equals 12.45585 with 11 degrees of freedom

24

• It can be shown that SSE equals 18.12167 with 19 degrees of freedom • Therefore the lack of fit sum of squares, 𝑆𝑆𝐿𝐹 = 𝑆𝑆𝐸 − 𝑆𝑆𝑃𝐸 = 18.12167 − 12.45585 and it equals 5.66582 with 8 degrees of freedom • Hence the mean square lack of fit equals SSLF divided by its degrees of freedom, 8 and equals 0.7082. • The mean square pure error equals SSPE divided by its degrees of freedom, 11 and equals 1.1323.

25

• Hence the F-value for the lack of fit test equals MSLF divided by MSPE and equals 0.625 • Since the observed 𝐹   value equals 0.625 which is less than the critical value 𝐹. (8,11) which is 2.95, we do not reject H0 and conclude that there is no significant evidence of any lack of fit for the given multiple regression model.

26

Section 4.6 Use of SAS to test for lack of fit • The following SAS program can be used to test the lack of fit of the model in example 2 • First of all, we create a data set called “ch4ex2” in the system. • There are 3 variables in the data set “a”. • They are y, x1 and x2. • The data are typed after the “datalines” statement. • The statistical tool used for lack of fit test in SAS is proc reg • We specify the model y = x1 x2 • Proc reg has an option “lackfit” to perform the lack of fit test • The option “lackfit” is placed in the proc reg statement.

27

• From the partial SAS output, there are 2 more rows between Error SS and Corrected Total SS. They are the Lack of Fit SS and the Pure Error SS • 12.456 is the pure error sum of squares which has 11 degrees of freedom. • The observed F-value for lack of fit test is 0.63 • and the corresponding p-value equals 0.74 • Since the p-value is larger than 5%, we do not reject the null hypothesis that there is no lack of fit.

28

• The first table shows the 𝑅  , adjusted 𝑅  , root mean squared error and so on. Root MSE is used to estimate σ • The second table gives the estimate of the 𝛽’s and their standard error. It also displays the t-test for testing 𝛽 = 0 and the corresponding p-value for the t-test. • Both tables are the usual output that we have in executing proc reg.

29

Section 4.7 Use of R to Test for Lack of Fit • First of all, we import the data set ch4ex2 and called the data frame ch4ex2 • Secondly, we fit the given model. The R command is lm(y ~ x1+ x2). In this case, the model is that 𝑦 = 𝛽 + 𝛽 𝑥 + 𝛽 𝑥 + 𝜖. • We apply the function “aov” to the object “model1” which stores the results for fitting the model 𝑦 = 𝛽 + 𝛽 𝑥 + 𝛽 𝑥 + 𝜖. • The SSE for this model is 18.122 with 19 degrees of freedom.

30

• • • • • • • • • • • • • • •

We then fit the best model to the given data set. What is the best model? The best model is to fit a parameter to each group of the repeated measurements Recall that the more parameters in the model, the better fit that we expect How to achieve that? We change the values of the predictors x1 and x2 to categorical variables Each value of the variable becomes one categorical variable. We make use of the function “factor” to create 2 sets of categorical variables fac.x1 and fac.x2. We then use the statement to fit the model y ~ fac.x1 + fac.x2 and store the results in the object “model2” The model y~fact(x1)*fac(x2) is the best model that we can fit to the data. Hence the SSE from this model is the SSPE for the given set of data. This model is the best because it does not assume any functional form to relate 𝐸(𝑦) to 𝑥 value other than the one that identifies the points in the repeated measurement. Use the function “anova” to compare 2 models, model 1 and model 2 In the output, the first entry under RSS (Short form of Residual Sum of Squares) (i.e.18.1217) is the SSE for model 1, given in the first argument of the anova function the second entry under RSS 12.456 is the SSPE (i.e. the SSE for model2 which is the second argument in the function “anova”) the entry under the sum of squares 5.6658 is the lack of fit sum of squares, SSLF which is the difference between SSE and SSPE. the observed F value for the lack of fit test is 0.6254 with a p-value 0.74. Since the p-value is greater than 5% , therefore there is no significant evidence against the hypothesis of no lack of fit.

31...