G*Power Notes - Instructions PDF

Preview

Summary

Instructions...

Description

C8203 Introduction to Research Design and Analysis Practical session 2: Introduction to using G*Power

This worksheet contains the instructions to use G*Power. You can download G*Power for free from the following website - https://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psychologie-undarbeitspsychologie/gpower.html. Alternatively, most computers on campus have it installed. During this practical you will learn how to use G*Power to calculate how many participants you have to recruit when you are carrying out some different types of research. You will also learn more about the relationship between effect size, statistical power, significance levels, and sample sizes.

Activity 1. Theory. When you carry out any research study, and you use null hypothesis significance testing to reach a conclusion about whether an effect exists or not, your conclusions are always based on probabilities. This means your conclusions may be wrong, and there are two ways in which you can be wrong: •

•

You might reject the null hypothesis (i.e., conclude there is a significant effect of your independent variable on your dependent variable) when in fact you should not have done so. This is called a Type I error. Or, you might accept the null hypothesis (i.e., conclude there is no significant effect of your independent variable on your dependent variable) when in fact that’s wrong. This is called a Type II error.

Unfortunately, you can never truly tell if you have actually made one of these two errors! However, the statistics you are using are based on long-run probabilities and this means that we are able to define and control the likelihood that we have made a Type I or Type II error. This principle is what G*Power is based on. Imagine you carried out a study where you compare the academic performance of 55 young people born with a cleft lip to 62 born without a cleft lip. You find that those born with a cleft lip have significantly better academic outcomes than those born without a cleft lip. Could this result be a Type I error? Circle: Y / N. Now explain your answer: YES – that’s what a Type I error is, concluding there is a sig effect when it is not in the population

Could this result be a Type II error? Circle: Y / N. Now explain your answer: NO – Type II errors are errors we can make when we conclude there is no significant effect. Here, we have concluded that there is an effect.

As noted above, a Type I error when you mistakenly reject the null hypothesis (i.e., say there is a significant effect when there is not). Type I error rates are primarily controlled by adjusting the level at which we set significance – that is, whether we accept a difference as significant when our p-value is below .05, or below 0.01, or whatever we decide is appropriate. These numbers have specific meanings: •

p < .05 means: if you find a p-value that is less than .05 then 5% of the time that conclusion will be incorrect (i.e., it will be a Type I error). The 5% simply comes from the fact that 0.05 is 5% of 1 (you can multiple any p-value by 100 to find out what it is as when expressed as a percentage). If you set your significance level at p < .01, then what does that mean? At what rate will you make Type I errors? __1__% of the time the result will be a Type I error if we set the p-value at p < .01.

•

In G*Power, you have an option to decide what level you want to set your significance at. This will be called “α err prob” because Type I errors are called alpha (α) errors.

Type II error rates are also meaningful. Recall that these are instances where you mistakenly conclude that there is no significant effect of your independent variable on your dependent variable. To control Type II error rates, we have to address the statistical power of our study: the more statistical power your study has the less likely it is that you will make a Type II error. •

If we have statistical power set at .80 this means that we are likely to make a Type II error 20% of the time. This is 20% of the time because Type II error rates are always the exact opposite of our statistical power – if statistical power is .80 that means that we are not going to make a Type II error 80% of the time and so we will make a Type II error the rest of the time that’s left: 20%.

If you set your statistical power at .95, then what does that mean? At what rate will you make Type II errors? __5__% of the time the result will be a Type II error if we set the statistical power to be .95.

We are almost ready to use G*Power. There is one last piece of the puzzle that it is important to get to grips with though: effect size. A bigger effect size indicates that your two groups or conditions are very different from one another (or, for a correlation, that scores on both your measures are closely related to one another). Even small effects can be significant if you have a large enough sample, but they may also trivial i.e. ‘significant’ in the statistical sense yet virtually meaningless in the real-world.

2

The aim of using G*Power is to find a significant effect for a given effect size. Usually the effect you want to find is determined in one of two ways: •

•

You can consult previously published work or, if you have it, pilot data. These can give you an estimate of the size of the effect of your independent variable on your dependent variable. You can then use G*Power to see how many participants you need to recruit to detect as significant this estimated effect size. You decide on the smallest meaningful effect you care about finding. Often, a medium-sized effect is the default here. However, you may decide that you are only interested in detecting large effects (e.g., effects that indicate that an intervention is wildly successful). Alternatively, you may be interested in detecting even small effects (e.g., if you were dealing with elite athletes then helping them to make even small performance gains may literally be a gamechanger).

In G*Power, the statistical power option is labelled as “Power (1 – β err prob)” because β is a symbol used to describe Type II errors and because Power is always the inverse of Type II error rate. So if β is 40% (or .40) then Power is 60% (or .60); if β is 30% (or .30) then Power is 70% (or .70), if β is 1% (or .01) then Power is 99% (or .99) etc.

3

Activity 2. G*Power for correlational designs. The following instructions demonstrate how to use G*Power to estimate a-priori sample size for a test of correlation. By a-priori we just mean that we are making this estimate before we start collecting any data. Go to ‘All programs’ from the Start-up menu. Select ‘G*Power’ and open the G*Power application. On the left-hand side, check that ‘Test family’ is set to: ‘t tests’. Set the adjacent ‘Statistical test’ to ‘Correlation: Point biserial model’. The ‘Type of power analysis’ should be set to ‘A priori: Compute required sample size…’. Note here that the effect size option (‘Effect size │p│’) is where you enter a specific correlation coefficient (the r that we see in correlations). The guidelines for effect sizes here are 0.1 = small, 0.3 = medium, and 0.5 = large. To set your significance level (your p-value) use the ‘α err prob’ box. Note as well that there is the option here to specify either a one-tailed or a two-tailed correlation. This should tally up with whether your hypothesis is one-tailed or two-tailed.

What sample size is required if you want to detect a correlation which is of small effect size, significant at the p < .01 level, making Type II errors only 10% of the time, and assuming you have a two-tailed hypothesis? N = 1477

What will happen to the sample size if you only want to detect a large effect? Will it go up or down? Why? Sample size will decrease because it is easier to detect a large effect and so you can detect it as significant even with a small sample

Try changing the effect size to large to see if you were correct.

Keeping the effect size as large, what will the effect be on the required sample size if you change the statistical power to 80% instead of 90%? Why? Sample size will decrease because you are making it more likely that you will make Type II errors (i.e. it is a weaker study, as we intuitively know it to be if there are fewer participants)

Try changing the statistical power to 80% to see if you were correct. [goes down to 1160]

4

Activity 3. G*Power for two-group independent designs. Go to ‘All programs’ from the Start-up menu. Select ‘G*Power’ and open the G*Power application. On the left-hand side, check that ‘Test family’ is set to: ‘t tests’. Set the adjacent ‘Statistical test’ to ‘Difference between two independent means (two groups)’. The ‘Type of power analysis’ should be set to ‘A priori: Compute required sample size…’. This is the sample size calculation you would use if you were going to use an independent groups t-test to analyse your data. Note here that the effect size option (‘Effect size d’) is where you enter a Cohen’s d value. These represent effect size as the number of standard deviations that the two means are apart from one another. The guidelines here are 0.2 = small, 0.5 = medium, and 0.8 = large. Note as well that there is the option here to specify either a one-tailed or a two-tailed effect. This should tally up with whether your hypothesis is one-tailed or two-tailed. To set your significance level (your p-value) use the ‘α err prob’ box. Finally, there is something asking for ‘Allocation ratio N1/N2’. Since you have two groups of participants, this is asking for your expected ratio of participants in group 1 to participants in group 2. Unless you have a reason to do otherwise, leave this a ‘1’ which means you are expecting to recruit equal numbers into each of your two groups.

What sample size is required if you want to detect a medium-sized difference between two groups, significant at the p < .05 level, making Type II errors only 5% of the time, and assuming you have a two-tailed hypothesis? N = __210____ ( ___105__ in each group)

What will happen to that sample size if you reduce your alpha from .05 to .01? Why? Sample size will increase because you are aiming to make fewer Type II errors – in order to make fewer Type II errors you need more participants because this gives you more accurate estimates of the differences between groups.

Try changing the alpha to .01 to see if you were correct. [goes up to 290]

5

Activity 4. G*Power for two-condition repeated-measures designs. Go to ‘All programs’ from the Start-up menu. Select ‘G*Power’ and open the G*Power application. On the left-hand side, check that ‘Test family’ is set to: ‘t tests’. Set the adjacent ‘Statistical test’ to ‘Difference between two dependent means (matched pairs)’. The ‘Type of power analysis’ should be set to ‘A priori: Compute required sample size…’. This is the sample size calculation you would use if you were going to use a paired-samples t-test to analyse your data. Note here that the effect size option (‘Effect size dz’) is where you enter a Cohen’s d value. These represent effect size as the number of standard deviations that the two means are apart from one another. The guidelines here are 0.2 = small, 0.5 = medium, and 0.8 = large. Note as well that there is the option here to specify either a one-tailed or a two-tailed correlation. This should tally up with whether your hypothesis is one-tailed or two-tailed. To set your significance level (your p-value) use the ‘α err prob’ box. What sample size is required if you want to detect a small-sized difference between your two conditions, significant at the p < .05 level, making Type II errors only 5% of the time, and assuming you have a two-tailed hypothesis? N = __327___

What will happen to that sample size if you decide you can make a one-tailed hypothesis instead of a two-tailed hypothesis? Why? Sample size will decrease because you are only aiming to detect effects in a single direction, and therefore you need a smaller sample since a lower difference score will now be significant.

Try changing your test from two-tailed to one-tailed to see if you were correct. [goes down to 272]

End: Writing up the results of your G*Power analysis. In the report of your research, you will need to explain how you did your a-priori sample size analysis using G*Power. Please see the Report Writing Guide for details of how to do this.

6...