Week 1 - Experimental and Nonexperimental Design PDF

Preview

Summary

Download Week 1 - Experimental and Nonexperimental Design PDF

Description

RESEA RCH METH ODS IN P SYCH OLOG Y B

TOPIC1

TOPIC 1

Experimental and Non-experimental design P R E PA R E D B Y T H E U N I T T E A M

Contents Experimental and Non-experimental design Introduction Experimental Designs Research strategies Internal validity and threats External validity and threats Balancing internal and external validity and additional threats Other threats to validity; and Open Science Experimental research and controlling for extraneous variables Between-subjects designs Within-subjects designs Non-Experimental Designs Correlational designs Quasi-experimental designs Non-equivalent group designs Developmental research designs Answers to review questions

1 © Deakin University

1 2 2 2 3 4 4 5 5 6 6 7 7 8 8 9 10

RESEA RCH METH ODS IN P SYCH OLOG Y B

TOPIC1

Introduction Internal and external validity of a study. Consideration of internal and external validity also helps you understand the strengths of experimental design, in relation to nonexperimental study designs. We will focus on two main types of experimental studies: between-subjects designs and within-subjects designs. In the second half of this topic you will cover three main types of non-experimental research designs: correlational research, quasi-experimental designs, and specific developmental research designs.

Experimental Designs Research strategies A research strategy is determined by the kind of question that the researcher hopes to answer. Both experimental and quasi-experimental research are designed to address if there is a cause-effect relationship(因果关系) between two variables. However, quasiexperiments cannot show unambiguous( 明确的) evidence for a cause-effect relationship. Non-experimental and correlational studies are designed to examine if there is a relationship between two variables. Finally, descriptive studies, as the name suggests, focus on the description of individual variables. Five approaches to research: 1

Experimental research

Two main type: Between & Within, Random assignment of objects to groups, Control of extraneous variables, controls help to eliminate alternation cause. (For the relationship between IV and DV) Strengths: • Allow for assessment of causality and directionality • (Can have) high internal validity Weaknesses: • Low external validity (experimental settings are often a poor representation of the real world) • Only applicable where randomisation is possible/ethical • Can be difficult/time consuming

2

Quasi-experimental research(准实验研究)

lack of randomisation, Some attempt at “control”, 没有随机分配实验对象到实验组和控 制组，严谨性略低，因而所产生的因果结论的效度比真正的实验研究低，但优点 在于所要求的条件灵活，在无法控制所有可能影响实验结果的无关变量时，具有 广泛的应用性。Eg: school-based intervention (class level, grade level, not individuals), male & female 3

Non-experimental research

Examine nature of observed relationships between groups (e.g. pubertal timing and grade differences) 4

Correlational research

Both variables are continuous, Examine nature of observed relationships between two variables (body dissatisfaction and self-esteem) Problems: third variable(第三变量), directionality problem(which cause which) 5

Descriptive research 2 © Deakin University

RESEA RCH METH ODS IN P SYCH OLOG Y B

TOPIC1

Observation, case study, describing individuals

Internal validity and threats As defined by Gravetter and Forzano (2012, p.167-168), the internal validity of a study or experiment “is the degree to which the study accurately answers the questions it was intended to answer.” 确实性程度，实验结论的真实性 And a threat to internal validity is any aspect of the research which “raises doubts about the limits of research results or about the interpretation of the results.” questions or doubts about the study Two of the main threats to internal validity include environmental variables (time of testing) and assignment bias (the groups are not equal, personality). Five other main threats to internal validity that need to be evaluated for designs that compare groups over time include history, maturation, instrumentation, testing effects, and regression toward the mean (these concepts are covered in the readings below and in your lectures). Optional Reading Then answer the following questions: 1

What is an extraneous variable?

Any variable in a research study other than the specific variables being studied is an extraneous variable. 2

What is a confounding variable 混淆变量?

A confounding variable is an extraneous variable (usually unmonitored) that changes systematically along with the two variables being studied. A confounding variable provides an alternative explanation for the observed relationship between the two variables and, therefore, is a threat to internal validity. 混淆变量是在研究中无法控制的变量 (uncontrolled variable)，有时亦称为「额外变量」(extraneous variable)，这些变量可能 会影响到自变量与因变量 3

What is assignment bias?

Assignment bias occurs when the process used to assign different participants to different treatments produces groups of individuals with noticeably different characteristics. 4 List and explain five time-related threats to internal validity.(examing groups over time): History is a threat to internal validity because any differences that are observed between treatment conditions may be caused by history instead of by the treatments. Maturation is a threat to internal validity because observed differences between treatment conditions may be caused by maturation instead of by the treatments.

Instrumentation refers to changes in the measuring instrument that occur during a research study in which participants are measured in a series of treatment conditions. Instrumentation is a threat to internal validity because any observed differences between treatment conditions may be caused by changes in the measuring instrument instead of the treatments. Testing effects, also known as order effects, occur when the experience of being tested in one treatment condition (participating and being measured) has an influence on the participants' scores in a later treatment condition(s). Testing effects threaten internal validity because any observed differences between treatment conditions may be caused by testing effects rather than the treatments. Statistical regression, or regression toward the mean, is a mathematical phenomenon in which extreme scores (high or low) on one measurement tend to be less extreme on a second measurement. Regression is a threat to internal validity because changes that occur in participants' scores from one treatment to the next can be caused by regression instead of the treatments. 3 © Deakin University

RESEA RCH METH ODS IN P SYCH OLOG Y B

TOPIC1

External validity and threats As defined by Gravetter and Forzano (2012, p.168), the external validity of a study or experiment “refers to the extent to which we can generalise the results of a research study to people, settings, times, measures, and characteristics other than those used in that study.” And a threat to external validity “is any characteristic of a study that limits the generality of the results.” Three main kinds of generalisations are covered by Gravetter and Forzano. These include: 1

Generalisation from sample to the general population

2

Generalisation from one research study to another

3

Generalisation from a research study to a real-world situation

In addition, three general threats to external validity are covered: 1

Participant characteristics (e.g., selection bias, restricting participants to age)

2

Features of the study (e.g., reactivity, experimenter characteristics)

3

Measurements (repeated measurement, use of specific measures)

Reading Now read Gravetter and Forzano (2012), pp. 171–176 Then answer the following questions. 1

What is selection and volunteer bias?

the sampling procedure favors the selection of some individuals over others. It should be obvious that selection bias is a threat to external validity. volunteers are not perfectly representative of the general population 2

How does selection bias limit the external validity of a study?

The question of external validity is always raised when a researcher selects participants based on convenience rather than using an unbiased selection process 3

What is the novelty effect?

Act differently in a new situation 4 How can the novelty effect limit the external validity of a study? 5

How can experimenter characteristics limit the external validity of a study?

6

What is assessment sensitisation?

7

What are the three ways in which measurement issues can threaten the external validity of a study?

Balancing internal and external validity and additional threats Researchers often need to make a trade-off between internal and external validity. One of the problems with increasing internal validity is that this often results in a reduction in external validity and vice versa. Also, it would not be possible to address all threats in one study. Therefore, the researcher needs to determine which type of validity and which threats are more important for any given study. 4 © Deakin University

RESEA RCH METH ODS IN P SYCH OLOG Y B

TOPIC1

Two additional threats to both internal and external validity covered by Gravetter and Forzano are experimenter bias and demand characteristics/participant reactivity. Reading Now read Gravetter and Forzano (2012), pp. 187–188 and make notes to understand how these factors can affect both internal and external validity.

Other threats to validity; and Open Science Other threats to the validity of research that do not often get discussed pertain to the honesty or self-deceptions of researchers themselves. These threats include fraud, deception and self-deception. While generally unlikely, there is the possibility of a researcher making up data and therefore outright fraud, but also the probably more common possibility that the research is unintentionally “slanted” (Navarro, 2019) through means such as data fabrication, hoaxes, misrepresenting data, not reporting necessary study design details, data mining (analysing data without ‘a priori’ hypotheses until a significant finding emerges), and publication bias (only publishing studies with significant findings). These additional threats are discussed in more detail in an optional reading, see section 2.7.12 of Danielle Navarro’s freely available ‘Learning statistics with R: A tutorial for psychology students and other beginners’ One partial solution to such threats is the recent Open Science movement. Doing Open Science means conducting research in a collaborative way, where your hypotheses, data, methods, or descriptions of your analyses (typically syntax) are made freely available. Making hypotheses available can be done by way of an important process called preregistration. Pre-registration entails specifying your research plan in advance of gathering data, and can be done on a large number of online tools. Following such Open Science practices increases the transparency and openness of the research process, and means that research can be replicated and far more easily evaluated by other researchers in order to better evaluate the validity of findings.

Experimental research and controlling for extraneous variables Experiments are designed to show if a variable manipulated by a researcher (independent variable) causes changes in a second variable (dependent variable). In addition, it is also necessary to demonstrate that no other variables differentiate the experimental and control groups and so could be responsible for the correlation found between the IV and DV. This is done by making conditions in the two groups as near to identical as possible, apart from the experimental manipulation. (e.g., use the same testing procedure, testing environment, experimenter, etc.). This is referred to as controlling extraneous variables. An important aspect of experimental research is to prevent an extraneous variable from becoming a confounding variable. In the context of an experiment, a confounding variable is one that affects the dependent variable and varies systematically with the independent variable. The three main types of extraneous variables are: 1

environmental (e.g., time of testing, different rooms)

2

participant (e.g., gender, age, IQ) 5 © Deakin University

RESEA RCH METH ODS IN P SYCH OLOG Y B

3

TOPIC1

time-related (e.g., weather changes, becoming fatigued)

One way of controlling for environmental extraneous variables is by holding these constant (e.g., all subjects have the same instructions, experimenter, testing room, etc.). Some participant variables such as gender and age can also be controlled by holding these constant. For example, a researcher can focus on one specific age group or one specific gender. Other participant variables can be held more or less constant by restricting their range (e.g., IQ). Another way of controlling all three types of extraneous variables would be to balance or match these across each level of the independent variable (e.g., treatment group). For example, a researcher can ensure that there are the same number of males in each treatment. If the researcher has two testing rooms, then an equal number of participants from each treatment need to be tested in the two rooms. Similarly, in order to control for time-related variables, a similar number of participants from each treatment can be tested early and others can be tested later. The final method for controlling participant variables is the use of random assignment. When subjects are randomly assigned to groups then there is a greater likelihood that participant variables will be distributed evenly across the groups and any participant differences between groups should even out. This method becomes more effective with larger samples but cannot be guaranteed with small numbers. The process of randomisation can also be used to control for some environmental variables. For example, a researcher may have to use a different testing room on some days. Each time this happens a coin can be tossed to determine whether treatment 1 or treatment 2 is assigned to the different room. As both groups are as likely to be assigned to this different room, the extraneous variable will be randomly distributed across the groups. Clearly, this would not be an easy way to control for all environmental variables. Therefore, it is more likely that a researcher would select to hold environment variables constant or balance these across conditions, as described earlier.

Between-subjects designs One main type of experimental design is the between-subjects design (alternatively known as the independent-measures design). In addition to meeting the conditions already listed in the previous section, the other defining feature of this type of experiment is that a participant can only take part in one treatment condition. 一组一个 Advantages • always an option • no practice or fatigue effects across treatments • having participated in one arm of the study doesn’t influence how you participate in another arm Disadvantages • require larger numbers • large individual differences can obscure treatment effect

Within-subjects designs The other type of experimental design is the within-subjects design or also known as the repeated-measures design. Unlike in the between-subjects design, there is one group of participants, and all participants take part in each of the treatment groups. 一组几个 The following are the main advantages of within-subjects designs: 1

They require fewer participants. 6 © Deakin University

RESEA RCH METH ODS IN P SYCH OLOG Y B

2

TOPIC1

They eliminate the variability due to individual differences, which in turn makes the design more powerful (more likely to detect a treatment effect).

However, the within-subjects design also has some disadvantages: 1

Within-subjects designs cannot be used effectively if there are carryover effects (that is if doing one condition will alter the way subjects perform in another conditione.g., long lasting drug effects).

2

Within subject designs are subject to time-related factors (e.g., fatigue, practice).

3

Within-subjects designs may not be practical if that study extends over a long period of time (e.g., too demanding for subjects, participant attrition).

In order to control for carry-over effects, an experimenter could use a between-subjects design. However, order effects that may be due to practice or fatigue can be controlled by counterbalancing. Counterbalancing 平衡抵消法 involves administering treatments in a different order “so that every treatment has some participants who experience the treatment first, some for whom it is second, some third, and so on” (Gravetter & Forzano, 2012, p. 266). Because participants complete the treatments in a different order, any effect of the independent variable cannot be attributed to the timing or order of the conditions. In addition, any time-related effect would be balanced out across the conditions. Two main types of counterbalancing: One is complete counterbalancing. In complete counterbalancing the treatments are presented in every possible order. However, this becomes impractical if you have more than four treatments. A more practical method is partial counterbalancing. Partial counterbalancing uses different orders to ensure that each treatment occurs in each position (first, second and so on). It is also essential that each different order is used the same number of times (i.e., participants need to be in multiples of the number of treatments). Finally, to also eliminate any order effects, you need to ensure that each condition does not always follow or precede another. Advantages: • control of participant and environmental variables across conditions • elimination of individual differences makes the design more powerful than between subjects designs • more powerful so require fewer subjects Disadvantages • subject to time-related factors (long term or short term) • may not be practical if study extends over a long period of time or if the demand of subjects is too high (participant attrition) • can not be used effectively with one type of order effects: carryover effects

Non-Experimental Designs Correlational designs This type of research primarily examines whether there are relationships between variables, and then seeks to describe the relationship in terms of strength of association, direction of relationship (positive or negative), and linearity of this relationship. Involves measuring more than two variables but usually involves relationships between two variables at a time. (e.g. Narcissism and aggression) A correlational study does not involve manipulating, controlling, or interfering with variables. Instead, the researcher simply measures two different variables for each individual. The researcher then looks for a relationship within the set of scores. 7 © Deakin University

RESEA RCH METH ODS IN P SYCH OLOG Y B

TOPIC1

Correlational study views the data as two scores, X and Y, for each individual, and looks for patterns within the pairs of scores to determine whether there is a relationship. Reading Read Gravetter and Forzano (2012), Chapter 12 (pp. 344; 350–351; 355-357)

Quasi-experimental designs Quasi-experimental designs, like the name suggests, are designs intended to approximate experiments, and they are typically conducted in applied settings. The researcher’s goal is to show a causal relationship b...