Lecture #6 Design-Related Confound II PDF

Preview

Summary

Download Lecture #6 Design-Related Confound II PDF

Description

Design-Related Confound II Design-Related Confounds (Recap) 1. Design-related confounds arise from decision about the design of the study. For example: a. Type of IV: PAV vs. manipulated b. How IVs are varied: within-participants vs. between-participants c. How participants are assigned to conditions of between-participants IVs: random vs. non-random 2. Design-related confounds can be classified into various types 3. Last week a. Person-variable confounds b. Differential attrition c. Testing confounds d. Sequencing confounds 4. This week a. History confounds b. Maturation confounds c. Instrumentation change d. Statistical regression artefacts

History Confounds 1. History Confounds are: a. Time-dependent and place dependent changes that: i. Happen to occur during the course of the study ii. And provide an alternative explanation for the results 2. Time-dependent and place-dependent changes can occur over:

a. The medium-term to long-term. For example: i. Effects of economic, social or legal changes ii. Effects of natural disasters, terrorist attacks, or war b. Or the short-term (especially in laboratory studies). For example: i. Construction noise during a laboratory task 3. History confounds can affect: a. Within-participants IVs (mainly) b. Between-participants IVs

History Confounds (Wiki) 1. Events outside of the study/experiment or between repeated measures of the dependent variable may affect participants' responses to experimental procedures. 2. Often, these are large-scale events (natural disaster, political change, etc.) that affect participants' attitudes and behaviours such that it becomes impossible to determine whether any change on the dependent measures is due to the independent variable, or the historical event.

History Confounds: Within-Participants Example 1. Effects of improving US-Russian relations on attitudes towards George Bush 2. Same participants surveyed early in 2001 and then again in early 2002 3. Potential history confound (unpreventable): 9/11 attacks 4. Was the increase in Bush’s popularity due to relations with Russia or public reaction to 9/11 and the aftermath

History Confounds and Longitudinal Designs 1. History confounds are a common and severe problem with longitudinal designs

2. Longitudinal designs: a. Test the same participants over long time periods b. Often used to assess developing/aging c. Avoid cohort effects (general between-participants history effects) d. I.e. differences in demographically similar groups of people because of changes over time in technology, lifestyle, standards of education, etc.

History Confounds: Within-Participants IV Example 1. Longitudinal studies of aging/development a. Issue: what social, political, cultural, technological, etc. changes are relevant? b. There is a major problem if the effects of these changes are attributed to normal aging/development 2. Specific example: a. Longitudinal study of the effect of aging on job satisfaction from 1970 (age 20) to 2015 (age 65; retirement) b. If job satisfaction drop between age 55 (2005) and age 65 (2015), is it an effect of aging? c. Job satisfaction may have declined because of actual decline in working conditions i. For example, due to history confounds such as the Global Financial Crisis d. Can’t conclude that job satisfaction normally declines in the pre-retirement years

Prevention of History Confounds in Longitudinal Studies of Development (WithinParticipants) 1. Use a time-lagged design (the “cohort-sequential approach) 2. This is separate out the effects of aging/development from history factors

History Confounds: Between Participants Example 1. Short-term lab study on the effect of an imagery encoding strategy on recall of details about work of art 2. 2 groups: imagery strategy vs. control (no strategy) 3. Potential history confound a. Building-noise during test impairs performance of the control group, but does not occur while the imagery group are being test 4. A study of the effect on job satisfaction of differing management practices in two similar factories in neighbouring towns a. Potential history confound: Fire, fraud scandal, etc. at the factory predicted to score poorly on satisfaction

Prevention of History Confounds for Between Participants 1. Randomly assign participants to groups a. History factors affect groups equally b. See Assignment x History interactions 2. Obtain DV scores concurrently for the groups 3. Obtain DV scores in multiple sessions rather than testing everyone in a condition together a. Also use multiple setting of they co-vary with the IV (see next slide) b. For example, this could be applied to the building noise example

Prevention of History Confounds for Between Participants With Different Settings 1. When different setting must be used in different conditions, need to take other measure to improve internal validity 2. For example, study the effects of management practices at different factories a. Replicate each condition over several factories b. So you get several factories that have the same management practices and several control factories that have don’t have the same management practices 3. For example, a lab study where different labs/settings must bed used for each group a. Conduct multiple waves of testing at different times (so a history factor can’t have a systematic effect)

Assignment x History Interactions 1. Groups that differ on person variables may react differently to history factors 2. From Lecture 5: If an IV is confounded by a person-variable, the confound may interact with other IVs (or with potential confounds of other IVs) a. This includes history factors 3. Example: Screeching microphones (Recap) a. Treatment to reduce public speaking anxiety b. Group IV: person-variable confound – control group more anxious than the treatment group (non-random assignment) – so this is due to the way you assigned the groups c. Test IV: History factor – microphone screeching only occurs at post-test d. Apparent Group x Test interaction may have been Assignment x History interaction

i. So, the control group were more anxious simply because of their personal characteristic of brought the characteristic of being anxious

4. Notes from lecturer a. If the two different conditions potentially differ on a person-variable, then if you do have some confound such as a history confound b. If it effects both groups then you can’t be confident that both groups effected the same way if those groups are not comparable on personvariables. c. So if you have randomly assigned to groups and there is a change in the stock market or something and that affects people’s willingness to invest money (as an example) d. If you have random assignment, it will affect both groups equally. e. However, if those two groups are not comparable on person-variable confounds, which may be an issue if you are not able to randomly assign,

there is more than likely chance to be an issue if you have a PAV variable involved f.

Then you have to consider the possibility that the history confound won’t have the same impact on both groups because of their person variable characteristics

g. If you have any person-variable confound on a between-participants IV, this will have an impact on the interpretation of the results for that particular IV. It will also affect/contaminate interpretations of interactions with that IV h. Or because if the effects of how the how the IV work for people who differ in person-related variables or because of confounds those other IVs effecting groups differently i.

Last week we talked about the example of practice. If you have one group that is more able/skilled than another group, the more able group will show more practice with practice with experience of a task than the less able/skilled group

Prevention of History Confounds and Assignment x History Interactions in Pre-Post Designs 1. If possible a. Include a comparison group that receives no treatment b. Randomly assign participants to groups; and c. Obtain DV scores concurrently for the groups 2. Changes due to time-dependent history factors will affect the groups equally a. Hence, any differential effect of the treatment will be evident as a Group x Treatment interaction b. For example, could be applied to the screeching microphone example

Maturation Confounds 1. Maturation confounds are time-dependent change that a. Occur within participants during the course of the study b. And provide an alternative explanation for the results: i. I.e. if most participants in a study (or group) show a maturational change in the same direction 2. NOT due to tasks, procedures or settings 3. Maturation confounds can affect: a. Within-participants (mainly) b. Between-participants 4. Maturation changes can affect: a. Short-term studies b. Long-term studies (more often)

Maturation Confounds (Wiki) 1. Subjects change during the course of the experiment or even between measurements. 2. For example, young children might mature and their ability to concentrate may change as they grow up. Both permanent changes, such as physical growth and temporary ones like fatigue, provide "natural" alternative explanations; thus, they may change the way a subject would react to the independent variable. 3. So upon completion of the study, the researcher may not be able to determine if the cause of the discrepancy is due to time or the independent variable.

Maturation: Short-Term Study Example 1. Participants complete a mental arithmetic task with a concurrent memory load.

a. IV = memory load (small vs. large). b. DV = arithmetic task performance. 2. Different conditions are tested at a different time of day (e.g., before vs. after lunch). a. Time of day may affect hunger and alertness, either of which could influence performance and confound the study. b. The same applies whether memory load is varied within- or betweenparticipants. 3. Prevention: hold time of day constant (B-Ps/W-Ps), counterbalance it (W-Ps), or vary it factorially (add as IV).

Maturation: Long-Term Study 1. Maturational changes that can threaten interpretation include: a. Changes in health. i. Especially cyclic conditions like back pain, depression, allergies, etc.. b. Developmental changes: i. Myelination of brain neurons & associated cognitive development ii. Puberty. iii. Cognitive/perceptual changes in old age. iv. Life-stage changes, etc. 2. Maturation can be an issue in longer-term studies where a comparison group is absent or inappropriate.

Maturation: Long-Term Study Health Example 1. Evaluation of treatment for a cyclic condition. 2. Pre-post design with no comparison group: X A X a. Potential confound if Ps enter the study at the same stage in the trajectory

i. For example, when depression or back pain is most severe. ii. For example, in the season when allergies are most severe. 3. Improvement may be due to seasonal changes in allergies, or cyclical changes in back pain or depression. a. If untreated, the condition is likely to improve over time for most. b. Experimenter may attribute improvement to the treatment. c. So, the problem here is that, time was the confound and not the treatment. As time went on they got better regardless of the treatment. This is because things like depression and back pain fluctuate with severity. Therefore, one solution could be random assignment so we can factor out time as the cause of the improvement in pain/depression

Maturation: Long-Term Study – Developmental Example with No Comparison Group

1. Within-Ps evaluation of logical reasoning training: X A X 2. Administer training in logical reasoning to children over a six-month period. 3. Results: Substantial improvement over the course of the program. 4. Potential confound: Is improvement in reasoning due to the program, or to normal cognitive maturational changes?

Assignment x Maturation Interaction 1. May be an issue, even when there is a comparison group, if the comparison group is inappropriate. a. I.e., if assignment to groups is non-random. 2. Group (Treatment vs. No Treatment) x Test (Pre- vs. Post-) interaction may occur a. Because a person-variable confound causes one group to be affected more than the other by a confound of the Test IV (i.e., maturation).

Assignment x Maturation Interaction: Health Example 1. Within-Ps evaluation of treatment for a cyclic condition. 2. Pre-post design with non-random assignment to a comparison group. a. Potential confound if different groups enter the study at different stages in the trajectory. b. E.g., depression treatment group are at the bottom of their mood cycle when treatment commences; comparison group are not. c. E.g., because Ps who sign up first (and feel worse) are assigned to the treatment condition. 3. Differential improvement may occur because participants whose condition is worst are the most likely to improve. Hence, may show greater improvement than controls

Assignment x Maturation Interaction: Developmental Examples 1. Evaluation of blood-pressure drug. a. 6-month study of the effects of a blood-pressure drug on memory in an elderly (70 +) group. b. Non-random assignment to control group – i.e., a young university student placebo-control (but reliably matched on memory performance at pre-test). c. Potential Confound: If the treatment group suffer a decline in memory over the course of treatment (while the control group do not) i. Is it due to the drug, or due to normal aging? 2. Evaluation of Extra-Curricular Maths Instruction Program for Grade 8 students. a. Semester-long study comparing volunteers for extra maths instruction vs. a control group who did not participate. b. In terms of maturation, unequal gender make-up of the groups could provide an alternative explanation the effect of the program. i. Potential maturation confound if the proportion of girls (vs. boys) is greater in instruction group than in control group.

ii. Why? Boys mature later than girls. iii. Hence, there may be more maturation occurring in the instruction group than in the control group at that age.

Pre-Post Design: Preventing Maturation Confounds and Assignment x Maturation Interactions 1. If possible: a. Include a comparison group that receives no treatment; b. Randomly assign participants to groups; and c. Obtain DV scores concurrently for the groups. 2. Changes due to maturation factors will affect the groups equally. a. Hence, any differential effect of the treatment will be evident as a group x treatment interaction. b. E.g., could be applied to the depression & maths instruction examples.

Instrumentation Change 1. Instrumentation change is: a. A shift in measurement over time in the direction predicted by hypothesis, b. Resulting in apparent change in the DV (and not to do with real changes with behaviour) i. C.f. maturation, history & sequencing: real change. 2. Instrumentation change can affect: a. Within-Ps IVs (mainly). b. Between-Ps IVs (if the groups are tested at different times). 3. Instrumentation change can result from: a. Equipment failure.

b. Observer criterion shifts.

Instrumentation Change (Wiki) 1. The instrument used during the testing process can change the experiment. 2. This also refers to observers being more concentrated or primed, or having unconsciously changed the criteria they use to make judgments. 3. This can also be an issue with self-report measures given at different times. In this case the impact may be mitigated through the use of retrospective pretesting. 4. If any instrumentation changes occur, the internal validity of the main conclusion is affected, as alternative explanations are readily available.

Equipment Failure 1. In psychophysiological & cognitive measurement: a. Electrodes can become unglued & lose contact. b. Computers or recording devices can malfunction. 2. But it’s rare that problems like these occur systematically to produce results consistent with the hypotheses. 3. Prevention: Test apparatus regularly.

Observer Criterion Shifts 1. Much more common & insidious! 2. Observer criterion shifts are a problem that can arise when people are the measuring instruments (because it can be subjective and not objective) a. This is often the case with behavioural observation (i.e., classifying and counting actions, utterances, etc.). For example: i. Aggressive verbalizations.

ii. Tantrums. iii. Self-denigrating statements. iv. Provocative actions. v. Nail-biting, etc. 3. Why do observer criterion shifts occur? a. Observing behaviour on-line (in real time) is difficult. i. E.g., issues of interpretation. ii. E.g., difficulties in accurately catching frequent or subtle behaviours. b. With experience, observers may become more skilled at identifying behaviours of interest. i. E.g., The observer may become more practiced at picking up on subtle behaviours (so, it may not be the actual behaviour has increased in frequency, it’s that the experimenter is getting better at picking up the behaviour) c. The criteria they apply when classifying behaviours may become more (or less) stringent over time.

Observer Criterion Shifts – Tantrums 1. Evaluation of program designed to reduce child tantrums a. Baseline and post-test observations of tantrums among children in a kindergarten playground. 2. With experience, the observer becomes better at distinguishing tantrums from other similar behaviours. 3. Potential confound: If the observer’s criterion for a tantrum gets more stringent over time a. The number of recorded tantrums will go down (as predicted)

b. Even if there is no change in the child’s tantrum behaviour

Observer Criterion Shifts: Prevention 1. Use well-trained, practiced observers operating with clear-cut criteria (i.e., precise operational definitions). 2. Use multiple baseline measures? a. So, you would get the observer to come in a few times for the baseline measurement and then average on the baseline b. Gives the observers more practice before the treatment. i. Does not remove instrumentation change effects entirely, but makes them somewhat less likely to confound the study. 3. Have multiple observers and assess their agreement? a. Does not necessarily fix the problem! b. Reliability may be confirmed, but i. It is possible that most observers get more stringent as they get more experienced with the behaviour. c. Also, just because they all agree, doesn’t mean they are all right

Prevention of Instrumentation Change Confounds in Pre-Post Designs 1. If possible: a. Include a comparison group that receives no treatment; b. Randomly assign participants to groups; and c. Obtain DV scores concurrently for the groups. 2. Changes due to instrumentation change will affect the groups equally. a. Hence, any differential effect of the treatment will be evident as a group x treatment interaction.

b. E.g., could be applied to the tantrum-reduction program example.

Statistical Regression Artefacts 1. A statistical regression artifact: is an apparent change in the DV from pre-test to post-test; a. I.e. when no true change has occurred (as with instrumentation change). b. Only occurs when participants are initially selected on the basis of extreme scores on: i. An imperfectly reliable measure, ii. Which is also the DV (or a correlate of the DV); and, c. Results from systematic measurement error. i. I.e., extreme scorers will show a shift towards their true mean on a repeated test with the same measure (or a correlate) – i.e., from extreme to less extreme. 2. Statistical regression can affect: a. Within-Ps IVs (mainly). i. I.e., affects the magnitude of pre-post changes. 3. Between-Ps IVs (if the groups are chosen on some measure prior to the study). a. I.e., affects group differences and test x group interactions. i. When groups show differential regression. 1. May occur if the groups are not comparable (non-random assignment).

Statistical Regression Artefacts (Wiki) 1. This type of error occur...