Lecture #7 Procedural Confounds PDF

Preview

Summary

Download Lecture #7 Procedural Confounds PDF

Description

Procedural Confounds Procedural Confounds 1. Procedural confounds are not primarily the result of design decisions a. A study free of potential design-related confounds may still be rendered uninterruptable by procedural confounds 2. Procedural confounds are alternative explanations that are introduced in the researcher’s procedures. For example a. Manipulations b. Tasks c. Materials d. Measures e. Instructions, etc. 3. Note: This can include procedures not specific to a particular variable. For example, a. General instructions that may produce participant expectancy effects

Operations Confounds (Competing Construct) 1. An operations confound is an alternative explanation (specifically, a competing construct) that is introduced in the procedures used by the experimenter to operationalize a construct a. I.e. the particular manipulations, tasks, etc. and the way they are implemented 2. For example, trauma vs. handling (competing construct) in the Puppy Study (Lecture 5)

Operations Confound Example: “Big Babies”

1. Hypothesis: Fear increases social affiliation, but anxiety causes people to seek isolation 2. Manipulation to induce “Oedipal” anxiety: Some participants (male college students) were told they would have to suck babies’ bottles and dummies in front of others after a short wait. Fear was manipulated with electric shocks 3. Results: While waiting, men were less affilliative (i.e. chose to wait alone vs. wait with others) if they were anticipating sucking bottles/dummies 4. Alternative explanation: a. Perhaps the manipulation produced embarrassment (my answer – correct)

Prevention of Operations Confounds 1. Some operations confounds can be prevented by holding procedures constant across levels, for example: a. Manipulation procedures (e.g. puppy study – you could control the amount of handling the puppies could be getting. So, the puppies in the long dose condition were handled the same amount as the puppies in the short dose condition) b. Experimenters (e.g. praise study example – this was keeping the confederate the same person) c. Settings (e.g. workplace incentives example) d. Instructions 2. This is, hold constant anything not relevant/essential to the difference between conditions

Materials Confounds (Competing Construct)

1. A materials confound is an alternative explanation (specifically, a competing construct) that is introduced in the materials used by the researcher to operationalize a construct 2. That is, a specific type of operations confound 3. A special problem that can occur when items in materials or stimulus are assigned to IV levels based on attributes that they possess a. That is, items may differ on characteristics other than those intended 4. Side Notes to Exemplify: a. For example, if testing people on what words appear more frequently we have to consider the fact that long words as seen less frequent when designing our experiment. b. It’s kind of like a PAV, except here we are dealing with characteristics that differ on materials and/or stimuli

Materials Confound Example: Walker Gait and Gender 1. Participants make personality judgments based on the gait of videotaped male vs. female models 2. Would any apparent gender-of-walker effect necessarily be due to differences in the way that men vs. women walk? 3. Competing constructs: Examples of other characteristics of the stimuli that may vary between male and female walkers (things that have nothing to do with the variable of interest) a. Idiosyncrasies of the particular walkers filmed i. For example, if, by coincidence, a few walkers of one gender had unusual/eccentric (off-putting) walk 1. So, this has nothing to do with their gender

b. Mean height

Materials Confound Example: Word Learning 1. Participants learn a list of one-syllable words, and are tested on their recognition on their memory 2. IV: Easy-to-pronounce (e.g. loop, cat) vs. difficult-to-pronounce (e.g. brooch, yacht, aisle) 3. Competing constructs: Example of other characteristics of the stimuli that many vary between easy-to-pronounce and difficult-to-pronounce words a. Word frequency – high vs. low (how common) b. Word length c. Orthographic regularity (i.e. consistency with the way the language usually associates letters to sound) i. You don’t see too many words in the English language spelt like aisle or yacht 4. Look at slides for another example

Prevention of Materials Confounds – Three Main Strategies 1. Replicate over stimuli a. For example, have many male and female walkers b. This reduces the impact of idiosyncrasies 2. Random assignment of items to levels of the IV a. For example, words, pictures, problems, faces, etc. b. But it is not possible for all stimuli to appear at random IV levels if the independent construct is an attribute of the item i. For example, gender-of-walker, word pronounce-ability

3. Hold extraneous item variables constant a. For example, walker height, word frequency and length, etc. b. There are several ways this could be done: i. Ensure that items do not differ on extraneous item variables (e.g. all walker the same height) ii. Ensure on average, conditions do not differ on the extraneous item variables 1. For example, match words on frequency, letter cluster typicality measures, length, etc. c. Cycle one set of items through all conditions i. For example, for a within-participants IV with two levels, divide items into two blocks, and counterbalance assignment of item blocks to IV levels of participants ii. But it is not possible for all stimuli to appear at all IV levels if the IV is an attribute of the items

Data Linkage (Competing Construct) 1. Data linkage issues occur when spurious results are produced to: a. Inappropriate analysis of data; or b. Inappropriate transformation of scores (i.e. preparing data for analysis) 2. For example, Simpson’s Paradox (see Lecture 1) a. Can occur when two tests differ in both i. Difficulty; and ii. The proportion of participants undertaking each test

b. For example, if the less able group of participants is over-represented in the easier test, it can look as though they are performing better overall if the scores are inappropriately proportionalised

Simpson’s Paradox – Shape Rotation Task Example 1. Participants given one of two shape-rotation tests 2. The researcher falsely assumed the tests were equivalent in difficulty 3. Results (i.e. number who passed/number who attempted)

Data Linkage Examples: Inappropriate Analysis of Data 1. IV-DV association inferred from chi-square (for categorical DV), when observations are not independent a. For example, participants contribute to multiple categories 2. Treating categorical data as continuous for regression analyses a. Where there is no interval or ratio scale assignment to categories is arbitrary b. For example, marital status (1 = never, 2 = divorced and unmarried, 3 = divorced and remarried, 4 = widowed)

i. If these number are entered in a regression analysis as a continuous IV, any relationship between the material status IV and the DV is spurious

Observer Expectancy (Reactivity) 1. Reactivity, in general, is when participants start to cotton on and realise what the hypothesis of the experiment is 2. Observer expectancy issues occur when an observer is biased to perceived participants’ behaviour as consistent with the hypothesis a. You’ll see people’s behaviour in line with your hypothesis 3. This is a bias of how a behaviour is scored 4. Wiki: Confirmation bias can lead to the experimenter interpreting results incorrectly because of the tendency to look for information that conforms to their hypothesis, and overlook information that argues against it. 5. Confounds due to observer expectancy are possible whenever observers: a. Take observational measures i. For example, when we are recording the number of tantrums, we may be more cautious when recording if it is in line with our hypothesis and overlook it if it is not in line with our hypothesis b. Score free narratives i. E.g. responses to open-ended questions c. Interpret ambiguous responses

Prevention of Observer Expectancy 1. Where possible, use observers who are blind to the hypothesis and conditions (i.e. which participant is in which condition)

a. Double-blind study I think it’s called 2. Videotape participants’ behaviour for independent observers to code 3. Where possible, use objective/unambiguous measures of behaviour a. For example, a physiological index

Experimenter Expectancy (Reactivity) 1. Experimenter expectancy issues occur when the experimenter communicates to the participant what to do to support the hypotheses. 2. You don’t really do this on purpose. Rather, it can be an unconscious thing whereby your desire to obtain the alternative hypothesis intertwines with your body language, etc. For example, we may express happiness to the treatment condition and indifference to the control condition. 3. Experimenters may do this: a. Through unintentional nuances in: i. Tone of voice. ii. Expression. iii. Wording of instructions, etc. b. Through the “demand characteristics” of the settings, stimuli, tasks, questions, etc., used in the study.

Demand Characteristics and Self-Report 1. In self-report studies, what the participant is asked about will influence the participant’s ideas about the aims of the study, and what the “appropriate” responses are. 2. Neuroticism & Attachment Style.

a. If participants are asked about both of these, it’s easy for them to work out what the hypothesis is. 3. Marital Satisfaction Study (from M&J): a. Participants asked to say how much they love their partner. b. Next, they received fake feedback saying that their partner loves them intensely. c. When participants are asked the same question again, the demand to give a more positive response is very high!

Prevention of Experimenter Expectancy 1. Preventing General Experimenter Expectancy a. Where possible, use experimenters who are blind to the hypotheses and conditions. 2. Preventing Demand Characteristics a. Include procedures that disguise the purpose of the study, such as: i. Cover stories. 1. E.g., Petty et al. (1981): participants told they were listening to judge whether the tapes were of broadcastable quality. ii. Irrelevant distractor tasks. iii. Irrelevant filler questions in self-report studies. iv. Presenting parts of experiments as independent studies by different experimenters. 3. Note: Ethical problems with deception (e.g., Milgram).

Participant Expectancy (Reactivity) 1. Wiki: When a research subject or patient expects a given result and therefore unconsciously affects the outcome, or reports the expected result 2. Participant expectancy may take several forms, these include: 3. Demand Compliance a. The participant responds in accordance with demand characteristics. b. The hypotheses will be supported, and the study confounded. 4. Participant Resistance a. The participant tries to resist what they think the experimenter wants them to do. b. I.e., the opposite of demand compliance. c. Usually a counter-confound 5. The Hawthorne Effect a. I.e., “When the treatment group changes their behaviour not because of the treatment itself, but because they know they are getting special treatment” (M&J). i. E.g., increase factory lighting à increased productivity; decrease factory lighting à another increase. b. I.e., due to demand characteristics inherent in an intervention. 6. Positive Self-Presentation (a.k.a. Social Desirability) a. Participants respond in a way that portrays them in a good light. i. E.g., responsible, honorable, intellectual, artistic, etc.

Prevention of Participant Expectancy 1. Prevent experimenter expectancy and demand characteristics a. As above

2. Use a placebo-control a. That is, a group receiving a treatment known to have no effect b. If the experimental group improves more than the placebo, the treatment effect is not due entirely to participant expectancy 3. Ensure participants are blind to hypothesis and conditions a. Double-blind control = both participant and experimenter are blind to the hypothesis and conditions

Floor and Ceiling Compression 1. Floor Compression: a. Occurs when a test is too difficult or inadequate time is given – scores are compressed near the minimum. 2. Ceiling Compression: a. Occurs when a test is too easy – scores are compressed near the maximum. 3. Floor or Ceiling Compression: a. Usually works against a significant effect (i.e., a sensitivity issue). b. Is a confound if null results were expected i. This is because the null results isn’t attributed to the predicted hypothesis, it is due to the test being easy c. To overcome, this you can put in another IV that you know has an effect, that is if you are predicting a null result

Ceiling Compression Example: Bilingual & Monolingual Children 1. Bilingual and monolingual children are tested for intelligence and syntactic analysis. 2. Results. As predicted: a. Bilingual children are better at syntactic analysis than monolingual children,

b. But they are not more intelligent.

3. Potential ceiling compression confound: a. If the intelligence test is too easy, a true difference between the groups may be masked by a ceiling effect

Floor or Ceiling Compression: Tell-Tale Signs to Look For 1. One or more means close to the maximum or minimum achievable (i.e., the effective ceiling or floor). 2. Skew in the frequency distribution of scores (if available). a. Ceiling compression = negative skew. i. I.e., scores bunched at top of distribution. b. Floor compression = positive skew. i. I.e., scores bunched at bottom of distribution. 3. Reduced variance (i.e., small standard deviation(s)). a. I.e., scores squashed around condition mean(s). b. Note: If only a subset of conditions is affected by ceiling or floor compression, their SDs will be much smaller than in other conditions.

Tricky Floor or Ceiling Effects 1. Sometimes the ceiling/floor is not at the end of the measurement scale, but depends on what’s achievable under the conditions 2. High floor example: a. Multi-choice test with 4 options b. Effect floor is at 25% chance (not zero) 3. Low ceiling example a. Participants given 60 arithmetic problems to solve in 1 minute b. The best mathematicians need 2 seconds/problem c. Effective limit on top score (i.e. ceiling) = 30/60 4. This is basically describing effective ceiling or flooring a. That is, we don’t have to always look at the top scores in order to achieve a floor or ceiling effect b. For example, in the case of the maths example, 30 would be the ceiling

Floor or Ceiling Compression: Interpretational Issues 1. With ceiling or floor compression, it’s not usually possible to fix the data 2. Major interpretational problem: a. The hypothesis can’t be properly assesse b. There’s no way of knowing what would happen if the compression effects were removed

Preventing Floor or Ceiling Compression 1. Choose/design a more sensitive test. For example: a. More difficult to avoid ceiling effects b. Easier to avoid floor effects

2. Pre-test measures for appropriateness a. That is, to avoid floor or ceiling compression b. Note: Pilot participants and procedures should be comparable to those in the planned study! c. This way we can detect whether we do have floor or ceiling compression by analysing the data and checking for things like skew, variance, etc.

Measurement Artefacts Differential Floor or Ceiling Compression 1. An ordinal interaction may be the result of differential floor or ceiling compression a. That is, produced by compression that does not affect all conditions/cells in the design (hence, “differential”) 2. Note: Differential floor or ceiling compression is an important and common problem a. If an interaction that may have been caused by differential floor or ceiling compression was predicted, the results are confounded b. Note: More likely with large main effects of the IVs 3. Why does it happen? a. Scaling problems with ordinal interactions

Scaling Problems With Ordinal Interactions 1. Whether an ordinal interaction is significant or not can depend on the measurement scale a. For example, how sensitive the DV scale is to change in the IV(s) b. This includes: Where the ceiling and floor lie i. That is, is change in the IV(s) likely to produce scores close to the ceiling or floor in some condition

2. If the DV is too close to ceiling or floor in some conditions (and not others), then an apparent interaction may be mere measurement artefact (i.e. spurious) a. Because the interaction is confounded, it is not possible to know the true result i. I.e. main effect only, or many effect + interaction?

Diagnosing Differential Floor or Ceiling Compression Confounds 1. To determine whether there is a confound: a. Check for possible ceiling or floor compression in the results i. See “Tell-Tale Signs to Look For” (above) b. Ask yourself: What might happen to the results if the floor/ceiling compression could be removed (i.e. if the scale could be extended)? i. If the interaction might disappear – confound ii. If the interactive pattern would get stronger – no confound; no problem!

Problematic Ordinal Interaction: Example Introverts and Extraverts 1. Hypothesis: a. The effect of physiological arousal on memory depends on extraversion b. We also predict that the degree to which there is high arousal, the effects for introverts will be much stronger, however, extraverts will still score higher 2. Here we have a main effect on extraversion 3. We also have a main effect on arousal 4. However, the benefits is greater for the introverts than the extraverts 5. This may be an artefact interaction because extraverts are performing at ceiling a. That is, they are getting 90-100%

6. If there was an extended scale the results may show that the extraverts show the same improvement as the introverts

7. Question: What might happen to the results if the ceiling compression could be removed (i.e. if scale could be extended) a. Answer: we might not have an interaction but still have the main effects

R e c a lle d M ean %

130 120 110 100 90 80 70 60 50

tr u e r e s u lt? (m a in e f fe c t)

E x tr a v e r ts

In tr o v e r ts

Lo w A ro u sa l

H ig h A r o u s a l

M ean %

R e c a lle d

Non-Problematic Ordinal Interaction: Example of Extravert and Introverts

100

E x tr a v e r t s

80

p o s s ib le c e ili n g c o m p r e s s io n

60 40

In t r o v e r t s

20 0 Lo w A rou sal

H ig h A r o u s a l

1. Question: What might happen to the results if the ceiling compression could be removed (i.e. scale extension) 2. Answer: We could get a stronger interaction

R e c a lle d M ean %

tr u e r e s u lt? (s tr o n g e r in te r a c tio n )

120 100

E x tr a v e r ts

80 60 40 20

In tr o v e r ts

0 Low Arousal

H ig h A r o u s a l

lle d

Disordinal Interaction: No Risk

100 80 p o s s ib le c e il in g c o m p r e s s io n

M ean %

p o s s ib le c e i l in g c o m p 4r e0 s s i o n

20

In tr o v e r ts

E x tr a v e r ts

Low A rou sal

H ig h A r o u s a l

0

1. Question: What might happen to the results if the ceiling compression could be removed (i.e. scale extension) 2. Answer: there would be problem with the interpretation of the results as we would get the same pattern of results

120 100

M ean %

R e c a lle d

t r u e r e s u lt ? (s a m e p a tte r n / s to r y )

80 60 40 20

I n tr o v e r ts