Exam 3 Notes PDF

Preview

Summary

Full notes from book and lecture, also quiz anwers...

Description

Chapter 7 1. Before-and-after designs: A quasi-experimental design consisting of before-and-after comparisons involving the same variables but different groups. 2. Causal effect: The finding that change in one variable leads to change in another variable, ceteris paribus (other things being equal). 3. Cohort: Individuals or groups with a common starting point. Examples of cohorts include the college class of 1997, people who graduated from high school in the 1980s, General Motors employees who started work between 1990 and 2000, and people who were born in the late 1940s or the 1950s (the baby boom generation). 4. Compensatory rivalry (John Henry effect): A type of contamination in experimental and quasi-experimental designs that occurs when control group members are aware that they are being denied some advantage and increase their efforts by way of compensation. This problem has also been referred to as the John Henry effect 5. Contamination: A source of causal invalidity that occurs when the experimental and/or the comparison group is aware of the other group and is influenced in the posttest as a result. 6. Control or comparison group: The group of subjects who are either exposed to a different treatment than the experimental group or who receive no treatment at all. 7. Demoralization: A type of contamination in experimental designs that occurs when control group members are aware they were denied some treatment they believe is valuable, and as a result, they feel demoralized and perform worse than expected 8. Differential attrition: A problem that occurs in experiments when comparison groups become different because subjects are more likely to drop out of one of the groups than the other, for various reasons. 9. Double-barreled question: A single survey question that actually asks two questions but allows only one answer 10. Endogenous change: A source of causal invalidity that occurs when natural developments or changes in the subjects (independent of the experimental treatment itself) account for some or all of the observed change from the pretest to the posttest. 11. Evaluation research: Research about social programs or interventions. 12. Ex post facto control group design: Nonexperimental design in which comparison groups are selected after the treatment, program, or other variation in the independent variable has occurred. 13. Expectancies of the experimental staff (self-fulfilling prophecy): A source of treatment misidentification in experiments and quasi-experiments that occurs when change among experimental subjects is due to the positive expectancies of the staff who are delivering the treatment, rather than to the treatment itself; also called a self-fulfilling prophecy. 14. Experimental group: In an experiment, the group of subjects that receives the treatment or experimental manipulation 15. External events (history effect): A source of causal invalidity that occurs when something other than the treatment influences outcome scores; also called an effect of external

events. 16. Factorial survey: A survey in which randomly selected subsets of respondents are asked different questions, or are asked to respond to different vignettes, in order to determine the causal effect of the variables represented by these differences. 17. Field experiment: An experimental study conducted in a real-world setting. 18. Fixed-sample panel design (panel study): A type of longitudinal study in which data are collected from the same individuals—the panel—at two or more points in time. In another type of panel design, panel members who leave are replaced with new members. 19. Hawthorne effect: A type of contamination in experimental and quasi-experimental designs that occurs when members of the treatment group change in terms of the dependent variable because their participation in the study makes them feel special. 20. History effect (external events): A source of causal invalidity that occurs when something other than the treatment influences outcome scores; also called an effect of external events. 21. Matching: A procedure for equating the characteristics of individuals in different comparison groups in an experiment. Matching can be done on either an individual or an aggregate basis. For individual matching, individuals who are similar in terms of key characteristics are paired prior to assignment, and then one member of each pair is assigned to the two groups. For aggregate matching, groups are chosen for comparisons that are similar in terms of the distribution of key characteristics. 22. Mechanism: A discernible process that creates a causal connection between two variables. 23. Multiple-group before-and-after design: A quasi-experimental design consisting of several before-and-after comparisons involving the same variables but different groups 24. Nonequivalent control group design: A quasi-experimental design in which there are experimental and comparison groups that are designated before the treatment occurs but are not created by random assignment. 25. Placebo effect: A source of treatment misidentification that can occur when subjects receive a treatment that they consider likely to be beneficial, and improve because of that expectation rather than because of the treatment itself. 26. Posttest: Measurement of an outcome (dependent) variable after an experimental intervention or after a presumed independent variable has changed for some other reason. 27. Pretest: Measurement of an outcome (dependent) variable prior to an experimental intervention or change in a presumed independent variable for some other reason. The pretest is exactly the same “test” as the posttest, but it is administered at a different time. 28. Process analysis: A research design in which periodic measures are taken to determine whether a treatment is being delivered as planned, usually in a field experiment. 29. Quasi-experimental design: A research design in which there is a comparison group that is comparable to the experimental group in critical ways, but subjects are not randomly assigned to the comparison and experimental groups. 30. Random assignment: A procedure by which each experimental and control group subject is placed in a group randomly

31. Regression effect: A source of causal invalidity that occurs when subjects who are chosen for a study because of their extreme scores on the dependent variable become less extreme on the posttest due to natural cyclical or episodic change in the variable 32. Repeated measures panel design (time series design): A quasi-experimental design consisting of several pretest and posttest observations of the same group. 33. Sample generalizability: Exists when a conclusion based on a sample, or subset, of a larger population holds true for that population 34. Selection bias: A source of internal (causal) invalidity that occurs when characteristics of experimental and comparison group subjects differ in any way that influences the outcome. 35. Selective distribution of benefits: An ethical issue about how much researchers can influence the benefits subjects receive as part of the treatment being studied in a field experiment 36. Solomon four-group design: An experimental design in which there are four groups. Two of the groups represent a classic experimental design in which there is an experimental and a control group that each receive a pretest. The final two groups represent an experimental and a control group, but neither receives a pretest. This design helps to identify the interaction of testing and treatment. 37. Time series design (repeated measures panel design): A quasi-experimental design consisting of many pretest and posttest observations of the same group. 38. Treatment: The manipulation that exposes subjects in an experiment to a particular value of the independent variable. 39. Treatment misidentification: A problem that occurs in an experiment when the treatment itself is not what causes the outcome, but rather the outcome is caused by some intervening process that the researcher has not identified and is not aware of. 40. True experiment: Experiment in which subjects are assigned randomly to an experimental group that receives a treatment or other manipulation of the independent variable and a comparison group that does not receive the treatment or receives some other manipulation. Outcomes are measured in a posttest Quiz 1. Ex post facto control group designs are different from nonequivalent control group designs in that the groups are designated after the treatment or intervention has occurred. a. True 2. Differential attrition can lead to a selection bias. (true) 3. As a general rule, random distribution of benefits is justified when the researchers do not know whether a treatment actually is beneficial or not. (true) 4. True experimental research is rare in the social sciences. (true) 5. True experiments and quasi-experiments are the same, except that there are two experimental groups in a quasi-experimental design. (False) 6. Suppose a researcher wants to study the impact of a new policy, such as the impact of closed circuit television surveillance on college campuses to reduce theft and vandalism.

In order to do this, he obtains campus police records of calls regarding theft for each month six months prior to the installation of CCTV and six months after the installation of CCTV. What type of research design is he using? a. Time series design 7. Sherman and Berk’s original study looked at the effect of arrest on recidivism for domestic violence offenses. In this study, when officers had probable cause to believe that an assault occurred, they were randomly assigned to make an arrest, give mediation, or separate the abuser and victim. Later, the researchers implemented a posttest through an interview and consultation of official records to see if the abusers had reoffended. What research design did they use? a. True Experiment 8. A group of research subjects is receiving a treatment that they consider likely to be beneficial. As a result of this expectation, and not because of the treatment itself, the research subjects improve. This is known as the: a. Placebo Effect 9. Having selected his sample from the population, a researcher moves to classify subjects into experimental and control groups. However, the groups strike him as “unbalanced” so he shifts a few subjects from the experimental group to the control and vice versa. This is an example of: a. Selection Bias 10. Which design uses pretreatment measures in the experimental group as the basis for comparison? a. Before and after design 11. The simplest type of before-and-after design is: (Fixed sample design) 12. All of the following are basic sources of internal invalidity except: (External validity) 13. Researchers conducting a medical study want to be sure that their results are valid and reliable. One method of ensuring this is the use of a double-blind study. They will most likely implement this to prevent which from occurring? (treatment misidentification) 14. A researcher is conducting a longitudinal study. The results of her study on school children are changing because the children are aging and becoming more mature. Her primary concern here is one of: (endogenous changes) 15. When the treatment group is receiving some service or benefit that the control group is not, what might happen? (differential association) Powerpoint I. Requirements for True Experiments A. Two comparison groups (in the simplest case, an experimental and a control group) B. Random assignment to the two (or more) comparison groups C. Assessment of change in the dependent variable in both groups after the experimental group receives the experimental condition II. 1. Two Comparison Groups

III.

IV.

V.

VI.

VII.

A. True experiments must have at least one experimental group (subjects who receive some treatment) and at least one comparison group (subjects to whom the experimental group can be compared). 2. Random Assignment A. Makes comparison group in a true experiment powerful tool for identifying the effects of the treatment B. How? By providing a good estimate of the counterfactual—the outcome that would have occurred if subjects exposed to the treatment actually had not been exposed 3. Pretest and Posttest A. Posttest – measurement of outcome (dependent variable) in both groups after experimental group has received treatment B. Pretest – measurement of dependent variable prior to experimental intervention Identical to posttest, but administered at different time Example: Prison Classification and Inmate Behavior A. Bench & Allen (2003) hypothesized that prisoners classified to medium security will have fewer disciplinary infractions than those classified to maximum security B. Labeling theory classification to maximum security gives inmates image of being hard to handle, prone to fight, high risk for escape C. Note that experiment had no pretest Quasi-Experimental Designs A. Quasi – Latin word meaning “resembling” or “having some, but not all of the features of” B. Quasi-experiments do not use random assignment C. Usually because random assignment is not feasible given topic or population being studied D. Have less explanatory power and more problems with validity (we’ll talk about experimental validity soon) E. Three major types :Nonequivalent control group designs, Before-and-After designs, Ex post facto control group designs Nonequivalent Groups Designs A. Most common type used, especially in program evaluation. B. Because control (comparison) and experimental (treatment) groups are not selected through random assignment, they are considered nonequivalent 1. Threats to validity are increased. Random assignment reduces or eliminates many threats to validity. C. Comparison group serves same function as control group D. Comparison group is selected to be as similar as possible to group that receives experiment 1. Comparison group must be selected in similar time frame as treatment group.

VIII.

IX.

X.

XI.

XII.

XIII.

Matching in Quasi-Experiments A. Matching is another procedure sometimes used to equate experimental and comparison groups, but by itself it is a poor substitute for randomization. B. Matching of individuals in a treatment group with those in a comparison group might involve pairing persons on the basis of similarity of gender, age, year in school, or some other characteristic. C. The basic problem is that, as a practical matter, individuals can be matched on only a few characteristics; unmatched differences between the experimental and comparison groups may still influence outcomes. Matching in Quasi-Experiments A. Individual matching 1. Determine a few (3-4 max.) key characteristics that you know are likely to affect the result a) Select an individual for experimental/treatment group b) Select an individual with ‘identical’ set of key characteristics for control/comparison group B. Aggregate matching 1. General characteristics of control group match general characteristics of experimental group on characteristics that you know are likely to affect the result C. Control group must be “similarly situated” to the experimental group Before-and-After Designs A. No comparison group B. Often used when all people in population receive intervention 1. New law is passed 2. New policy is implemented C. Weakest type of design because cannot draw conclusions about effectiveness of ‘treatment’ from results Comparing Quasi-Experimental Designs A. Non-equivalent Control Groups: Experimental/Treatment Group Control/Comparison Group B. Before-and-After: Experimental/Treatment Group, No comparison group Time-Series Designs A. Most ‘powerful’ type of Before-and-After design B. Often called “Interrupted Time-Series” design 1. Make before and after measures of some phenomenon 2. Then “interrupt” time by implementing an intervention C. Usually does not have separate comparison group (but may) Ex Post Facto Control Group Designs A. Ex post facto - another Latin term, meaning ‘from what is done afterward’, after the fact, retrospectively, looking back

XIV.

XV.

XVI.

XVII.

B. Identify treatment and comparison groups after the fact 1. after results have already occurred C. Not really an experimental design … 1. …but often can identify comparison groups that are reasonable D. Often, if an experimental or good quasi-experimental design had been possible, any researcher would have done it instead of this design. Internal Validity A. Conclusions reflect what actually occurred in the experiment, rather than something that got in the way or some error. B. Ability to yield valid conclusions is determined by comparability of experimental and control groups 1. This is why random assignment is best: the only difference is the intervention/experiment C. Internal validity is crucial to being able to draw conclusions from experiment. Causal (Internal) Validity A. Typically understand validity by examining sources of invalidity (also called ‘threats to validity’) B. Five basic sources of invalidity 1. Selection bias 2. Endogenous change 3. External events (history effects) 4. Contamination 5. Treatment misidentification 1. Selection Bias A. Treatment and comparison groups are different at beginning or end of study 1. Random assignment solves this problem at beginning of study IF it is done properly (DV experiment) 2. At end groups can differ based on differential attrition B. if some types of people are more likely to drop out of the study C. also called experimental mortality D. Ex.: long-term residential substance abuse treatment programs. Women with kids are more likely to drop out because getting the kids taken care of becomes harder as time goes on 2. Endogenous Change A. Endogenous (internal, within the person) B. Maturation 1. People are continually changing; if an experiment takes place over a fairly long time, this can be a problem. Especially an issue with research on children. C. Testing 1. The process of testing and retesting will influence people’s behavior. They

XVIII.

XIX.

XX.

XXI.

may become more sensitized to the issue that is the object of the experiment. (This is one case in which researchers may be justified in some deception) D. Regression 1. Changes in person may affect scores (having a bad day influences test performance) 2. Special problem when people are chosen for study because of a very high or low ‘score’ a) Sometimes things just get better on their own, regardless of any intervention 3. External Events A. External events B. Also called 1. Exogenous Events 2. History Effect :An event may occur during the course of the experiment. 3. Things happen in environment that changes results of experiment a) Awareness of domestic violence during and after the OJ Simpson case. Calls to police went up. 4. Contamination A. Experimental and Control groups communicate and Control group gets some of the intervention, or groups otherwise affect each other 1. Also called diffusion B. Can be intentional or unintentional and may result in 1. Compensatory rivalry = control group member works harder to make up for not having the intervention 2. Demoralization = control group member feels left out and becomes worse than they would have been without the study 5. Treatment Misidentification A. Treatment does not cause outcome, but rather the outcome is caused by something that went on in the conduct of the experiment B. Expectancies of experimental staff 1. Researcher wants study to be success 2. Problem in social programs where success measures can be subjective 3. Double-blind - common in medical research, but difficult in social research C. Two major types 1. Placebo effect - People improve because they think they are getting something that will make them improve 2. Hawthorne effect - People may change because more attention is being paid to them Enhancing External Validity

A. Interaction of Testing and Treatment 1. Experiment is effective only when conditions created by the experiment occur 2. Often occurs when studying attitudes and subjects become sensitized to issues by their i...