Cozby & Bates, Chapter 5, 7, 8 PDF

Preview

Summary

Professor Putney...

Description

1

HDFS 425: Research Methods in Human Development and Family Studies Professor Jennifer Putney September 28, 2020 *Upcoming Assignments: Journal Club posts due 9/28 and 10/1; SmartBook Chapter 5 due 10/2 Cozby & Bates (CB), Chapter 5: Measurement Concepts Methods in Behavioral Research Learning Objectives ● Define reliability of a measure of behavior and describe the difference between testretest, internal consistency, and interrater reliability. ● Define construct validity and discuss ways to establish construct validity. ● Distinguish between face validity, content validity, predictive validity, concurrent validity, convergent validity, and discriminant validity. ● Describe the problem of reactivity of a measure of behavior and discuss ways to minimize reactivity. ● Describe the properties of the four scales of measurement: nominal, ordinal, interval, and ratio. Reliability of Measures ● Measure: tool uses to measure or provide value for a behaivor, a phenomenon, an interaction; it refers to a tool or a survey oftentimes ● Reliability: the consistency or stability of a measure. Your everyday definition of reliability is close to the sicneitci one. You may say professor burkholder is reliable because she starts class at 10 each day without fault. Can also think about hte concept of true score & emasurmenet erro ● True score: the real, “true” value on a given variable. ● Measurement error: the difference between a true score and the measured score. ● Pearson product-moment correlation coefficient: a common method of calculating a correlation coefficient to tell how strongly two variables are related to each other. ○ The Pearson correlation coefficient (symbolized as r) can range from 0.00 to +1.00 and 0.00 to −1.00. ○ The closer to +1.00 or −1.00, the stronger the correlation. ○ Correlation of value 0 tells uis the two values are not related at all ○ 1 or -1 indicates that there is a stronger relationship between the two values ○ + and - value sign indicates the direction of that relationship

2

■ (+) = positive linear relationship; high scores on one variable associated with high schores on the other variable ■ (-) = negative linear relationships; high scores on one variable associated with low sixers on the second variable FIGURE 1

● Comparing data of a reliable and unreliable measure. ● Showing variability of a test score as far as what it’d look like if a measure was reliable vs. unreliable ● If an idnivudal’s intelligence was tested repeatedly, a reliable measure would have a very narrow variation in scores. Everytime they took the test, they would get evry close to th esame score ● a n unreliable measure would show a broad range of scores. Thus, their true intelligence could not bne measured with this pairutclar survey or tool Reliability of Measures: Test-Retest Reliability ● Test-retest reliability is assessed by measuring the same individuals at two points in time and comparing results. ○ A high correlation between test and retest indicates reliability. ○ EX. Reliability of a test of intelleignece would be assessed by giving the test to a gorup of people one day and then again a week later ■ One would then have two scores for each person, and a correlation coefficient could be calculated to determine the relationship between the first test score and their re-test score ○ The correlation might be artificially high because individuals remember how they responded the first time ● Alternate forms reliability uses two forms of the same test for testing instead of repeating the same test. ○ This avoids problems with participants remembering and repeating earlier responses.

3

Reliability of Measures: Internal Consistency Reliability ● Internal consistency reliability is the assessment of reliability using responses at only one point in time. ○ Because all items measure the same variable, they should yield similar or consistent results. ● One indicator of internal consistency is split-half reliability: the correlation of the total score on one half of the test with the total score on the other half. They are breaking the measure in half in order to assess its relaibility as one half of the test equal to the other half of the test. ○ The two halves are created randomly, dividing the items into two parts ○ A high correlation indicates that the questions on the test are measuring the same thing. ○ Cronbach’s alpha: provides us with the average of all possible split-half reliability coefficients; the most commonly used indicator of reliability based on internal consistency; uses an alpha sign; something you can learn within Stat’s classes, but researchers use online analysis tools to calculate this quickly ● Item-total correlations: correlations of each item score with the total score based on all items. ○ Low-correlation items are actually measuring a different variable and can be eliminated. ○ Very informative as it provides information about eahc individual item within the measure Reliability of Measures: Interrater Reliability ● Interrater reliability is the correlation between the observations of different raters. ○ The extent to which raters agree with their observations ○ A high correlation indicates raters agree in their ratings. ○ A commonly used indicator is Cohen’s kappa. ● Note that reliability indexes do not indicate if a particular measure is an accurate measure of the variable of interest. ○ A measure can be highly reliable but not accurate. FIGURE 2 Three strategies for assessing reliability ● Reliability is important when researchers develop measures of behaviror, but it is not the only characteristic of a measure or the only thing researchers worry about ● Realiabuiltiy tells us about the measurement error but does not tell us if we have a good measure of the variable of interest

4

● Test-Retest Reliability ○ A measure is taken two times. The correlation of the two scores represents testretest reliability. The correlation between two versions of a measure is called alternative forms reliability. ● Internal Consistency Reliability ○ Cronbach’s Alpha: Correlation of each item on the measure with every other item on the measure is the Cronbach’s Alpha reliability coefficient. ○ Split-Half Reliability: The correlation of total score on one half of a measure with the total score on the other half of the measure presents split-half reliability. ● Interrater Reliability ○ Evidence for reliability is present when multiple raters agree in their observations. Cohen’s kappa is a commonly used indicator. Construct Validity of Measures ● Construct validity → the adequcny of the operational definition of variables; or, to what extent does the definition actually reflect the true theortical meaning of the variable ○ EX. If I were studying intelligence and I elected to give someone an IQ test, in order to have a high degree of construct validity, we would have to be able to say the questions on the test and score derived was highly correlated with the actual level of someone’s intelligence; does it accurately measure intelligence? ● A measure has construct validity if it actually measures what it is intended to measure. ● Face validity: when a measure appears to accurately assess the intended variable ● Content validity: based on comparing the content of the measure with the universe content that defines the construct. ● Predictive validity: research that uses a measure to predict some sort of future behavior and the accuracy of those predictions ● Concurrent validity: the extent to which scores on the measure are related to a criterion measured at the same time (concurrently) ● Convergent validity: scores on the measure are related to other measures of the same construct. ○ If we are evaluating a measure intended to measure the level of depression someone is experiencing, I can check the convergent theory by comapring that meausre to other measures of depression ● Discriminant validity: scores on the measure are not related to other measures that are theoretically different.

5

TABLE 1 Indicators of construct validity—Examples 1

Reactivity of Measures ● Reactivity is a potential problem when measuring behavior. ● A measure is said to be reactive, if awareness of being measured changes an individual’s behavior. ● Measures of behavior vary in terms of their potential reactivity. Variables and Measurement Scales ● There are four kinds of measurement scales: ○ Nominal scales have no numerical or quantitative properties; categories or groups simply differ from one another.

6

○ Ordinal scales allow us to order the levels of the variables under study. ○ Interval scales are numeric scales in which the intervals between numbers on the scale are equal in size. ○ Ratio scales have an absolute zero point that indicates the absence of the variable being measured. ● The conclusions drawn about the meaning of a particular score on a variable depend on which type of scale has been used. ● Interval and rating scales provide quantitative information about variables; nominal scales do not. ● The scales used in a study determine the types of statistics that are appropriate for analyzing its results. TABLE 2 Scales of Measurement

Cozby & Bates (CB), Chapter 7: Asking People About Themselves: Survey Research Methods in Behavioral Research Learning Objectives ● Discuss reasons for conducting survey research. ● Identify factors to consider when writing questions for interviews and questionnaires. ● Describe different ways to construct questionnaire responses, including closed-ended questions, open-ended questions, and rating scales. ● Compare the two ways to administer surveys: written questionnaires and oral interviews. ● Distinguish between probability and nonprobability sampling techniques, including

7

simple random sampling, stratified random sampling, and cluster sampling; convenience (or haphazard) sampling, purposive sampling, and quota sampling. ● Describe how samples are evaluated for potential bias, including sampling frame and response rate. Why Conduct Surveys? ● Survey research employs questionnaires and interviews to ask people to provide infnormaiton about thesleves → their attitudes and bleiefs/ demographics/ and other facts, and past or intended future behaviors ● Survey and interview (sibling) ● Enable researchers to collect data directly from participants by asking them questions. ○ Survey data provide useful information for making public policy decisions. ○ Survey research is also important as a complement to experimental research findings. ● Many important variables including attitudes / self- reports of behavior are most easily studied using questionnaires or interviews Why Conduct Surveys? Part 2. ● Accuracy and truthfulness are potential issues with surveys. ○ A response set is a tendency to respond to survey questions from a particular perspective rather than answering questions directly. ■ Can effect useful of data obtained from self-report ■ The most common response set is called “social desirability” or faking good ● The social desirability response set is the tendency to answer questions in the way that would reflect most favorably on the respondent. ○ Should not be assumed people consistently mis-represent themselves Constructing Questions to Ask ● The first thing the researcher must do is explicitly determine the research objectives. ○ Survey questions must be tied to the research questions that are being addressed. ● There are three general types of survey questions: ○ Facts and demographics ■ Factual questions ask people to indicate things they know about themselves and their situation

8

○ Behaviors; and ■ Can focus on past or intended future behaviors ○ Attitudes and beliefs. ■ Focus on the ways people evaluate and think about different issues Constructing Questions to Ask: Question Wording ● Cognitive psychologists have identified a number of potential problems that often make it difficult to understand the question ○ Unfamiliar technical terms; ○ Vague or imprecise terms; ○ Ungrammatical sentence structure; ○ Phrasing that overloads working memory; and ○ Embedding the question with misleading information. ● The questions should be relatively simple. ○ Avoid jargon and technical terms when possible. ○ For more complex questions, provide a brief description of the background. ● Avoid double-barreled questions. ○ Asks two things at once ○ Examples: “Should senior citizens be given more money for recreation centers and food assistance programs?” Or, “do you love snakes and ice cream?” ● Avoid loaded questions. ○ Written to lead people to respond in a particular way ○ Includes emotionally- charged words, which can influecne the way people respond and then leads to biased responses ○ Examples: “Do you favor eliminating the wasteful excesses in the budget?” → indicating to the participant which way they want them to respond ● Avoid phrasing questions with negatives. ○ Example: “Do you feel that the city should not approve the proposed women’s shelter?” someone may answer in a way that does not align with their beliefs due to the confusing question wording ● When you ask several questions, a respondent may employ a response set to agree or disagree with all of them. ○ Yea-saying is the tendency to agree consistently. ○ Nay-saying is the tendency to disagree consistently. ○ One way to detect this is to word questions so that consistent agreement is unlikely. Responses to Questions

9

● Questions on a questionnaire or survey may be either closed- or open-ended ● With closed-ended questions, there are a limited number of response alternatives. ○ It is easier to assign values to responses. ○ There is a fixed number of response alternatives. ○ Closed-ended questions are more likely when the dimensions of the variables are well defined. ● With open-ended questions, respondents are free to answer any way they like. ○ This can yield valuable insights into what people are thinking. ○ More time is required to categorize and code the responses. Responses to Questions: Rating Scales ● Rating scales, which assign scores along some numerical dimension, are very common in many areas of research. ○ A graphic rating scale requires a mark along a continuous line anchored with descriptions at each end ○ The semantic differential scale, however, is a measure of the meaning of concepts—rating them on a series of bipolar adjectives using 7-point scales ● Nonverbal scales are appropriate for populations, such as young children or adults with specific cognitive deficits, they may choose this type of scale, such as a smiley face scale

Responses to Questions: Labeling Response Alternatives ● Sometimes researchers need to provide labels to more clearly define the meaning of each alternative. ● One common type of scale assumes a middle alternative is a “neutral” point halfway between endpoints (i.e., Ordinal Scale) ○ Strongly agree, agree, undecided, disagree, strongly disagree. ● Other examples: ○ Comparative rating, such as the following: lower 50%, upper 50%, upper 25%, upper 10%, upper 5%. ○ Frequency of behavior, such as in: less than twice a week, about twice a week, about four times a week, and so on. Finalizing the Survey Instrument ● Formatting: ○ Provide an attractive and professional appearance.

10

○ Ensure it is neatly designed, typed, and free from spelling errors. ○ When using a particular scale format, use it consistently. ● Sequencing of questions: ○ Best to put the most interesting and important questions upfront to capture the attention of your respondents and motivate them to complete the survey ○ Group-related questions when they address a similar theme or topic ● Refining questions: ○ Consider pilot- testing your survey: before you actually administer it to your larger population, you should give the questions to a small group of individuals and have them think aloud while answering. Administering Surveys: Questionnaires ● Can administer a survey through the use of questionnaires or through interviews ○ Questionnaire: written form provided to participants where respondents write their answers as a response ○ Questionnaires generally cost less than interviews. ○ They can be administered in person to groups (i.e., new employees at an orientation, a community group gathering, or in a classroom setting ) or individuals, through the mail, or on the Internet and with other technologies (apps) ○ Respondents must be able to read and understand the questions, especially if you opt to mail your participants their surveys ■ Lower response rate, mor eo fan opportunity for people tonot understand quesitosn and wing it on their responses and more hclalengign to respect a persons confidential status if your sending it to theori home since ou know their address ■ A problem of motivation may arise: many people find it boring to sit by themselves reading and responding to questions. ○ Online forms: anonymous surveys or confidential surveys; can ask open and close ended type questions; the way most research is collected at this point in time Administering Surveys: Interviews ● Benefits of interviews: ○ Response rates tend to be higher. ○ The interviewer can clarify any problems the person might have in understanding questions. ○ An interviewer can ask follow-up questions. ● Potential problems: ○ Interviewer bias: refers to when the interviewer inadvertently shows approval or

11

disapproval of certain answers. ○ Interviewers may have expectations that could lead them to “see what they are looking for” in respondents’ answers. ○ Whereas your online confidential or anonymous survey does not involve a face to face meeting, an interview does involve two humans coming together for an exchange ● In face-to-face interviews the interviewer and respondent meet to complete the interview. ● Telephone interviews are less expensive than the ftf interviews and are more efficient. ○ Don’t require travel ○ In a computer-assisted telephone interview (CATI),a computer shows questions and records responses. ○ With interactive voice response (IVR) technology, respondents can respond via the telephone keypad, or speech recognition. ● A focus group is an interview with a group of about 6 to 10 individuals brought together for a brief period of time (usually 2 to 3 hours) ○ Any topic can be explored within a focus group ○ Questions tend to be open-ended ○ A researcher is NOT asking a series of close-ended questions but posing a question for the group to respond to and then interact with one another Survey Designs to Study Changes Over Time ● When researchers wish to make comparisons over time, they may administer the same questions year after year. ● In a panel study, the same sample of subjects is studied at two or more points in time. ○ Must use the exact same individuals ○ You can see if attitude,s behaivors, & experience schangeh overtime (i.e., how coping strategies relate to udnergraudte students may change with their experience) ● Three-wave panel study: three surveys are provided. ○ Example: You can provide an online survey to incoming freshmen in the class of 2020 to gauge their attitudes, preparedness, stress, experience and then you can survey them again at the end of freshmen year and again as they enter senior year; must Sampling from a Population ● Sample: the members of a population selected to participate in a research investigation. ● Population: all individuals of interest to the researcher. ○ EX. Population- Undergraduate students at the UW of WI madison; sample members selected to participate in study about undergrad wellbeing

12

● With proper sampling, we can use information obtained from the participants who were sampled to estimate characteristics of the population as a whole, not just the individuals we sampled ● Confidence interval: the level of confidence that the true population value lies within an interval of the obtained sample. ○ Gives you information about the likely amount of error ○ A larger sample size will reduce the size of the confidence interval (more likely to yield data that accurately reflect the true population value) ● Sampling error...