Practical 4 Designing a questionnaire PDF

Preview

Summary

Practical: Questionnaires...

Description

Practical 4: Designing a questionnaire

Introduction to the Lab Report Assessment Getting Started: Ethics •What is the ethical status of this project? •What ethical problems might we encounter? Getting Started: Designing Questions •How can you write good questionnaire questions? •What are the important features that our questionnaires should cover? Covers Observational Methods (the two lectures and the practical class) A Canvas test to complete to complete while working through content of four powerpoints (designed by Dave Leavens) Things to note: •Not timed •Can be done at home •Can be started and then continued later (AS LONG AS YOU DON’T PRESS SUBMIT) •Press submit by Week 7 deadline How do we measure the immeasurable? •Self-efficacy related to learning statistics: confidence in their ability to learn statistics Developing a new questionnaire to measure this construct Report should cover: •Is our questionnaire a good way to measure this construct? Does questionnaire cover the factors which fall under self efficacy and learning stats. What do you get out of the questionnaire •What factors (ie primary elements) make up the construct? •Is the questionnaire consistent and reliable? •Does it correlate with other questionnaires about related constructs? If it is a good one if ours measures self efficacy it should score correlation to someone else's self efficacy questionnaires

The Assessment •Due in Week 11 – the week after Easter break The Report •2,000 word HARD LIMIT (no 10%!) •Excludes abstract, table/figure captions, references, and appendices •(Almost) “Typical” lab report structure •Abstract, methods, results, discussion, references, appendices •BUT no intro •Non-”typical” lab report content •Focuses on scale development and validation •Final results: is this a good questionnaire or not? •NOT: does X predict Y? Do people in different groups score higher/lower? -

Draw conclusion is it a good questionnaire not the results

Write items to be included •Start today, finalise next week Complete the questionnaire Conduct analyses once data is released •Factor analysis •What is the underlying structure of the construct? •Reliability analysis •Does the questionnaire measure consistently and reliably? •Validity analysis •Does the questionnaire measure the construct we think it is? Write up the report •See Canvas for detailed help!

Ethics for Research : Three major components: •Ethical Review

-

Are the tasks safe and/or are appropriate safeguards in place?

•Informed consent -

Dont need to know the theory of what were doing they just need to know theyre going to be safe. Does Not need to be full disclosure Participants know what they’re getting into They agree to do so without deception or coercion Fine for a little bit of deception but withhold some proportion of the truth Results will be moe important than the little deception created in the study

•Data protection (ie anonymity, confidentiality, and right to withdraw) -

What happens to the information that you get in your research - can include safeguards before and after

You will be assigned a research ethics problem to work on •Discuss this problem and agree on a resolution •Look up sources to back up your decision Once time is up, you should be prepared to share your answer with a different person/group • Take notes and have clear points in mind! Some potentially helpful resources: •The BPS website/ethical guidelines The Sussex ethical clearance site

Your questionnaire is about self-efficacy, but telling your participants this may bias the way that they answer questions (because these qualities may be socially desirable). How can participants give informed consent if they don’t know what the questionnaire is about? Food for thought: •What is (or is not) informed consent? •What do you have to tell participants ahead of time? •Is withholding the purpose of the study “deception”?

Your questionnaire contains a few items that may be very upsetting to a very small

group of people. It’s unlikely, but possible, that your participant recruitment will include people from this group. The items in question are the crucial element of your study, so you can’t remove or rephrase them. How will you mitigate the risk of doing harm? Food for thought: •What options are there for completing the questionnaire (or not)? •How can you check whether you did happen to recruit a member of this vulnerable group? •What precautions can you put in place to help mitigate the negative effects? Ethics for research task 3: To avoid GDPR and data storage issues, you want to anonymise your data at point of collection so that no case can be connected to a particular person in your dataset. How do you still maintain a participant’s right to withdraw? -

Give participants a number

Food for thought: •How can you uniquely identify cases aside from using personal information? •What is the difference between “anonymous” and “confidential”? -

Confidential = not disclosed to 3rd parties but researcher knows whos data is who Anonymous = participants know but no one else does Way to avoid any problems best way if you don't need to identify participants anonymise at collection.

Reform groups that contain at least one person who worked on each task For each task, recap what your group discussed •What points did you find difficult, or have to look up? •What conclusions did you come to? •Were there any points you didn’t agree on, or couldn’t find a good solution to? Ethics should always inform your research design •Can substantially impact the data/results

•Required for approval to run the study • Looking forward to final year projects •You will be required to submit a full ethics application! • This project already has low-risk ethics approval •We must abide by low-risk restrictions for item writing, recruitment, and data protection Ethics should always inform your research design •Can substantially impact the data/results •Required for approval to run the study • Looking forward to final year projects •You will be required to submit a full ethics application! • This project already has low-risk ethics approval •We must abide by low-risk restrictions for item writing, recruitment, and data protection

Writing Questionnaire items Discrimination •Distinguishes between people with different levels of the psychological construct in question Validity •Measures what you think it measures Reliability •Measures consistently across people and situations

Discrimination : Score differences On a measure of arachnophobia (fear of spiders): •Andy got 30 points •Graham got 15 points

•Vlad got 10 points Points to consider: •Is Andy twice as phobic as Graham and three times as phobic as Vlad? •Is Graham more similar to Vlad than to Andy with regard to his fear of spiders? → both get score of 2 but a likes socialising and b likes alcohol Proportional difference between scores on questionnaire -

Dont want people to agree to everything or nothing

Validity What it means: •Questionnaire does what it says on the tin •That is: it measures what you think it measures Examples of non-valid measures: •Using brain size to assess intelligence •Using date of birth to assess personality This means that… •The questionnaire should include all important aspects of the construct (content validity) •Questionnaires that (are meant to) measure the same thing should correlate highly with each other (convergent validity; i.e., aspect of criterion validity) -

To check validity of your questionnaire measure get them to complete another valid questionnaire at same time to check its measuring what you want it to measure

Reliability: What it means: •Questionnaire measures the construct consistently This means that… •Items on the scale correlate well with each other and represent the construct cohesively (internal consistency) •Administering the same questionnaire to the same people twice will give very similar results (test-retest reliability)

Familiarity with the construct On Canvas Week 4, find the resources for today’s topic Using the recommended readings, the Internet, or any other resources, brainstorm with your partners: •What is the definition of this construct? What are the scientific vs layperson definitions? •What thoughts, behaviours, and characteristics typify this construct? •What does it look like to score high on this construct? •What does it look like to score low on this construct? •What questionnaires already exist to measure this construct? •What problems or gaps do these measures have? •How can you improve on these existing measures?

Brainstorm 4+ items Write items that are: •Neutral statements (not questions) •Suitable for answering on a 5-point Likert scale ranging from “strongly disagree” to “strongly agree” •Ethically low risk! • Include items with reverse meaning: •Positive: A “strongly agree” response indicates higher self-efficacy •Negative: A “strongly agree” response indicates lower self-efficacy

Start small ¡Begin with a specific, focused idea and build out, rather than getting bogged down in a large and complex idea What factors do you think (from your research!) will contribute to your construct? ¡Construct items to measure these aspects

¡Your factor analysis will show you whether your items have captured these factors successfully (and whether they contribute to your construct as you thought!) Think about what behaviours might reflect your construct ¡Eg instead of “I am an efficacious person”, try “I can quickly learn how to use statistical analysis software”

Generate LOTS of items and then pare down ¡Don’t try to make the perfect items first thing! RESEARCH FIRST ¡Inform from other questionnaires and previous research Include reverse meaning items ¡EG optimism ¡“I like to imagine the best possible outcome” ¡“strongly agree” still indicates high optimism ¡Reverse meaning: “I like to imagine the worst outcome.” ¡“strongly agree” indicates pessimism...