Study guide key PDF

Preview

Summary

This study guide gives you the correct answers ...

Description

MEC1 Test Study Guide

1. When a teacher wants to ensure that the content being measured on her test is consistent with the purpose of the exam, what is he/she measuring? Content validity 2. What is the purpose of construct validity? a. Determine whether an assessment correlates highly with another assessment b. Determine how closely the assessment represents the material is should be measuring c. Identify whether the assessment measures the intended concept and not an unintended variable. d. Analyze how well the assessment predicts outcomes of interest e. Identify the penalty for scoring low on an assessment 3. What is the purpose of content validity in objective assessments? a. Analyze how well the assessment can predict future outcomes b. Determine how closely the assessment represents the material that should be assessed c. Identify whether the assessment measures the intended concept and not an unintended variable d. Determine whether the assessment correlates with another assessment 4. What is the purpose of criterion-related validity? a. To see if an assessment is a reasonable method to get desired results b. Measure the degree of correlation between an assessment and second assessment c. Measure how well an assessment measures the intended concept and not an unintended variable d. Measure the degree of agreement between a theoretical concept and an assessment e. Measure how well an assessment measures what it is designed to measure 5. Determine whether each definition below describes each of the following validity types. Write the term in the left hand column. Terms: Consequential Validity, Concurrent Validity, Construct Validity, Criterion-Related Validity, Content Validity TERMS Consequential Validity

DEFINITIONS degree which results of an assessment is used for a particular purpose and has social consequences

Criterion-related validity

degree of correlation between results of one assessment and another

Construct

degree it measures what the specifications indicate it is intended to measure

Concurrent

degree an assessment correlated with an existing measure of that performance what is being measured is consistent with the exam’s purpose to measure how well an assessment device measures what it was designed to measure To measure the degree of agreement between a theoretical concept and an assessment

Content Content Construct

6. How do you ensure validity of a test? (Multiple Answers) a. The scoring procedures are evaluated for objectivity b. Test is evaluated for evidence of bias c. Test is evaluated for content consistency d. Scores are evaluated for test-retest stability over time e. Scores are evaluated with the Spearman-Brown correlation formula 7. What are threats to validity? (Multiple Answers) a. Using a homogeneous grouping of prompts b. Subjective scoring methods c. Administering a pretest d. Cheating by students e. Ambiguous test directions f. Grammatical errors 8. Determine whether each definition below describes each of the following reliability types. Write the term in the left hand column. TERMS: Internal, Equivalency, Inter-rater, Stability TERMS Internal

DEFINITIONS Divide a test into 2 comparable halves and correlate scores on those halves

Internal

Estimate reliability that is computed from a single administration of the test

Equivalency

The extent in which 2 or more forms of a test are consistent

Inter-rater

When a well-designed behavior is evaluated by expert judges and the scores are in agreement

Stability

Test administered and then repeated on the same subjects at future date; results are compared and correlated with first test

Inter-rater Inter-rater Internal

The consistency of a scoring system The extent in which 2 or more evaluators agree Measured by three different approaches: split-half consistency, Kuder-Richardson, and Cronbach’s alpha The degree of the relationship between scores The degree in which two tests measure the same content

Inter-rater Equivalency

9. Test-retest reliability is also known as what? Stability 10. Split-test reliability is also called what? Internal 11. Why is reliability essential? A. Ensure an assessment approximate the real-life situation being measured B. Demonstrate an ability to perform and maintain effectiveness C. Ensure an assessment measures the construct that it purports to measure D. Demonstrates a causal relationship between two variables 12. Which of the following are methods to determine reliability? (Multiple Answers) A. ANOVA B. ANCOVA C. MANCOVA D. Inter-rater E. Parallel Forms F. Split-half measure of internal consistency G. Cronbach’s alpha internal consistency measure H. Analysis of covariance I. Face validity J. Kuder-Richardson 13. High reliability coefficients show what? What about low coefficient numbers? High = highly reliable. Low – It has low reliability.

14. Which are characteristics of reliability coefficients? (Multiple Answers) a. Measure how well scores hold up over time b. Measure less than the square root of the correlation between two test versions c. Measures the degree to which items on a test are homogeneous d. Measure the closeness of judges’ scores

e. Measure the correlation between predictor and predicted f. Measure accurately reflects the concept being assessed

15. Determine whether each scenario below would have a high or low reliability coefficient: DESCRIPTIONS Constructed-response test Projective test (e.g., personality tests) Heterogeneous test groups Large number of test items Short test Significant disparity in students’ scores Several subtest components of one larger test Negligible standard error of measurement Significant standard error of measurement Minor differences in students’ scores Inconsistent rating of constructed response items

HIGH OR LOW RELIABILITY COEFFICIENT? Low (due to inter -rater) Low High High Low Low Low High Low High Low

16. Is the Spearman-Brown Correlation used for validity or reliability? Reliability 17. Determine whether each definition below describes each of the following assessment or evaluation types. Write the term in the left hand column. Formative, Summative, Performance Based, Placement

TERM Formative Performance Based Formative Placement Summative Summative Formative Formative Performance Based

DEFINITIONS Never graded; anonymous Provides information about how a student understands and applies knowledge in a real-world scenario Conducted to improve the quality of student learning and not for grading students Measure skills in relation to the abilities needed in certain classes Final course evaluation End-of-course survey to measure student satisfaction with the teacher Review of interim and final project deliverables by the development team Review of design and interface by internal and external designers to improve navigation Assessment of the students’ ability to conduct a Chemistry experiment

Summative Summative Summative Formative Formative Summative Summative Summative Summative Formative Formative Summative Performance Based Placement Summative Formative Formative Summative Formative Formative Summative Summative Summative Formative

A multiple-selection test taken at a test center Conducted at the end of a curriculum to address long-term effects of the instruction Assess the quality and impact of implemented curriculum Expose curriculum weaknesses to improve student learning Provide information to improve teaching Provide evidence for possible cause-and-effect relationships Evaluations investigating the demonstrable effect of curriculum Data outcome used to determine if a program should be discontinued Informs stakeholders of the extent to which curriculum met its goals Use information gathered in each stage of evaluation to revise the curriculum Allows midcourse corrections Determine cost-effectiveness Students create a map showing names and places in a novel Program entrance assessment Determine impacts and effects of instruction Assess organization of materials and presentation process of instruction Implementation Evaluation Impact Evaluation Evaluate student learning by using diagnostic surveys Evaluate student learning through ongoing classroom assessment techniques Measure if your technology skills improved to the level predicted in the program’s proposal Used to justify expenditures or funds for a program Measures effectiveness of the curriculum against original objectives Provides information for decision makers who evaluate program development

18. What is the purpose of a formative assessment? Improve/revise instruction – formative is about the curriculum and not about grading students. 19. Which is an element of a formative evaluation? A. Inductive analysis B. Affective evaluation C. Meta-analysis D. Implementation evaluation 20. What is the purpose of a summative evaluation? See if goals have been met; make decisions

21. What are the components of a summative evaluation?

 Identify the subject (e.g., product, program) of the evaluation.  Define the purpose of the evaluation. What do you want to know? Why? How will the results be used?  Identify stakeholders (i.e., those who will be impacted by the evaluation results).  Select the evaluation questions.  Plan the evaluation.  Determine what data you will need in order to answer your evaluation questions.  Determine where the data reside (i.e., sources).  Plan how you will gather the data.  Decide on your data collection methods.  Choose your data collection instruments.  Write the evaluation procedure. This is a narrative that chronologically describes what you will do throughout the evaluation, with a special focus on when and how all data will be collected. Explain each phase separately (do not combine phases).  Decide how you will analyze the collected data.  Implement the evaluation plan.  Gather the data.  Analyze the data.  Write the results section.  Interpret the results.  Report to stakeholders. 22. Compare/contrast constructed-response tests with selected response tests. Selected response – choose an answer; easier to grade; higher reliability Constructed response – construct an answer; higher order skills measured; harder to grader; lower reliability. 23. What are the components of a well-designed constructed response assessment? Clear stem/directions; provide ample space for answer; give details about what is required in answer 24. Determine if the statements below describe interviews or questionnaires. Write your answer in the left column beside the description. Interview or Questionnaire? Interview Questionnaire Interview Questionnaire Questionnaire Interview Interview Interview

Description Offer framed oral conversations that result in data collection Provide answers to well-defined questions in electronic format Provide opportunities for interaction between the researcher and participants Are aggregated through the use of electronic forms Provide anonymity for potentially sensitive responses Recognized as having a high contact and response rate Permit researchers to record responses and reactions Allow flexible inquiry process with a high response rate

25. Which of the following are types of QUALITATIVE data collected during assessments? (Multiple Answers)

Interval Circular Scale Discrete Ratio Nominal Test scores Ordinal 26. Which of the following are types of QUANTITATIVE data collected during assessment? (Multiple answers) a. Portfolio artifacts b. Dichotomous scores c. Polytomous scores d. Discrete variables e. Holistic scoring of essays 27. What is a technique for quantifying qualitative data? Please provide specifics. Coding/Indexing 28. Determine whether the data gathered in each scenario below is either QUALITATIVE or QUANTITATIVE. Write your answer to the side of the description. a. Administering a summative evaluation using student test scores Quantitative b. Using focus groups Qualitative c. Administering a survey with closed-ended questions Quantitative d. Obtaining academic records (e.g., GPAs) from management Quantitative e. Giving an essay exam Qualitative f. Using a Likert scale that asks students to rate items from 1 to 5 Quantitative g. Questionnaires with closed-ended questions to determine satisfaction Quantitative h. A comparison of pre-test and post-test scores Quantitative i. A semantic differential scale to indicate attitude on a continuum Qualitative j. Administering surveys with open-ended questions Qualitative k. Obtain tabulated statistical data from management information systems Quantitative l. Interviewing students to specifically capture scaled responses Quantitative m. Observing students to describe types of interaction Qualitative n. Generating theoretical insights through focus groups of students Qualitative o. Scores of assessments Quantitative p. Observations to record the number of times students are off task Quantitative q. Observations to record the amount of time a student remains on task Quantitative r. Observations to gather types of behavior seen during instruction Qualitative s. Time to complete a test Quantitative...