Objective items notes - Lissete Tolentino PDF

Preview

Summary

Lissete Tolentino...

Description

Learning Objectives •Know the advantage and disadvantage of objective items; •Know what should and should not do when create the objective items; •Create multiple-choice items, true/false items, matching items, sorting items, consequence items, and short-answer items. 1. Definition Definition:Any questions to which the response can be marked right or wrong without the need for expert/human judgment. Characteristics: 1. Students do NOT need to form or provide the answers in their own words. 2. Teachers do NOT need to involve subjective judgment when grade the items. Advantages and Disadvantages -

Advantages

-

Easy to sample widely across domain of interest Highly manageable (Short period of administration time, Easy/fast to score,

-

Cheap) Raise mean achievement and fewer missing answers Students like them

-

Higher reliability

Disadvantages - Hard to write quality items -

Usually test surface level processing

-

Does not reveal thinking process Guessing factor

-

Recognition of answer & test skills

-

May require good reading level

-

Reusing tests may be problematic (to the same group)

MCQ’s Components Typically MCQs have three parts: a) a stem, b) the best/correct answers(the keys), and

c) several wrong answers(distractors/alternatives). Advantages Less influenced by guessing than T/F Distractors provide diagnostic information Can use item analysis to improve Performance can be compared from class to class and year to year (Reliability) Disadvantages Require good reading level Constructing good distractors is time consuming. •Difficult to determine why individual students selected incorrect responses. Exercise Which of the following learning outcomes is most likely to require the best-answer type of multiple-choice item? A.Can justify methods and procedures B.Can identify proper grammar usage C.Can distinguish between fact and opinion D.Can understand specific facts

Number of response options •Typically 3-5. (4 or 5 is better than 3 because more alternatives reduce guessing and increase discrimination). • •4is most common, but 3 is defensible when an acre is larger/smaller/equal to a hectare? •Don’t get the number to 5 options by adding All of the above None of the above Variations in Style

. It also can be the following format: Choose the best word for each sentence: He looked for an [accomodation, acommodation, accommodation, acomodation] at the hotel. Circlethe underlined letters to show where capitals are needed. Tomorrow is saturday and i am going to visit rangi in taupo. 2. Options can be labeled from A to E. It is also common that they are labeled as I, II, III, IV… or 1. 2. 3. 4… etc.à Make sure the labels are consistent. Tips for Writing GoodMCQs’ answers Check, check, re-check! Keep clear & concise – “specific is terrific!” 1.Required:Do not repeat words in stem Use paraphrase;Do NOT copy the questions directly from a studied source;Check the answer is not elsewhere in paper. 2.Required:Make sure the answer is written well (grammar error/typo); is actually correct (not); is sufficient to answer the question (e.g. trash bin). 3.Required:Only one correct answer (otherwise should be stated in the instruction) 4.Required: Emphasise NOT if you must ask a negative question 5.Required:Place correct answer at random (A answer is most often) 6.Recommended:Arrange in a logical order -alphabetical, numerical, time series … 7.Recommended:Ask a Question (What is the name of this tool?) 8.Recommended: Avoid improbable qualifiers -e.g., never, always 9.Recommended:Options are similar to each other- length, the style as answer Generating distractors Use common misunderstandings •Be Plausible –Not silly or plainly wrong –Connected to • - a commonly held misunderstanding, or • - an over generalisation or a narrowing of application •

•Method: Present the questions to students as an open-ended item, and then collecting the incorrect responses. A bad example •Which of the following phrases best describes the process of evolution? a.The development of man from monkey-life ancestors. b.The change of simple to complex organisms. c.Marine mammals evolved directly from the fishes. d.When you step on the foot, the month will open, therefore, the human was developed from the trash bin. Objective Questions: True-False Items 1.Definition: the questions are usually a declarative statement that the student must judge as true or false. 2. Binary choice –True or False (T/F) –Right or Wrong (R/W) 3. True-false questions are typically used •To measure the ability to identify whether statements of fact are correct; •To evaluate student understanding of common misconceptions. Exercise T/F items are most useful for measuring which of the following learning outcomes? A.Distinguishing historical facts. B.Identifying the dates of historical events. C.Understanding specific historical facts. D.Selecting the reason a historical event occurred. MultipleTrue/False Items Improve the guessing factor by grouping three (or four) similar or relevant T/F

items.Therefore, to earnonemark, you have to get all three correct. Compound True/False Items Improve the guessing factor by asking for correction if answer is false. Tips for Writing Good True-False Items 1.Required:One idea only E.g. Professor X studies Engineering at UF 2. Required: Statement must be absolute T/F E.g. UF is the 7thbest university in USA 3. Required: Statement should be short and clear 4.Required: Don’t emphasize the trivial E.g. Who is the first president of the U.S.? Who is the driver of the first president of the U.S.? 5.Recommended: The number of true statements and false statements should be approximately equal. 6.Recommended: Don’t arrange answers in a pattern (i.e., TTFFTTFF) 7. Recommended: Provide the true/false column for the examinee to circle Matching Items Requires association of a major topic with correct sub-topic –Languages—vocabulary and definitions –Geography & Science—terms and definitions –Mathematics—equations and definitions Tips for Writing Good Matching Items 1.Required:More options than prompts (best if 2 more in options than the prompts); 2.Required: Keep lists fairly short and homogeneous (15 prompts or less); 3.Required: Put response in some logical order (chronological, alphabetical, etc.); 4.Required: Specify the directions (which list or columns is the options); 5.Recommended: Put all items on a single page;

6.Recommended: Use letters and numbers (drawing lines can get messy if students change minds); 7.Recommended: The short list usually on left. Sorting (classifying) into Categories Many curricular areas require students to be able to impose a pattern on a group of objects. Here are characters from Little Red Riding Hood Put them into 2 groups according to whether they are ‘good’ or ‘bad’ The wolf, the mother, the grandmother, the woodsman,Red Riding Hood Sample Categorisation This is more challenging if there are more than two categories The challenge is scoring Give a score per item being classified, then divide by number of categories Sequencing •A useful cognitive and curriculum ability is to be able put things in the correct process order –Glass-making process –How icebergs are formed –How human digestion takes place –What is the value of x for x2+ 2x – 5 = 0 •Sequence materials are provided and students have to organise them in correct order –No writing, but requires thinking –Scoring is challenging Sequence or order Write the numbers 1 to 6 in the first column to show the correct order for

developing a multimedia project. Scoring is problematic Work out what is the maximum value for getting it completely wrong, then work out the student’s distances from correct. Completion Items 1.Definition: it requires students to supply an important word, number, or phrase to complete a statement. Blanks are provided to be filled in by students. Tips for Writing Good Completion Items 1.Required:Short answer (one word, number, or phrase) 2.Required: Indicate units of measurement for numerical responses 3.Required:Avoid grammatical or other clues to the correct response. 4.Required:Be sure there are only one or a few of correct responses. 5.Recommended:Use question format 6.Recommended:Put the blank spaces at end of the sentence General Rules 1.Plan ahead (write the items in advance of test administration); 2.Write more items than needed; 3.Prepareanswer key or expected answers in advance of test administration; 4.Matchitems to intended learning outcomes/learning objectives (T/F questions may not be used to assess students’ creativities); 5.Emphasizekey tasks rather than small details; 6.No textbook verbatim questions; 7.Have another teacher read it and answer it. (Ifthey don't get the same answer that you got, it's probably because you didn't write it well.) General Rules for Writing/Language 1.Be clear and unambiguous; 2.Writeconcisely (15-30 words);

3.Be grammatically correct (free of spelling and typing errors); 4.Avoid jargon and academic language; 5.Write each item at an appropriate reading level (who will answer the question?); 6.No unintended/irrelevant clues. General Rules for Organizing 1.Begin with easy questions and then move to difficult ones; 2.One correct answer (or providing clear instructions); 3.Similar formats/items grouped together; 4.Independent items (don’t take answers from one and put it in another question/item); 5.Diagramsand tables above, NOT below; Common flaws Flaw 1: small details Poor example: The product-moment coefficient of correlation was developed by a.JohnGosset b.Sir Ronald Fisher c.Karl Pearson If an item on the product-moment coefficient of correlation is to be included in a test, it should concern some basic understanding or skill useful in determining when and how to apply the technique. Better example: The product-moment coefficient of correlation is used to determine the degree of relationship between a. two dichotomous variables. b. a dichotomous variable and a continuous variable. c. two continuous variables. Flaw 2: unintended clues Poor example: A test which may be scored merely by counting the correct responses is an _______________ test.

a.consistent b.objective c.stable d.standardized e.valid ● ● ● Better example: A test, which may be scored merely by counting the correct responses, is said to be _______________. a.consistent b.objective c.stable d.standardized e.valid Flaw 3: repeating the words in stem again in the responses •What is the right way to matcha pretest and posttest result? •A. Match each individual’s performance on pretest and posttest. •B. use the mean of the whole class’s performance on pretest and posttest. •C. normalized all individuals’ performance. •D. classify the performance into 3 levels. • If you use the word “match" in the question stem, do not use the same word in only one of the answers, as it will lead people to select that specific response. It will be better when using“pair” etc. Flaw 4: overlapping response questions Poor example: What should be the index of difficulty for an effective mastery-model test item? Less than 0.1

Less than 0.2

More than 0.8

More than 0.9

Options/Alternative should be mutually exclusive. Better example: What should be the index of difficulty for an effective mastery-model test item? Approximately 0.1

Approximately 0.2

Approximately 0.8

Approximately 0.9

Flaw 5: more than one idea Original: World War II was: A. The result of the failure of the League of Nations. B. Horrible. C. Fought in Europe, Asia, and Africa. D. Fought during the period 1939-1945. The alternatives should be homogenous. Better: In which of these time periods was World War II fought? A. 1914-1917 B. 1929-1934 C. 1939-1945 D. 1951-1969 E. 1961-1969 Flaw 6: negative statement •Poor: TheSupreme Court is not composed of nine justices.

T

F

T

F

• •Improved: The Supreme is composed of nine justices.

•Negative statement is hard (especially for non-native speaker) to decide whether T is agree or disagree with the statement. •Unless it is necessary, do not use negative statement. •If you are going to use negative statement, make sure that you highlight the “NOT” etc. Flaw 7: double negative statement

Poor: Preparing the table of specification is notan unwise idea for designing the tests. Better: Preparing the table of specification is a wise idea for designing the tests. Flaw 8: too many irrelevant words/sentences The purchase of the Louisiana Territory, completed in 1803 and considered one of Thomas Jefferson’s greatest accomplishments as president, primarily grew out of our need for a.the port of New Orleans b.helping Haitians against Napoleon c.the friendship of Great Britain d.control over the Indians Too many clauses mess up text items. Solution: Use simple sentences. Flaw 9: using qualitative definition Poor During what age period is thumb-sucking likely to produce the greatest psychological trauma? A. Infancy B. Preschool period C. Before adolescence D. During adolescence E. After adolescence Better During what age period is thumb-sucking likely to produce the greatest psychological trauma? A. From birth to 2 years old B. From 2 years to 5 years old C. From 5 years to 12 years old D. From 12 years to 20 years old E. 20 years of age or older Flaw 10: jargon

Poor Example: A test is valid when it: •a. produces consistent scores over time. •b. Correlates well with a parallel form. •c. Measures what it purports to measure. •d. can be objectively scored. •e. has representative norms. • Better Example: The validity of a test may be determined by •a. measuring the consistency of its scores. •b. comparing its scores with those of a parallel form. •c. correlating its scores with a criterion measure. •d. inspecting the system of scoring. •e. evaluating the usefulness of its norms. Summary: Flaws •It is not a white or black decision. Whether a specific item is appropriate or is a flaw is depending on your learning objectives and the characteristic of the test-takers. • •For example, the example used to explain the “Jargon” flaw usually only provide clues to students who are familiar with measurement and test theory. Likewise, if you are a psychology student, you will not be confused by the difference between “before adolescence” and “during adolescence” as you are familiar with the development theories. • Therefore, pay attention to the possible flaws rather than been fettered by them...