Psychometrics - For BLEPP PDF

Preview

Summary

Psychological Testing everything from administration, scoring to interpretation of test scores; process of measuring psychology-related variables by means of devices or procedures designed to obtain a sample of behaviour. Characteristics of a Psychological Test 1. Objective Freedom from the subjecti...

Description

Psychological Testing - everything from administration, scoring to interpretation of test scores; - process of measuring psychology-related variables by means of devices or procedures designed to obtain a sample of behaviour. Characteristics of a Psychological Test 1. Objective Freedom from the subjective influence of the examiner. 2. Standardized Uniformity in the administration and interpretation of results. 3. Reliable Consistency of scores obtained by the same persons when reexamined with the same test on different occasions or with different sets of equivalent items. 4. Valid Concerned with what the test measures and how well it does so. Psychological Assessment - gathering and integration of psychology- related data for: - what: making a psychological evaluation, - how: tests, interviews, case studies, behavioural observation and specifically designed apparatus/ measurement procedures. Process of Assessment 1. Collaborative psychological assessment (Constant Fischer) - assessor and assesee may work as partners; - assessee: viewed as expert about his/her current views and remembered life events; 2. Therapeutic psychological assessment (Stephen Finn, et al) - an approach that encourages therapeutic self-discovery and new understandings through the assessment process. 3. Dynamic psychological assessment - a model and philosophy of interactive evaluation involving various types of assessor intervention during the assessment process. Objectives of Psychological Assessment 1. diagnosis; 2. etiology or cause of behavior; 3. prognosis or anticipated course of the symptoms; 4. treatments that may ameliorate or alter that course; and

5. the degree of functional impairment in both routine and specialized life functions. Requirements of Psychological Assessment 1. Identifying the problem to be addressed; 2. Selecting and implementing methods for extracting the information needed; 3. Integrating sources of information around the original purposes; 4. Reporting opinions and recommendations. Rights of Test Takers 1. Informed consent 2. Relevancy 3. Confidentiality and privacy 4. Results 5. Least stigmatizing label Assumptions about Psychological Testing and Assessment 1. Psychological Traits and States Exist. Trait any distinguishable, relatively enduring way in which one individual varies from another. State distinguishes one person from another but is relatively less enduring. 2. Psychological Traits and States can be quantified

and measured.

3. Test-related behaviour predicts non-test-related behaviour. 4. Tests and other measurement techniques have strengths and weaknesses. 5. Various sources of error are part of the assessment process. 6. Testing and Assessment can be conducted in a fair and unbiased manner. 7. Testing and Assessment benefit society. Tools of Psychological Assessment 1. Psychological tests 2. Interview 3. Case history data 4. Behavioural observation 5. Role play tests

6. Computer assisted psychological assessment 1. Test-retest reliability - an estimate of reliability obtained by correlating pairs of scores from the same people on two different administrations of the same test. - coefficient of stability 2. Parallel-forms/alternate-forms reliability - coefficient of equivalence - parallel: means and variance of observed test scores are equal; - alternate: different version of a test that have been constructed so as to be parallel. 3. Split-half Reliability - is obtained by correlating two pairs of scores obtained from equivalent halves of a single test administered once. - estimate of inter-item consistency - odd-even reliability - spearman brown formula: correction formula to account for shortening/lengthening of a test. 4. Kuder-Richardson reliability - KR-20 - is used when items of a test are scored on an all or none basis; - coefficient alpha 5. Inter-scorer reliability - is the degree of agreement or consistency between two or more scorers/judges/raters; - is often used for tests of creativity or projective tests; - coefficient of inter-scorer reliability 1. Reliability coefficient 2. Standard Error of Measurement

Standard Error Measurement - standard error of a score - provides an estimate of the amount of error inherent in an observed score.

- a tool used to estimate or infer the extent to which an observed score deviates from a true score. - an index of the extent to which one individual’s scores vary over tests presumed to be parallel. - useful in establishing the confidence interval, range/band of test scores that is likely to contain the true score. Trinitarian view - classic conception of validity - construct validity is visualized as the umbrella validity since every variety of validity falls under it Face Validity - refers to what a test appears to superficially measure to the person being tested than to what the test actually measures; - described as the “Rodney Dangerfield of psychometric variables” Content Validity - a judgment of how adequately a test samples behaviour representative of the universe of behaviour that the test was designed to sample. Validity Ratio (CVR) - developed by Lawshe, measures agreement among raters regarding how essential an individual test item is for inclusion in a test. Criterion Related Validity - a judgment of how adequately a test score can be used to infer an individual’s most probable standing on some measure of interest --- the criterion. Criterion - standard against which a test or test score is evaluated. Concurrent Validity - an index of the degree to which a test score is related to some criterion measure obtained at the same time. Predictive Validity - an index of the degree to which a test score predicts some criterion measure. Both are based on 2 types of statistical evidences: 1. Validity coefficient 2. Expectancy data Incremental Validity

- degree to which an additional predictor explains something about the criterion measure that is not explained by predictors already in use. - predictor measures are included only if they can explain something about the criterion measure not known from other predictors. Expectancy table - provides information that can be used in evaluating the criterion-related validity of a test. - shows the percentage of people within specified test-score intervals who subsequently were placed in various categories of the criterion. Selection ratio - a numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired. - percentage of people hired under the existing system for a particular position. - extent to which a particular trait, behavior, characteristic or attribute exists in the population expressed in proportion. Hit rate - proportion of people a test accurately identifies as possessing/exhibiting a particular trait, behavior, characteristic / attribute. Miss rate - proportion of people the test fails to identify as having/ not having a particular characteristic/attribute. False positive - a miss wherein the test predicted that the examinee did possess the particular characteristic/attribute being measured when the examinee did not. False negative - a miss wherein the test predicted that the examinee did not possess the particular characteristic/attribute being measured when the examinee did.

Construct Validity - judgment about the appropriateness of inferences drawn from test scores regarding individual standings on a construct. Construct - an informed, scientific idea developed/hypothesized to describe or explain behavior. Convergent Validity ( convergent evidence)

- scores on a test tend to correlate highly in the predicted direction with scores on older, more established and already validated tests designed to measure the same ( or a similar) construct. Discriminant Validity (discriminant evidence, divergent validity) - insignificant relationship between test scores and/or other variables with which scores on the test should not theoretically be correlated. Error - component of the observed test score that does not have to do with the examinee’s ability. True variance - variance from true differences Error variance - variance from irrelevant random sources. Transient error - a source of error attributable to variations in the examinee’s feelings, moods or mental state over time. Factor Analysis - designed to identify factors/specific variables that are typically attributes, characteristics, dimensions on which people may differ. - employed as a data reduction method where several sets of scores and the correlations between them are analyzed. - purpose: identify the factor or factors in common between test scores on subscales within a particular test or the factors in common between scores on a series of tests. Exploratory Factor Analysis (EFA) - entails estimating or extracting factors, deciding how many factors to retain and rotating factors to an interpretable orientation; Confirmatory Factor Analysis (CFA) - a factor structure is explicitly hypothesized and is tested for its fit with the observed covariance structure of the measured variables. Nature of the Test 1. Homogeneity vs. heterogeneity of items the more homogenous, the better 2. Dynamic vs. static characteristics anxiety vs. intelligence

3. Restriction or inflation of range 4. Speed tests vs. power tests True Score Theory - estimate the portion of a test score that is attributable to error. Domain Sampling Theory - estimate the extent to which specific sources of variation under defined conditions are contributing to the test score. Item Response Theory - latent trait theory, focuses on the extent to which individual test items are useful in evaluating individuals presumed to possess various amount of a particular trait or ability. Generalizability Theory - based on the idea that a person’s test score vary from testing to testing because of variables in the testing situation. - given the exact same conditions of all the facets in the universe, the exact same test score (universe score) should be obtained. Generalizability Study - examines how generalizable scores from a particular test are if the test is administered in different situations. Coefficient of generalizability - influence/impact of particular facets on the test score (reliability coefficient). Decision Study - designed to tell the test user how test scores should be used and how dependable those scores are. - involves the application of information from the generalizability study; - examines the usefulness of test scores in helping the test user make decisions.

Norms - are transformations of raw scores into a more meaningful scale derived from the performance of a large sample of persons representative of one or more specified groups. Norms referenced testing - an individual’s score is interpreted by comparing it with the scores obtained by others on the same test. Criterion referenced testing - first proposed in 1963 by Glaser, this term uses as its interpretative frame of reference a specified content domain rather than a specified population of persons.

Developmental norms - indicates how far along the normal developmental path an individual has progressed. 1. Mental Age a child’s score on a test corresponds to the highest year level or age level that he can successfully complete. 2. Grade Equivalent assigns achievement on a test or battery of tests according to grade norms. 3. Ordinal Scale designed to identify the stage reached by the child in the development of specific behavior functions. - refers to the uniform progression of development through successive stages. Within group norms - the individual’s performance is evaluated in terms of the performance of the most nearly comparable standardization group. 1. Percentiles - expressed in terms of the percentage of persons in the standardization sample who fall below a given RS. - indicates the individual’s relative position in the standardization sample. 2. Standard scores - derived scores which uses as its unit the SD of the population upon which the test was standardized. 3. Deviation IQ - standard score on an intelligence test with a mean of 100 and an SD that approximates the SD of the Stanford-Binet IQ distribution. Item Analysis a. Index of Item Difficulty b. Index of Item Discrimination c. Spiral Omnibus Format

INTELLIGENCE - (Binet) reasoning, judgment, memory and abstraction; - (Wechsler) An aggregate or global capacity of the individual to: a. Act purposefully; b. Think rationally; and c. Deal effectively with his environment. Academic Intelligence (Sternberg, et al.) Verbal ability, problem-solving, ability and social competence;

Everyday Intelligence (Sternberg, et al.) practical solving problem ability, social competence, character and interest in learning and culture. Theories on Intelligence 1. Interactionism - complex concept by which heredity and environment are presumed to interact to influence the development of one’s intelligence. 2. Factor-Analytic Theories a. Two-factor Theory of Intelligence (Charles Spearman - g : portion of the variance that all intelligence tests have in common; - s: specific components - group factors: neither as general as g nor as specific as s. b. Multiple Intelligence (Gardner’s) - linguistics

- musical

- logic-mathematical

- interpersonal

- spatial

- intrapersonal

- bodily- kinesthetic

- naturalist

c. Cattell’s Theory on Intelligence a. Crystallized intelligence - formal/informal education and skills/knowledge dependent on exposure to a particular culture b. Fluid Intelligence - nonverbal, relatively culture-free and independent instructions.

Horn’s revision: c. Vulnerable abilities - decline with age and tend not to return to preinjury level following brain damage; d. Maintained abilities - tend not to decline with age and may return to preinjury levels following brain damage. Horn’s additional factors: 1. visual processing (Gv)

2. 3. 4. 5. 6. 7.

auditory processing (Ga) quantitative processing (Gq) Speed of processing (Gs) Facility with reading/writing (Grw) short-term memory (Gsm) long-term storage & retrieval (Glr)

d. Three-stratum theory of Cognitive Abilities - heirarchical model: all abilities listed in a stratum are subsumed by or incorporated in the strata above. - top stratum: generalized intelligence (g) - second stratum: composed of 8 abilities and processes

1. fluid intelligence (gf) 2. crystallized intelligence (Gc)

5. broad auditory perception (U) 6. broad retrieval capacity

(R) 3. general memory and learning (Y) 7. broad cognitive speediness (S) 4. broad visual perception (V) 8. processing/decision speed (T) - third stratum: narrowly defined abilities and processes e. CHC model (Cattell-Horn & Carroll Model) - McGrew-Flanagan CHC model - has 10 broad stratum abilities Decision/reaction time or speed (Gt) Crystallized intelligence (Gc) Quantitative knowledge (Gq) Reading/writing ability (Grw) Long-term storage and retrieval (Glr)

Visual processing (Gv) Auditory processing (Ga) Processing speed (Gs) Short-term memory (Gsm) Fluid intelligence (Gf)

3. Information-Processing View

Wechsler Intelligence Scales The principal goals of a Wechsler Scales administration are threefold: - to assess current and/or premorbid levels of

intelligence;

- to test or generate hypotheses about the presence of organic brain dysfunction and psychopathological conditions; and - to make predictions as to how these conditions will affect the client’s response to treatment.

WAIS-R (18-75 years old)

VERBAL

PERFORMANCE

1. 2. 3. 4. 5.

Information Digit Span Vocabulary Arithmetic Comprehension 6. Similarities

1. 2. 3. 4. 5.

Picture Completion Picture Arrangement Block design Object assembly Digit symbol

WISC-R (6-18 years old)

VERBAL 1. 2. 3. 4. 5. 6.

PERFORMANCE 1. 2. 3. 4. 5. 6.

Information Similarities Vocabulary Arithmetic Comprehension Digit Span

Picture Completion Picture Arrangement Block design Object assembly Coding Mazes

Wechsler Preschool & Primary Scale of Intelligence ( 4 – 6 ½ yrs)

VERBAL 1. 2. 3. 4. 5. 6.

Information Vocabulary Arithmetic Similarities Comprehension (Sentences)

PERFORMANCE 1. 2. 3. 4.

Animal House Picture Completion Mazes Geometric Design 5. Block Design

Subtests 1. Information 2. Comprehension

WPPSI X

X

X

X

X

4. Arithmetic

WAIS X

X

3. Similarities

5. Vocabulary

WISC

X

-

X

X

X

X

X

X -

6. Receptive Voc.

X

-

7. Picture Naming

X

-

-

8. Digit Span

-

X

X

9. Letter-Num

-

X

X

10. Picture Comp

X

X

X

11. Pic. Arrg

-

-

X

12. Blk Design

X

X

X

13. Obj. Assm

X

-

X

14. Coding

X

X

-

15. Sym Srch

X

X

X

16. Mtx Rsng

X

X

X

17. Dgt Sybl

-

-

X

18. Wrd Rsng

X

X

-

19. Pic Concpts

X

X

-

20. Cancellation

-

X

-

Subtests and their descriptions: 1. Information: - taps general knowledge, learning and memory; 2. Comprehension: - taps social comprehension, the ability to organize and apply knowledge;

3. Similarities: - taps the ability to analyze relationships and engage in logical, abstract thinking; 4. Arithmetic: - taps learning of arithmetic, alertness, concentration and short-term auditory memory; 5. Vocabulary: -a good measure of intelligence; 6. Receptive Vocabulary: - taps auditory discrimination and processing, auditory memory and the integration of visual perception and auditory input; 7. Picture Naming: - taps expressive language and word retrieval ability; 8. Digit Span: - taps auditory short-term memory, encoding and attention; 9. Letter-Number Sequencing: - taps attention, sequencing ability, mental manipulation and processing speed; 10. Picture Completion: - draws on visual perception abilities, alertness, memory, concentration, attention to detail and ability to differentiate essential from nonessential detail; 11. Picture Arrangement: - taps the ability to comprehend or “size up” a situation, attention , concentration and the ability to see temporal and cause-and-effect relationships; 12. Block Design: - draws on perceptual-motor skills, psychomotor speed, and the ability to analyze and synthesize; 13. Object Assembly: - taps on pattern recognition, assembly skills, and psychomotor speed; 14. Coding: - draws on factors such as attention, learning ability, psychomotor speed and concentration ability; 15. Symbol Search:

- taps cognitive processing speed; 16. Matrix Reasoning: - taps perceptual organizing abilities and reasoning; 17. Word Reasoning: - taps verbal abstraction ability and the ability to generate alternative concepts; 18. Picture Concepts: - the ability to abstract as well as categorical reasoning ability; 19. Cancellation: - visual selective attention and related abilities. The Wechsler Scales can be used to develop hypotheses about the quality and character of the client’s: A. Cognitive - Problem-solving skills ( concrete and abstract operations, integration, c...