Psychometrics

-      Assessment

Four Pillars of Psychological Assessment

-      Formal Test

-      Interviews

-      Formal Test

o   Published and standardized and researched you know how good it is because of research

-      Interviews

-      Observations

o   Art to knowing what is relevant and what to observe, both restricted and free environments

o   Things you assess while searching for something else, may lead from one to another

Assessment

-      Anything we use to help us gain information, earliest done in China bur around since we studied behavior

World War I (1914-1918)

-      Where assessments took hold in this country, referred to as modern era of assessment

-      Using psychological tests to classify recruits, use info to get people into appropriate jobs

-      Device was developed for this:

o    Army Alpha, group given personality test

-      Interest in psychological tests is peaked at this time, psychologists used them for everyone – everywhere

-      This led to rights getting abused when done rampantly without a goal or plan in mind

o   group given personality test

o   grandparent of the current classification test (verbal)

o   Found a huge illiteracy rate, entrants couldn’t read or write to take the test

o   Developed Army Beta (nonverbal) still group administered

Civil Rights Act (1964)

-      Encompassing for everyone everywhere

-      Used as foundation for further legislature for psychological testing

Diana versus Board of Education (1971)

-      Plaintiffs were all Hispanic with English as second language with tests in English and scoring badly, so they were put into special education and though to be retarded

-      Violated by giving test without parents permission, then it being in English, then segrating them.

-      Changed by setting had to have parents permission for testing, then tests in first language, then if found deficit parents had to permit them to be in a special class

-      Applied to everyone in California any nationality then other states addressed similar issues and got to a critical mass

-      Federal Law to get parents permission in first language, and parent permission for placement

-      Gave a lot of jobs to psychologists

-      Created comparable legislature in jobs, institutions, etc.

Grigg’s Decision:

-      Anything that accompanies a job must be relevant to the job

-      Example: Interview, tests, applications, etc.

-      EEOC the watchdog of the industry (Equal Employment Opportunity)

Legislatures

-      Happened because of individual cases repeating over different states

-      Laid the foundation

Case of Larry P. (1970’s)

-      In California a class action suit parents of African American child

-      Intelligence testing were bias and put the children at a disadvantage because it was inappropriately normed

-      Law passed that you couldn’t use conventional intelligence tests for these children in California

-      Then in Illinois they didn’t pass the law

Good test must have:

-      Reliability

o   How consistent a test is, if not consistent it is worthless

-      Validity

o   Does it measure what we want it to, the accuracy of the test

-      they are interrelated , but different

-      must have reliability to have validity

-      Discrete Data (Dichotomous)

o   Fits into a specific category

-      Naturally occurring traits or characteristics can be either

-      Continuous Data

o   Exists in some degree

o   Example: height, weight, speed, intelligence, etc.

o   More complex data is continuous

-      Naturally occurring traits or characteristics can be either

Parametric tests

-      Used for continuous data

-      Some degree

Non-parametric tests

-      Used for discrete data

-      Specific Category

Statistical Scales

-      Nominal

-      Ordinal

-      Nominal

o   Non parametric/Continuous Data

o   Example: Name, social security number, etc.

o   not very descriptive, but a good start, can use for anything with a name

-      Ordinal

o   Non parametric/Continuous Data

o   Names and ranks order

o   Tell a little more than nominal

-      Interval

o   Parametric/Discrete Data

o   Name, rank, and numbers

o   Qualitative jump, can tell amounts and intervals

o   Can’t give absolute rates

-      Ratio

o   Parametric/Discrete Data

o   Absolute zero

o   If you’re going to measure something can’t conceive zero

Bar Graph

-      Discrete, nominal, and ordinal

Bell Curve

-      Continuous, ratio, interval

Population

-      Entails everything that contains a certain attribute we are measuring

-      Very large, but can be small as well

-      All members that have a certain trait

Sample

-      Same population small section representative of the population

-      Must reflect the population

Two qualities the sample must have to reflect the population

-      Variety/Hetergeneous/Diverse

-      Large (largeness alone will not ensure diversity)

Statistics

-      Allows us to collapse our data

-      Take a lot of data and make it more manageable

Mean of Central Tendency

-      One number that best represents the data/sample

-      Mean/average, median, and mode

Mean

-      Average

-      Used often

-      Is very susceptible to high or low scores

-      Example: 1, 2, 2, 1, 3 average would include and reflect scores

-      Example: 1, 2, 2, 1, 30 average would not reflect central tendency

Median

-      Fifty percent of scores above and fifty percent of score below

-      Not really used in statistics

Mode

-      Not used in statistics

Measure of Variability/Dispersion

-      Shows the diversity in the sample

-      Standard deviation, range, variance

Standard deviation

-      Scores show central tendency and how much dispersion

-      Measure of error

Range

-      Easy to get can’t do anything with it

Normal Distrubution

-      Bell shaped

-      Evenly distributed

-      If sample is large and diverse and SD is low you should have normal distribution

-      Mesokurtic

Skewed distribution

-      Refers to the shape

-      Positively Skewed:

-      Negatively Skewed:

-      Bimodal Distrubution:

-      Positively Skewed:

o   Skewed to the right

o   Because mean is more positive than it should be so it throws it off

-      Negatively Skewed:

o   Skewed to the left

-      Bimodal Distrubution:

-      if put positive and negative skewed together

Kurtosis

-      steepness of the curve

-      Platykurtic

-      Leptokurtic

-      Platykurtic

o   sample with good diversity but low number

o   still symmetrical but not bell shaped

o   add subjects to fix it

-      Leptokurtic

o   Extreme population

o   Large number sample, but low diversity

o   Symmetrical but not normal

o   Normal distribution

o    high number of sample and high diversity

Standard Scores

-      Can convert between T, Z, and A scores because normal distribution

-      All have mean and standard deviation

-      Based on interval data

-      Z Distribution, Z score

-      T Distribution, T score

-      A Distribution, A score

-      IQ Scores

-      Z Distribution, Z score

o   X = raw score

o   Z score = raw score expressed in standard deviation

o   Raw score minus the mean divided by standard deviation

o   In Z distribution the mean is always zero

-      T Distribution, T score

o   Mean equals 50

o   SD equals 10

-      A Distribution, A score

o   Mean equals 500

o   SD equals 100

-      IQ Scores

Norms and Standardization

-      Statistically based

-      Norms

o   Point of reference

-       Standardization

o   Process of getting norms

-      Types of Norms

  o   Age

o   Grade

o   Percentile (%ile)

o   Age

§  Not based upon interval data

§  Difference between 2-3 yrs old is not the same as 17-18 yrs old

§  They collapse after a certain age

o   Grade

§  Not equivalent 1^st grade to 2^nd and 11^th to 12^th

§  Also not interval data

o   Percentile (%ile)

§  Example: 60^th %ile means 60% of people scored below you and 40% person scored above you median = 50^th %ile

o   Stanine (Standard Nine)

§  Divides into nine section

How we standardize a test

-      Get a sample, has to be large and heterogeneous

Randomization

-      A tool that helps get diversity, random sample

-      Everyone in your population has the same chance of being chosen for your sample

-      No randomization you have a quasi-experiment, not a true experiment

-      Very difficult usually use variations

Stratified Random Sample

-      Divide the members of population into regions then you randomly select individuals from the naturally occurring regions

-      Difficult to do as well, very expensive

Purposive Random Sample

-      Section of country that does a good job of representing the entire population

-      Can work if the area is a micro causem of our population

Incidental (Convenience) Random Sampling

-      Can make this work

-      Example: live in Fairmont, take sample from Fairmont State

Correlations

-      Tells us how two variables are related

Regression

-      Logical outgrowth of a correlation

-      When we use one variable to predict another

-      R = coefficient, R stands for regression

Two major types of correlation

-      Positive/Direct Correlation

-      Negative/Indirect Correlation

-      Number less than one

-      Gives us an idea of how right or wrong our prediction is

-      Tells us the strength of our correlation

-      Must square correlation to interrupt it

-      The sign + or – DOES NOT tell us the strength

-      The number DOES tell us the strength

-      Positive/Direct Correlation

o   both up or both down

-      Negative/Indirect Correlation

Correlational Test

-      Used depends on what kind of data you have

-      Whether your variables are discrete or continuous

Linear relationship

-      Between 2 variables of results, they don’t change positive is always positive and negative is always negative

Curvilinear Data

-      Some degree of continuous data but somewhere there is a indirect change

-      What was positive is now negative and so on then you must use eta

Error

-      Could be subject itself

-      Actual test presenters/examiners

-      Actual test itself

-      Never can eliminate error, just keep at a minimum

-      Acknowledge the existence and deal with it

-      Reliability and validity help us tell how much error there is

-      Could be subject itself

o   Hawthorne Effect

o   Example: sick, unprepared, on purpose, etc.

o   Rosenthal Effect

o   Example: persuasion, bias, unprepared, etc.

-      Actual test itself

o   Example: unclear, bad directions, not normed, etc.

Qualities of test

-      Any assessment device should have reliability and validity

-      Validity = accuracy

-      Reliability

o   Consistency of any predictor

o   X = true score plus error

§  Predictors:

·         Anything which we use to forecast or predict something

·         Examples: observations, interviews, research, tests.

§  Test-Retest Reliability

·         Example: give the 1^st test, then time passes and you give the 2^nd test

·         Interpret the percent directly, all other correlations you must square it

·         Advantages:

o   straight forward

·         Disadvantages:

o   difficult to do again

o   Less individuals return

o   May have learned, they are not the same people (carryover effect)

o   Impractical

o   Takes times and resources

§  Equivalent/Alternate Form Reliability

·         Same material, same format

·         Different questions

·         Example: give subjects form A at one point in time, then wait and give form B and correlate those results

·         Advantages:

o   straight forward

·         Disadvantages:

o   may not be equivalent

o   difficult to do again

o   Less individuals return

o   May have learned, they are not the same people (carryover effect)

o   Impractical

o   Takes times and resources

§  Internal Consistency

  Only one administration of test

Comparing test to itself

Three components of Internal Consistency

Split-Half (Odd-Even)

Cronbach’s Coefficient Alpha

Kuder-Richardson 20 (KR20)

o   Split-Half (Odd-Even)

§  treating the one test as two tests, splitting it in half, compare the even items to the odd items (recommended)

§  first half (1-25) to last half (25-50) (not recommended because a person can get tired or run out of time.)

§  Formula is called the Spearman-Brown Formula

o   Cronbach’s Coefficient Alpha

§  Inner comparisons to every other item. (Ex: item 1 to 2, then 2 to 3, etc.) with continuous data: interval or ratio (Inner item comparisons)

o   Kuder-Richardson 20 (KR20)

§  Inner comparisons to every other item. (Ex: item 1 to 2, then 2 to 3, etc.) with discrete data (Inner item comparisons)

How consistent our predicator is…

-      what is the likelihood of getting the same result at a later point in time, comparing the one administration to another.

-      Rare to get the same exact result but there is an acceptable range. Based on correlation and standard deviation of the test.

Standard deviation:

-      measure of error, want it to be the lowest number, measure of diversity, more extreme the more variability

Standard Error of Measurements (SEmeas):

-      Standard error is directly related/correlated to the standard deviation of the test, whereas the standard error is negatively/inversely correlated/related to the reliability.

-      Don’t have to calculate but must be able to explain and understand how it works

-      (formula written) Standard Deviation times the square root of one minus the reliability (the more error the more dispersia and vice versa, want high reliability)

Standard Error of Difference (SEdiff):

-      Measure very different attributes and characteristics (modalities) of the same individuals, that measure different things. (apples and oranges)

-      Example: Verbal skills versus Performance skills different but both contribute to mental abilities

-      (Formula written) Square root of the first test standard error measurement plus the second test standard error measurement

Factors that affect reliability:

-      Configuration of your sample

o   large

o   heterogenous (diverse)

-      Items that comprise our test

o   large numbers (the larger the more reliability the test will be)

o   content homogenous in nature

Inter-rater reliability:

-      Can two different individuals using same instrument and get the same results

-      Kappa statistic

o   type of correlation that measures inter-rater reliability

Validity:

-      does the predictor measure what you want it to measure (accuracy)

-      most important quality- has to be accurate

-      independent from reliability but related

-      in order for any predictor to be valid it must be reliable (in order for it to be accurate it must be consistent) but just because the given predictor is reliable does not mean it is valid.

-      Three Main types of validity: content, criterion-related, and construct

-      Content Validity

o   Very important in any test that measures achievement or accomplishment

o   Non-statistical in nature, no real coefficient (number) must have experts in the field agree the content is appropriate.

-      Face Validity

o   Just looking at the test, does it look okay, not a true test of validity

-      Criterion-Reference Validity (Criterion Related Validity)

o   very statistical in nature

o   little v coefficient ??? (30 min)

o   relationship between score and ??? (30min)

o   Criterion:

§  some reference point

o   Two methods of Measuring Criterion:

§  Predictive Method/Validity

§  Concurrent Method/Validity

§  Predictive Method/Validity

·         Most statistically ideal

·         Doesn’t have a restricted range, but not practical

·         Restricted range:

o   very bad for reliability and validity, want diversity. (30min)

·         Example:

o   develop test, then give test to all applicants right up front, then have employer put all applicants on the job, then get ratings on the job by employers, applicants who score the highest on test should have been rated the highest as well, then you can use the test to predict the best employees, v=.70 must square the validity actually only 40% good prediction.

§  Concurrent Method/Validity

·         Not as sound

·         Example: give it to secretaries who are already on the job, as many as you can, the highest rated ones should score the highest on the test, the correlation is used as the coefficient.

·         Have a restricted range, but there is diversity in the job

Validity ranges:

-      .80 are good, even .70 isn’t bad, but once squared it is below 40%, you can used validity range even lower, because we use multiple predictors combined, we never depend on ONE predictor alone.

Multiple Predictors:

-      when we combine multiple predictors and we calculate the total package(battery)

-      the total won’t be lower than the validity of that single highest package

-       most likely the combined package will be higher

-       The high score brings up the low ones.

Two methods of Multiple Predictors:

-      Multiple Regression

-      Multiple Cut-offs

-      Multiple Regression

o   seems to be more fair

o   high scores compensate for low ones

o   Example: if you had two bad semesters early on in college career, but last two years were great, they would over look it because of good recommendations, and GRE scores, you might get into grad school anyway.

-      Multiple Cut-offs

o   More exclusionary

o   Must have a minimum score on each predictor to make the cut

o   Example: wanting to be in the Air Force, scored great on test, but didn’t pass vision test, you won’t get into the Air Force to fly planes.

Predictors & Criterions

-      can be anything

-      Predictors Example: tests, observations, etc.

-      Criterions Example: getting into grad school, good grades, etc.

-      We can set the predictor where ever we want, although we have no control over the criterion , instead we just try to foreshadow it

-      Example: make it easier to get into medical school, but a doctor cannot function with a low level of knowledge. It would be dangerous to patients.

-      We want to find a point that indicates the minimum score that reflects the criterion that still occurs

Hypothesis Testing

-      Four quadrants

o   False Negative/False Rejection (incorrect)

§  Example: scored bad, didn’t hire, would have been good

o   True Positive/Valid Acceptance (correct)

§  Example: scored good, we hired, they were good

o   True Negative/Valid Rejection (correct)

§  Example: scored bad, didn’t hire, would have been bad

o   False Positive/False Acceptance (incorrect)

§  Example: scored good, we hired, but they were bad

-      Correct and incorrect decisions based upon the testing

-      If you decrease the valid acceptance and the false acceptance, you will increase the false rejection and valid rejection and vice versa.

-      You can only move the predictor line left and right, CANNOT move the criterion line at all.

-      To make it harder to get in move predictor to the right, especially if it is costly. (Example: the Astronaut program)

-      To make it easier to get in more predictor to the left

-      Want to find a point that indicates the minimum score that reflects the criterion still occurs

-      We can set the predictor wherever we want, but we have no control over the criterion we try to foreshadow it

-      Example: make it easier to get into medical school, but a doctor can’t practice with a low level of knowledge it would be dangerous

Utility

-      Is how effective is the predictor in helping us make a prediction/decision

-      Utility is dependent upon three things:

o   Validity

o   Base rate

o   Selection Ratio

o   In order to have good utility we want high validity, high base rate, and low selection ratio low.

o Validity(Utility)

§  more valid the better of a predictor

§  want it as high as possible

o   Base rate

§  the percentage of correct decisions you can make without using that particular predictor

§  want it to be as high as possible even 50%

§  Example: to get into grad school look at other things beside the GRE such as GPA, interview, and reference letters.

o   Selection Ratio

§   Ratio of the number of openings and the number of applicants

§  The actual amount of individuals you can actually accept.

§  Want this to be low, have a lot of applicants and to increase other factors important (GRE)

§  Example: when it comes down to everything being equivalent, GPA, interview, and reference letters then you will go with the GRE.

o   Incremental Validity

§  Taylor-Russell Tables

·         These have taken every combo of base rates, selection ratio, and validity. (see handout)

·         Enhances predictive value

·         Test with zero validity is useless SD ratio can’t help

Construct Validity

-      Involved in pure research

-      It is statistical in nature; do statistical research to get the coefficient

-      Accumulated over time

-      Very difficult to come up single test or number to show it

-      Must show that it is measuring the construct we think it is

-      Very difficult to use

-      Construct

o   Abstract psychological concept

o   Way of explaining something, an idea, theory, or a definition

o   Example: intelligence, personality, motivation, etc.

-      Methods of measuring Construct Validity

o   Convergent Validity

o   Discriminate Validity

§  Test must correlate well/highly with measures of the same construct

§  How we do it? Example: create intelligence test of college students, then study, to see if intelligence tests correlate with mine

§  Must be consistent with other tests

o   Discriminate Validity

§  must not correlate with measures of a different construct, therefore it must discriminate

§  Example: your intelligence test should not correlate to personality tests, this would show your intelligence test would be more of a personality test instead

Two most fundamental traits of a test

-      Power

-      Speed/Time

-      Power

o   Individual is given enough time to perform the test, but the content is hard

o   Mastery of the content, produces the high scores

-      Speed/Time

o   Relatively easy items, but you have a limited time to get them all done

o   Strictly timed period

o   Mastery is not accuracy, but how fast you can get them done.

o   Don’t want to use speed test with internal validity, because you would cut your items down.

Suppressor Variables

-      Don’t require something in the selection process that you don’t have to perform in the job.

-      Example: Commissioned by Sears, to create a selection process for automotive employees,  both written test and hands on, but you didn’t have to read to do the job, but you required them to read for the test.

Moderator Variables

-      Identifies subgroups

-      Example: You get a large and representative sample, give a test, publish it, works well, but certain group/minority it doesn’t work, so don’t use it for the subgroup because it puts them at a disadvantage.