Two options:

1) Locate an existing selection device (buy it), or

2) Construct your own selection device (make it)

•  Both options start with job analysis to determine the KSAs that:
• are needed to perform the job’s essential functions,
• are needed on Day 1,
• the organization is not going to provide training on

Choosing an Existing Selection Measure

Need to clearly and completely understand the attribute or construct you want to measure
 
Read reviews and evaluations of the assessment that claims to measure the construct
 
Order a sample test and study any information relative to its reliability, validity, fairness, intended purpose, method of administration, time limits, scoring, appropriateness for specific groups, norm data, cost
 
Is it appropriate for my organization’s needs?

Advantages of Existing Selection Measures

Usually less expensive and time-consuming

Reliability, validity, and other data available

May be superior to what could be developed in house.

Typically the skills needed to develop an assessment are not available in house.  

Steps in Constructing a New Selection Measure

1) Analyze job for which a measure is being developed

Specifies the KSAs needed

Determines what constructs need to be assessed

2) Select the method of measurement

3) Developing specifications

4) Pilot test, analyze, and revise the measure

5) Conduct reliability and validity study on revised measure

6) If all good, implement the measure in the selection process

2) Method of Measurement to be Used    

Example methods – paper-and-pencil tests, work sample tests, interviews, etc.

• Issues that need to be considered:
• Skill of the individuals scoring/administering the test
• Number of applicants
• Skill level of applicants (e.g., reading level)
• Costs of testing and budget for selection
• If large number of applicants – paper and pencil test
• If  social skills are critical – behavioral exercise

Example checklist for matching the job requirements with the method of measurement for industrial electricians

3) Developing Specifications for the Selection Measure

Specifications to consider:

• Purposes and uses of the measure
• Nature of the population
• How behaviors or knowledge, skills, abilities, and other attributes (KSAOs) will be gathered and scored
• e.g., should items be multiple choice or open ended?
• Sampling designs and statistical procedures to be used in selecting and editing items, determining number of items, etc.
• How many total items?  Items per KSA to be measured?

4. Pilot test, analyze, revise

preferably at least 100-200 participants

Item analyses (e.g., % correct; by group)

Item fairness/differences among subgroups

Fix problems

5. Reliability and validity study on revised measure 

Are scores dependable for making decisions?

Are the test scores predictive of job success?

6. Implement test in selection process 

Collect norm data as you go

Start to finish the process on average takes about 3-5 years

interpreting scores on selection measures 

You got a 69 on a selection test.  Is that a good score?  How do you interpret that score?

• What if top score was 73?  What if everyone else got 73?
• Until you know how everyone else did, an individual score is hard to interpret

Norms show how well an individual performs with respect to a specified comparison group

• Need to use a relevant norm group
• Use local norms rather than national norm data (if available)
• Norms are transitory – they may change over time

Percentiles and standard scores are frequently used to interpret test scores with respect to norms

Percentile Scores

Shows percentage of persons in a norm group who fall below a given score

The higher the percentile score, the better a person’s performance relative to others in the normative sample

Percentiles specify how far a data point is from the bottom, not the top

Misuses of Percentile Scores

Are not percentage (raw) scores (69th percentile not same as 69% score (69 out of 100)

• Percentile scores are based on an ordinal scale of measurement, not a ratio scale
• Can only do greater than/less than comparisons
• Score at 60th percentile is not 2x better than score at 30th

Standard Scores

Indicate, in common measurement units, how far above or below the mean score any raw score is

These scores represent adjustments to raw scores so that is it possible to determine the proportion of individuals who fall at various standard score levels

Most common standard score is a z score

Z-scores measure how outstanding an individual is relative to the mean of a population using the standard deviation for that population to define the scale.

Note that percentiles use the median as the average (50th percentile), while z-scores use the mean as average (z-score of 0). 

The formula for computing z scores is

Z-score example

Z-scores allow us to compare scores from different populations

Comparing ACT and SAT scores; assume…

SAT tests: mean = 500, sd = 100

ACT tests: mean = 18, sd = 6

Compare SAT score of 680 and ACT score of 25

SAT z-score = 1.8   [(680-500)/100]

ACT z-score = 1.17   [(25-18)/6]

Which is a better score?