Essentials:

a. A general knowledge of measurement theory

 

b. Explicit and well-formulated goals for test use

 

c. A consideration of alternative and supplemental measurement procedures, especially examinees coming from very different backgrounds.                                        

d. A careful review of the development of tests being considered for use.                                                   

e. The competence to administer each test used for decision-making purposes.

                                         

f. Established conditions for test administration that enable all examinees to do their best.

                                         

g. A rationale for decisions based on test scores.

                                         

h. A consideration of alternative interpretations of scores.

                                          

i. The correct use of norms and the avoidance of scores that have                  technical problems—for example, age or grade scores (see Chapter 7).                                                          

j. Periodic reviews of the effectiveness of tests and decision-making         procedures for institution tests.

             

*Other issues in _____ for professional judgment; tests can be useful tools, but they can also be dangerous if misused*

 

They revolve around technical competency and wise professional judgment.

How many constraints do we have in assigning numbers to a phenomenon (relations between attitudes matches relations between                numbers which can be an ordinal restraint)?

Interval constraint: movie x > movie y > movie z

          therefore,

       2 m(y) = m(x) + m (z)

1. “Linear Linear Linear”: Figure 2.9 (Curvilinear relationship); you have to examine the scatterplot; most human characteristics are linearly related; the single best statistical tool that you have is your eyeball… always eyeball the scatterplot…

2. “Not Causal Not Causal Not Causal (no matter what the structural equationists people say)”: Correlation does not equal Causation.

3. Linear transformations of x or y have (potentially) two effects (i.e., changing of the signs v changing of the magnitude) (e.g., interview (x), job performance (y) you may want to conduct a linear transformation of x or y or both; if conducted it does not change the magnitude of the relationship; however, the transformation may change the sign of the correlation…

One of the most common mistakes with testing is researchers will not triple check the direction of the scales they use; so the sign of the correlation will be backwards from what they think it is (published papers even have this error)… Example: Job performance measured by supervisor ratings; supervisor ratings are on a 1 to 10 scale where 1 represents really poor job performance and 10 represents really great job performance scores; you find a positive correlation for structured interviews; when you conduct study later, you get data from another company and plug it in and then you find a negative correlation; the problem was that your data was in a different format (e.g., rankings) and the order of the scale was reversed… this mistake happens most often when the authors are reading other articles and these authors don’t specify the direction of the measures… Example: personality test scores, introversion/extraversion, if you are writing an intra/extra test, you can choose your scores and authors don’t always have consistent ordering of scales… so the problem is that researchers are unable to find what the direction of the measures meant when they conduct literature reviews… don’t make this common mistake!

If you go and look at the correlation b/w amount of antibiotics administered and mortality rate, there is a positive correlation… seems like a mortality is caused by antibodies but this is not true; usually antibodies are prescribed to severely ill patients along with other factors; rather than reporting the mortality rate, describe the survival rate… Presentation of the data…

4.  Correlations that deal with marginal distributions: The distribution of x without regard to y; the distribution of y without regard to x.

            Quick review: Variability, differences in scores (e.g., interview scores); common                                        Measures of variability: standard deviation, standard error, range, variance.

 5. Correlations that deal with conditional distributions: Distributions of y for a given x.

             So you can have a conditional distribution of x with a bunch of ys or vice versa.

     If you put a regression line through a cloud of x, y points (shows a graph), and you have a large sample you can calculate the mean squared ys from the actual ys… s²_y·x (symbolic notation for conditional distribution); for every single value of x that you have, if you calculate, add, and take the square root… one particular x score how many possible y scores.

6.  Standard error of estimate: in a perfect world, if job interview predicted job performance, variance would be zero and the SEE would be zero and predictions would be perfect; the lower the SEE the better (tests are used to measure people and predict their behaviors; like interview skills to predict job performance)! If the y’s are close together, low SEE, if the y’s are spread apart from each other, high SEE (which is NOT good).

2 reasons for the importance of the SEE: 1. The classic index of test score accuracy (you can

                                                                      determine whether to buy a test from the SEE).

                                                                  2. It’s used to construct confidence intervals around test                                                                     scores.

            Very important when you are comparing individuals’ data (e.g., SAT scores, GRE scores); so the SEE is the true measure of the closeness of individuals’ scores and determines the accuracy of test scores… Item stats (a way to shut TA’s down)… One little note: common mistake, don’t confuse SEE with Standard Error of the Mean (standard deviation of the sampling distribution)… sampling distribution is important.

*Be careful with regards the way in which you describe correlations (e.g., correlations does not equal causation)!

7.  Variance Explained (r²): An index of predictability NOT causality (if I have an interview score for person y, can I predict how well they will perform on the job?); accuracy of test.

         Example: Relationship between interview and job performance is .4 and variance is .16. Precise definition of .16, is sixteen percent of the variance in job performance (if you look at people’s job performance not everybody performs at the same level) can be predicted by a person’s interview score.

        *Often in academia, r², is not used; r is often used in academia because the sign is not obscured compared to r² (caution: be careful of the direction of the relationship of the variance for either your or other published tests.). Reverse score items (Fred is wary of use of this method); equation 2.25 (in textbook).

8. Correlation is directly related to linear regression: Quick and dirty review of linear regressions (drawing graph, scatterplot, and attempting to fit a line (or curve) to the dots, he’s showing line fitting… how do we determine where the put the line? Principle of least squares (we are minimizing the squared deviations to get rid of the sign or so they don’t add up to zero) another method of fitting a line to points on a scatterplot is maximum likelihood.

9. Linear Regression Functional Equation: How are the x and y in the correlations related?

            Y = a + bx

Functional equation: y (hat) = r_xy (S_y/S_x) (x_i – x(bar)) + y(bar)

                 The Ss in this equation are the proportion of the two variances (they are a correction factor). If you know how far off they are off from the average x, then you know how far off they are from the average y and you get the regression equation… (review)

10. Confidence intervals of correlations (alpha level, If we want to construct a confidence interval; with some percentage level of confidence, we are sure that the score that we saw from the person is that person’s actual score. Ex: when you take a test, you get a score. A person is brainwashed and doesn’t remember the test. If test is taken for the second time, you will not get the same score. Why? Preparation, sleep, etc.

*When discussing error, reliability, etc. you have to go back to the basics to go to the root of low scores, predictions, etc. (e.g., test taking conditions)… are you putting people in test taking situations where they will be resistant to factors in class or will you place people in situations where small disruptions can have large effects on test scores… This is a very important point.

Confidence intervals measure all of the distractions that enhance or detract from people’s test scores; an index of how much you have to worry about extraneous variables…

Confidence interval equation: y (hat) ± z_c S_y·x

 “95% is not written anywhere in stone on the planet”- Frank Schmidt

        You have to give an explanation as to why you chose that number…

         S²_y·x = conditional variance and S_y·x = SEE

11. Confidence intervals are great but here is the catch:

     They depend on three assumptions:

            a. The relation between x and y (humans have relationships, variables have “relations”) is linear; confidence intervals depend on the assumption that x and y are linearly related (be careful for his tests by looking at the data on a graph).

            b. The conditional distribution of y is normal (this assumption is often violated).

            c. The conditional distribution of y for every single x that you have is the same (i.e., homoscedasticity).

   When you use confidence intervals, check these three assumptions; if these three are not met, CI’s will be misleading.

12. Correlations are influenced by two main factors: 1. Range restriction (attenuator)

                                                                                    2. Range enhancement: Artificially inflates correlation coefficient.

            Ex. of Range Restriction: Let’s say you are working in a business and you are giving a test to applicants and this test will determine who should and shouldn’t be hired; if you are conducting the test you are using people who have been hired already, hence it’s range restriction; population is very big but the test sample is very small very limited.

            Ex. of Range Enhancement: Let’s say you’re doing educational research and you are interested in gifted kids and you go to a classroom at a school with gifted kids and the reality is that there are gifted kids (some recommended by teachers, some based on tests, some based on class availability) and as a result the sample is bigger than your original sample population. Hence correlation will be bigger than it ought to be. Range enhancement happens more rarely but occurs in education, sociology, etc.

13. Correlations are also influenced by lack of reliability or measurement error (these terms are the same thing); lack of reliability can occur in the x (predictor) or y (predictor 2).  The lack of reliability always attenuates the correlation.

*Normally observed correlation is an underestimation... Let’s say you’re looking at correlation b/w SAT scores and how good they will be as teacher (teacher rating), if the reliability of SAT is .88, and the reliability of the teacher rating is .7, observed correlation is going to be .78 of the actual correlation; you will underestimate the correlation by .22.

14. Correlations are influenced by sampling error (attenuator); observed correlation will not be true correlation if you are trying to extrapolate to the gen. population.

15. Correlations are effected by moderator variables (moderator is a third variable that affects the direction and relation between two variables; look at page 35 Figure 2.15 in the text book; Figure 2.16… this figure is about if the different combo of the groups changes the direction and relation between the variables).

The covariance divided by the product of the standard deviations OR the average product of the standard (Z) scores for the two variables (x and y).

1. two dichotomous variables: phi coefficient

2. two artificially dichotomized variables: tetrachoric correlation coefficient

3. one true dichotomous and one artificial continuous variable: point-biserial                               correlation

4. artificially dichotomized variable and one continuous variable: biserial correlation.

A subset of the GT theory; it is a simple, quite useful model that describes how errors of measurement can influence observed scores. The model assumes certain conditions to be true; if these assumptions are reasonable, then the conclusions derived from the model are reasonable; if not, model leads to faulty conclusions…

1. X = T + e(unsystematic error)

       (observed score = true score + error)

    “True” score has a very unfortunate name b/c it’s not really the “true score”.

    E.g., true typing ability score doesn’t reflect one’s true typing ability. In reality true score is the mean of the test scores (??).

2. Expected value of X = T (true score)

    If you could repeatedly conduct tests over and over expectancy is the mean of the frequency distribution.         

                                                            Implications:

        1. Theoretical, that the population will be the             same individual that will take the same                 test at the same time (theoretical).

                                                                                 2. The relationship b/w the test score and                 the real traits is validity.

3. Error scores & true scores are uncorrelated; how   could they be related? Example of teacher who puts noisy troublesome students up front and good students in the back; as a result good students      will do worst on test b/c they can’t hear instructions of the teacher v good students (so big error correlated with big ability)…

4. Error on different tests is uncorrelated; how could they be correlated?                                       

Big 5 personality test, five test; hence this assumption can be false with regards to the test score being affected by effects of the environment, such as fatigue, practice, examinees’ mood, stereotype threat; how do you fix it? Have a homogeneous testing environment and use common sense.

Homogenous testing environment across three domains:Testing conditions, examinees, time

 A discussion of stereotype threat…

          Social facilitation studies (1897), social facilitation is the fact that people’s automatic nervous system sensitivity goes up in the presence of other people v being along which can be either good or bad… a little bit of anxiety or nervousness is a good thing but too much is a bad thing                                                                   (e.g., stage fright)… Yerkes Dodson (??) curve… How far down does the social facilitation phenomenon go? Down to cockroaches (hmm…)

5. Error scores on one test are uncorrelated with true scores on another test; how could they be correlated? Moderating variable… g factor, if you    have a test that is highly g loaded that requires a lot of general intelligence, it’s possible that g score on this test (true score) is correlated with error scores on other g related tests… How do you fix it? You should try to make sure that validity is high as you can possibly get it.

      **An error of measurement is unsystematic error**

Problem is that some error can be systematic…

6. Parallel Tests: One’s true score on one version of the test must equal must equal one’s true score on another test (w/some level of variability) AND the error variance on one test has to equal the error variance on the other test; this deals w/in a given population.

      **Note that this mean the true score variances and error scores variance will be the same for both tests but the observed scores will not be the same  for both tests**

7. τ-equivalence tests: True scores are equivalent but error variances don’t have to be.

Validity study. The measurement instrument produces data to be used in making decisions or reaching conclusions. In a _-study, basically it is a form of a validity study, what are using the test scores for? What decision are you making with the test scores (another term within generalizability; how do I decide who should go to therapy v who shouldn’t)?

7 Different Definitions of _______:

1. C of measurement (is your instrument measuring the same way every time you use it? do you want it to be ___ across time, people, testing conditions? At the very least specify the conditions you are looking for ____ in)

2. Correlation b/w observed scores on parallel tests (r = correlation).

3. Proportion of variance in X explained by a linear relation with X’, where X is a score from a test and X’ is a score from a parallel test.

4. Ratio of true score variance to observed score variance.

5. Squared correlation between true scores and observed scores.

6. R coefficient = 1-r² (squared r) between observed scores & error.

7. 1-ratio of error variance to observed score variance (we don’t want error variance to be really big compared to the observed score variance; we want to keep the errors low)

A desired level of _____ is around or above .80 (.70 minimum; but in reality .8 on up)…

*You make sure that your test score can be interpreted by various persons in the same way*

You give people test, wait, and re-administer the test. Problems with this method: carryover effects (boredom—one does worse on test 2 b/c he/she has already read items on test—test item, memory, practice, etc.); know thy construct!

            What should be the appropriate time interval? Appropriate time where person doesn’t change and a time that is long enough…

You have to make a correction for the fact that you have shortened the test and reliability is related to the # of items; ______ formula solves this issue (what would occur if the test was longer)…

When you add more items true score variability increases more than the error… there is a catch, untrained people don’t know that the addition of item that you are plugging in actually works.

1. Observed scores are the sum of n test items (test items have to be independent and binary, right and wrong).

2. Your true score (defined differently in CTT) is your probability of getting a given item right (this implies two different things, one it is operating at the item level not at the test score level each item functions as its true score; two, true score is expressed as a probability, a knowledgeable person has a probability of scoring higher than a less knowledgeable person)… so at first it was “what grade did you get,” but the real score lies with regards to the test items (reason why people don’t do IRT and why CTT exists is b/c that IRT requires lots of items and lots of people compared to CTT; if it didn’t require lots of items and lots of people, we could use it more and improve tests).

3. Assumption that the aforementioned probability is the same for all items (60% of getting first item right, 60% chance of getting second item right, so on and so forth)…

Jumping ahead for a second, don’t confuse unidimensionality with composite scores (a lot of tests are used for making decisions regarding selection, grade school advancement, etc.); you often are measuring different things (e.g., professors performance, teaching, research, and service) but you should be sure that different measures are measuring the same constructs…

One note about binomial error model is that it doesn’t give you confidence interval levels; let’s say you have an examinee with an observed score and what’s the probability of getting a certain score on the test if their true score is a certain score; what’s the probability of Ann getting score

x if her true score is y? You have to make a lot of judgment calls.

A term we need to know (regarding the binomial error model): Zeta, which means true score (assumption #2 for the binomial error model)…

There is also out there a compound binomial model called “compound binomial error”… (binomial error model when you are uncomfortable that the standard error of measurement is the same from person to person)…

1. Factorial validity: Do you have a unidimensional construct? If not, how many dimensions are in your construct?

       E.g., human emotion (2 dimensions of human emotions)…

    Note: it is really easy to come up with a hypothesized effect of “emotion” but we use this construct very sloppily…  KNOW THY CONSTRUCT!!!

2. Convergent validity