- Cannot be touched, felt, heard, tasted etc...

- Things like depression and anxiety

-Cannot be directly measured

- We rely on behaviors that accompany the construct

- Presumed to cause item score

- Rely on scale to measure variable although we are not interested in scale scores.... if studying depression, we are not interested in BDI scores but level of depression....

- cannot directly see what true score is

- always some random variability

- we are left with inter-item relationships

observered score=true score + measurment error

- it is the difference between the variable and how it is represented in measurment

Assumptions regarding error in CMT

- amount of error varies randomly

- error terms uncorrelated across items

- error terms not correlated with the true score of latent variable

Relationship between item score and true score (path coefficients)

- the cross product of path coefficients is equal to the correlation between the items

- standardized path coefficients represent the strength of the causal relationship between latent construct and items

- amount of influence from the latent construct to each item is the same

- eac item is assumed to have the same amount of error

- we know these things aren't true

- all items share a common latent variable

- latent variable need not exert same strength of relationship to all items ( path coefficients can be different )

- error variances can be different

- this allows for more accurate representation of magnitude of latent variables on items

- proportion of varaince attributable to the true score of latent varaible

- also consistency of measurment

- can be measured by internal consisitency (coefficient alpha)

- temporal stability (test- retest correlation)

- alternate forms reliability

- split- half reliability

• Test-retest reliability
• Assumptions: varaible is stable over time and there is no error associated with time of measurment
• Possible moderater involvement

• Kappa Coefficient

• Inter-rater reliability
• Two raters rate independently
• 2x2 matrix

                       Rater 1

                          +                   -

Rater 2       +   31                   6

                   -    11                 52

Pr(a)- proportion of agreement among raters (look at diagonal)

Pr(a)= .31 + .52= 0.83

Pr (e)- hypothetical chance agreement

Pr (e)= (.42)(.37) + (.58)(.63)

                 0.16     +     0.37= 0.53

k= Pr(a) - Pr(e)

       1 - Pr(e)

   = .83-.53

          .47

= 0.64

• Greater than 0.7 = acceptable
• 0.40 to 0.59 = moderate level
• 0.60 to 0.79 = Substantial
• Greater than 0.80 = excellent

- Validity coefficients is constrained by reliability

- increased reliability = increased power

• want difficult items on a test- should distinguish b/w high and low on characteristic being measured
• (want it around .5 unless dichotomous)
• average item difficulty slightly below midpoint between 1.0 and chance level
• should have range of item difficulty

di= item discrimination index =

(nhi/hi) - (nli/li)

Nhi= no. of persons in high scoring group that passed item

hi= no of persons in high scoring group

nli= no of persons in the low scoring group that passed item

li= no of persons in low scoring group

Relationship between item and rest of items on test

The item-total correlation test arises in psychometrics in contexts where a number of tests or questions are given to an individual and where the problem is to construct a useful single quantity for each individual that can be used to compare that individual with others in a given population. The test is used to see if any of the tests or questions ("items") do not have responses that vary in line with those for other tests across the population. The summary measure would be an average of some form, weighted where necessary, and the item-correlation test is used to decide whether or not responses to a given test should be included in the set being averaged. In some fields of application such a summary measure is called a scale.

The test

An item-total correlation test is performed to check if any item in the set of tests is inconsistent with the averaged behaviour of the others, and thus can be discarded. The analysis is performed to purify the measure by eliminating ‘garbage’ items prior to determining the factors that represent the construct;^[1] that is, the meaning of the averaged measure.

It is supposed that the result for a particular test on a given individual is initially used to produce a score, where the scores for different tests have a similar range across individuals. An overall measure for an individual would be constructed as the average of the scores for a number of different tests. A check on whether a given test behaves similarly to the others is done by evaluating the Pearson correlation (across all individuals) between the scores for that test and the average of the scores of the remaining tests that are still candidates for inclusion in the measure. In a reliable measure, all items should correlate well with the average of the others.

A small item-correlation provides empirical evidence that the item is not measuring the same construct measured by the other items included. A correlation value less than 0.2 or 0.3 indicates that the corresponding item does not correlate very well with the scale overall and, thus, it may be dropped.^[2][3]

Does the test measure what it purports to measure

The meaning of the scores produced by an instrument

Content validity-

Criterion Validity-

Construct validity-

degree to which elements of an assessment instrument are relevant to and representative of the targeted construct for an assessment purpose

Scale has content valididty when its itmes constitute a randomly selected subset of items drawn from the universe of items

• items are omitted that reflect important facets

• items measuring facets outside of domain are included

• aggregate score is disproportionately influenced by any one facet

an item of scale that has empirical association with some criterion or gold standard

also called predictive validity

Same trait- same method (reliability)- should be strongest

Same trait, different method- should be a bit lower (validity)

Different trait- different method- should be lowest

convergent- measures that assess same construct should be highly correlated

divergent- measures that asses different constructs should not be highly correlated

- used instead of correlation coefficient for accuracy

- 2x2 matrix with measure vs gold standard

- Diagnose each respondent (dichotomize)

- Dichotomize test scores

Test Result

                                      +                       -

                 Present      TP(hit)             FN (miss)

Diagnosis

                 Absent       FP (false alarm)  TN (true negative)

Probability of having a positive test result among those who really should be positive (with disorder)

          Hits         

Hits + Misses

The probability of having negative test result among those who really dont have disorder

          TN        

TN + Misses

Positive Predictive Power

The probability of having disorder among those whith a positive test result

                   Hits           

Hits + false alarms

the probability of not having disorer among those with a negative test result

           TN       

TN + Misses

Same as efficiency

How efficient is the test, overall, in accurately detecting presence and absence of pathology

       Hits + TN     

all cases

P

P' = Total - P (compliment)

Prevalence of diagnosis (in signal detection)

=

TP + FN

N

the average variance of your sampling distribution

To compute: You can find for TN, TP, FP, FN, EFF, Q, P

√ (TN)(TN')

N

Q

Q'= Total no. - Q

How many positive and negatives of test (in signal detection matrix)

= TP + FP

N

Mean (SE)= SE (1-pⁿ)

evaluator descides how large a sample to gather (N) and takes a random (or representative) sample of that size from the population of interest- each patient receives a diagnosis and a test

Unbiased estimator

a representative sample of size N_o is drawn from the populat and each person is diagnosed. This is the screening sample. The a random sample size of N₁ is drawn from the among those in the screening sample with a positive dx and a random sample size of N₂ from among those with a negative diagnosis. Both samples must have min. no. of 10 people.

This is a biased estimator

-More powerful

- power is maximized if outcome robability = .5

As cutoff changes for statistics of a test, the values associated with them will change as well.

Sensitivity and specificity have a negative relationship.

The line is chance level- greater distance is better- this is the ideal place for sensitivity and specificity tradeoff.

Bigger distance=better test

0.5 area is significant difference

- IRT to see poorly performing items

- Internal consistency?

- Kappa coefficient?

Item-factor loadings (>0.4)?

Look at the proportion of items performing poorly.

- estimate relative proportions of variance for your measure and others

- estimate unique variance

- examine interaction (moderator) efffects associated with sex, age, SES etc...

- degree of colinearity or shared variance among predictors

- strength of association between each predictor and criterion

- forced step-wise regression

- enter comparison measures

- enter new measure

- difference in R2 is the index of incremental validity

- R2 is variance accounted for- singularity is really bad- that means you aren't explaining any more variance

Sum items = estimate of latent construct

- compute item total correlations (obtained set)

- compute projected inter-item correlations

subtract projected inter item correlations and item total correlations

With a good model- they should be the same

We get beginnings of a residual matrix

- to identify latent structure of an assessment instrument

- item refinement and scale development

- relationship to content validity and construct validity of instrument

(if your not explaining varaince-> you are not capturing facets of constructs)

more theoretically driven, testing explicit predictions

- trying to see if model reflects real life (goodness of fit)

- need at least 5-10:1 ratio

- N>125

- independnet

-noncorrelated

When first doing CFA- you get a communality of 1 across all items

you need to regress item on all remaning items to get estimate of R2- which serves as an estimate of communality

Look for elbow

- elbow is how many factors you are extracting

- can handle measurement error

- can handle non-measurement error

- can reject models

- enables advanced treatment of mising data (full info max lilklihood)

- they are intervally scaled

- they have a multivaraite notmal distribution

- have sufficient sample size- need more for complex models

- at least 100 to 150

- effect of latent variable on the measure; if a measure loads on only one facotr, the standardized loading is the measure's correlation with the factor

- this can be interepeted by the square root of the measures reliability

- the varaince in the measure isnt explained by latent variable

- this doesnt mean its randon, just that its unexplained by latent variable

- the casaul and correlation links between latent variables

exogenous- not caused by another variable in model

endogenous- caused by one or more variables in the model

Components of a correlation

decomposing a correlation

Direct effect

a------------------------>y

indirect effect 

a---------------->b-------------------->y

spurious effect (common cause)

Y<------B---->A

Unaalyzable components (correlated causes)

B---->Y

                                   (

C--->A

- correlation b/w any varaibels equals the sum of the products of the paths or correlation from each tracing. 

- when we trace we need to set the varaince at 1, otherwise we have more unkowns than knowns in equation- makes it impossible

1) specification- describe nature of relationships among variables

2) Identification- can in theory and in practice be estimated with observed data

3) Estimation- the models parameters are statistically estimated from data

4) Model fit- the estimated model parameters are used to predict the correlations or covariances between measured variables and the predicted correlations or covariances are compated to the observed correlations or covariances

Sample size- higher is easier to get significant results (bad in SEM)

Correlation size- if higher in the model, the poorer the fit (significantly different)

comparative fit index

incremental fit index

goodness of fit index

adjusted GFI

CFI- takes into account for sample size (can be small)

IFI- compare model with saturated model and independent model

GFI- calculates proportion of variance estimated by covaraince  matrix

AGFI- does same but with saturated model (rewards for parsimony)

Root mean squared error of approximation

- model misfit

- lower the better

min of 0.07 or lower

Model evaluation

(theoretical, technical, and statistical)

theoretical- appropriateness of general causal structure, right variables and does it fit with previous findings

- technical- identification status, appopriate estimation method, and apprpriatness of instrumental variables (coninuous)

- Statisical - reasonable parameter values, good coefficients, endogenous are well explained, and model of fit

Model ID

(define and min. condition)

-Items arranged on a contiuum to measure one attribute

- Passing an item implies greater possession of attribute

- Items differ in level of difficulty but measure same attribute

A- slope (more steep the better)

B- item difficulty- where it lies on theta

C- pseudo guessing parameter (random prob of guessing right answer)

1 parameter logistic model- A

2 Pl- A+B

3 Pl- A+B+C (rarely used)

CMT- st error is constant across scale

IRT- st error of measurment differs across scores but generalizes across populations

CMT- longer is more reliable

IRT- not necessarily 

CMT- meaningful scale scores are obtained by comparisons of position in score distribution (distance from mean)

IRT- meaningful scale scores are obtained by comaprisons of distances from various items 

CMT- comparing test scores across multiple forms depends on parallelism and adequate equating

IRT- comparings cores from multiple forms is optimal when test difficulty levels vary across persons