Shared Flashcard Set

Details

ORG7020 Psychometric Theory Final
ORG7020 Final Study Guide
286
Psychology
Graduate
05/14/2011
Click here to study/print these flashcards.

Create your own flash cards! Sign up here.

Additional Psychology Flashcards

 

 

Cards

Term
Short answer tests contain ____ that MC tests do not.
 Definition
measurement error
Term
Effects of guessing are usually _____.
 Definition
underestimated BC can usually eliminate at least one distractor
Term
Short answer tests _____ more reliable than MC tests.
 Definition
may or may not
Term
Reliabilities are influenced by _____.
 Definition
Inter-item correlations
Term
Increase reliability by ____ not ____.
 Definition
increasing # items; NOT by increasing alternative responses
Term
With power tests, correlation between ___ and __ approaches 1.0, even if two means are different.
 Definition
correlation between scores obtained with the time limit; hypothetical scores
Term
A test is considered a power test when ____.
 Definition
scores obtained with and without time limits are highly correlated
Term
Is it ever advised to replace a power test with a speed test?
 Definition
Only if underlying trait obviously involves speed
Term
Reliability and construction of power tests relies heavily on ______.
 Definition
the sizes and patterns of correlations among items (has to do with internal structure of speed tests)
Term
A time limit affects ____, _____, and _____.
 Definition
1)pattern of correlations among items
2)pattern of item-total correlations
3)average correlations
Term
What are 2 problems associated with instructing people not to guess?
 Definition
1)difficult to frame instructions clearly
2)effects of the instructions vary over students and create an irrelevant source of individual differences
Term
How does guessing affect parameters?
 Definition
1)causes estimated mean score to be larger than it would have been if left blank
2)people who know the least gain the most from guessing
--> variance of errors due to guessing adds to the measurement error from the previously considered sources
3)makes scores less reliable for high ability
Term
What is the blind guessing model?
 Definition
Abbott's formula
Assumes the probabilities of correctly guessing on different items are independent
1/k = probability of guessing correct
Q = 1-p = (L-1)/K = probability of guessing incorrect
R-W/(k-1) = score a subject receives on a MC test - # wrong responses /(# alternative responses - 1) = Abbott's formula = Correction for guessing
Term
What is response bias?
 Definition
Measurement artifact that emerges from specific situations
Term
What is response style?
 Definition
Characteristic of an individual that is consistent across situations
Term
What is reverse regression?
 Definition
Artifact of the regression model in which the same data may simultaneously appear to demonstrate that of a focal group is both underpaid and underqualified
Term
WRT the impact a test has, which approach is oriented toward individuals? groups?
 Definition
Linear or moderated multiple regression
Quoteas
Term
Speed tests have ___ which ____.
 Definition
time limits
Maximize the time differences and therefore reliability
Term
What is and is not appropriate with speed test methods?
 Definition
Appropriate: Test-retest
NOT appropriate: Coefficient alpha
Term
What is a power test?
 Definition
a test in which the presence of a time limit does NOT contribute to individual differences
Term
What is Abbott's formula?
 Definition
correction for guessing
assumes subjects either:
(a) know the correct answer or
(b) guess blindly
Term
guessing lowers reliability BC ____.
 Definition
two individuals with the same underlying knowledge may get different scroes BC of differences in luck
Term
Oblique rotation in component solution will be more nearly ____ than oblique rotation produced by Common Factor solution BC _______.
 Definition
orthogonal;
Unique error that is part of the component structure will attenuate correlation
Term
Common factor solutions will be (more/less) biased, so the mean value will be closer to ___.
 Definition
less; 0
Term
Component solution will ___ actutal magnitude of correlations BC ___.
 Definition
overestimate
bias is present
Term
In Principle Components, residuals ___ when number of factors _____.
 Definition
decrease; increase
Term
Absolute magnitude will always ___ in Component solution.
 Definition
decrease
Term
With Common Factor solution, magnitude will ____.
 Definition
sometimes increase
this implies too many factors have been extracted
Term
Residuals will be ____ in absolute magnitude in the Common factor solution for a given number of components.
 Definition
smaller
Term
Component Eigenvalues will ____ Common Factor Eigenvalues BC ____.
 Definition
exceed
Accounts for more error
Term
Common factors will fit the model ___ than a component solution BC ____.
 Definition
better
less error
Term
Promax seeks to ____.
 Definition
maximize spread of variance of pattern elements on a factor
starts with orthogonal structure and then determines an ideal pattern having greater spread than the orthogonal structure
Term
What does Quartamax do?
 Definition
Cleans up items
Useful when wish to stress a general factor with which all variables correlate
Term
What does it mean when pattern and structure matrices are similar?
 Definition
the model is stronger
Term
What is a varimax rotation?
 Definition
Designed to eliminate general factors
Captures meaning of simple structure within the confines of an orthogonal framework
Term
If correlations are low in a factor correlation, what type of rotation should you use?
 Definition
Orthogonal
Term
Pattern elements ____. They are always ____ and may be ____.
 Definition
-Describe one variable per unit changes in a factor, holding all other variables constant
-They are always regression weights
-They may be standardized into beta weights
Term
h2 of ____ is the sum of the squared structure AND pattern elements.
 Definition
oblique
Term
h2 of ____ is the sum of the squared structure OR pattern elements.
 Definition
orthogonal
Term
In an orthogonal solution, the S matrix is ____ the pattern matrix.
 Definition
equal to
Term
The more _____ factors are, the more different structure and pattern matrices become.
 Definition
different
However, in orthogonal solution, the correlation matrix plays NO role BC assumes items are not correlated.
Term
All relevant properties are included in the ___ matrix.
 Definition
structure
Term
Oblique factors generally represent ___ variables better than orthogonal factors.
 Definition
slient
Term
In a nutshell, what is the difference between varimax, quartamax, and promax?
 Definition
Varimax: cleans up constructs
Quartamax: Cleans up items
Promax: Starts orthogonal, then goes oblique
Term
What is the difference between pattern and structure matrices?
 Definition
Pattern: relationship b/n item and components,HOLDING THE ITEMS CONSTANT
--> Contains correlations b/n the variables (rows) and factors (columns)

Structure: Relationship of the items and components, IGNORING THE OTHER ITEMS
--> Contains the regression weights used to predict the variables (rows) from the factors (columns)
Term
What will a proper factor rotation do?
 Definition
Strengthen the relationship b/n variables and factors
Factors will better represent variables that belong to it, rather than those that do not belong
Concentrates the variation shared by 2 variables that correlate highly on a single factor, rather than on several factors
Term
Will structure or pattern elements be higher? Why?
 Definition
Structure
BC it includes unique variance
Term
Define iterations.
 Definition
How many times you have to work the model to get a solution.
Term
What does the factor correlation matrix contain?
 Definition
Correlations among factors.
Term
Which guessing model states that you have a higher probability of guessing correctly on a second try?
 Definition
Blind guessing model
Term
What is the mean and standard deviation of K-alternative forced choice tasks WRT incorrect alternatives? How about mean value correct alternatives?
 Definition
Set at 0, SD of 1
Set at d', SD of 1
Term
True/False: Imposing a time limit makes a measure a speed test.
 Definition
False.
Term
True/False. Test items for a speed test are of trivial difficulty.
 Definition
True. (example = addition)
Term
With the sophistocated guessing model, ___ is a sufficient estimator of d'.
 Definition
# correct responses obtained when subjects respond to all trials
Term
Factor analysis can be used to determine:
 Definition
1) groupings or clusterings of variables
2) which variables belong to which group and how strongly they belong
3) how many dimensions are needed to explain the relaitons among the variables
4) a frame of reference (coordinate axis) to describe the relations among the variables more conveniently
5) scores of individuals on such groupings
Term
What is an "effect indicators"?
 Definition
Observable variables that are regarded as outcomes of an underlying latent variable
e.g. Common factor analysis
Term
What are "components"?
 Definition
variables are simply transformed to other variables for convenience
Term
What are "Causal indicators"?
 Definition
latent variables that are regarded as the outcomeof the observables
Term
What is the key to successful factor analysis?
 Definition
1) careful choice of variables
2) selection of subjects to ensure that all variables of interest correlated highly iwth other variables
Term
Factor analysis is a general method of ____.
 Definition
Decomposing the variance of a measure into 1+ common factors reflecting what variables share + additional unique factors which normally describe variance in a mesuare that CANNOT be shared among other individuals
Term
In Factor Analysis, variables are expressed as ___.
 Definition
weighted linear combinations of factors where the weightings are termed "pattern elements"
Term
How does the component model differ from the commmon factor model?
 Definition
It IGNORES unique factors
Term
Unique variance is broken down into (2 parts):
 Definition
1) Measurement error (unreliability)
2) Specific variance which is systematic but not shared with other variables in the analysis
Term
What is the difference between a general and group factor?
 Definition
General factors relate to all variables
Group factors relate to some variables
Term
What are the 2 stages of factor analysis?
 Definition
1) the direct solution condenses the variance shared among the variables & defines the # of factors
2) Second stage of rotation makes final result more interpretable
Term
Direct solutions are nearly always _____ (correlated/uncorrelated).
 Definition
Uncorrelated
Term
What are the 3 approaches to condensation?
 Definition
1) Defining a factor's content in advance (e.g. as the sum of variables in the analysis --> Centroid analysis)
2) Maximizing a property of the sample (e.g. by accounting for the most possible variance --> Principle component and Principle axis analysis)
3) Estimating population parameters (e.g. choosing the most probable outcome given the data ML analysis)
Term
What is the general rule about Eigenvalues WRT principle components?
 Definition
Eigenvalues greater that 1.0 are misleadding
Term
What do factors reflect?
 Definition
Combinations of observable variables (aka measures, tests, indicators, observables)
Term
List 3 ways linear combinations are used in factor analysis.
 Definition
1) Effect indicators
2) Components
3) Causal indicators
Term
What are "effect indicators"?
 Definition
Linear combinations in which the observables are the results (effects, outcomes) of the factor

-Observables = DV --> Contain error
-Factor = IV --> Error free
Term
What are "components"?
 Definition
Simple linear combinations of observables (therefore are observables in their own right)
Knowing one pair implies knowing the other through a simple transformation (if only one term in a given pair is unknown, theother pair is indeterminate)
Term
What are "causal indicators"?
 Definition
Linear combinations in which the factor depends upon the observables
--> factor becomes the criterion in a regression analysis sense
-Identifies error soley with the factor, however BC the observables can be observables, observables may also contain error
(e.g. people have high socioeconomic status BC they are wealthy and well-educated; they do not become wealthy or well-educated BC they are of high socioeconomic status
Term
Exploratory factor analysis defines factors in terms of ____.
 Definition
Best fit
--> "Most variance accounted for"

It is data-driven
Term
A rotated factor is a ____.
 Definition
linear combination of the initial factors
Only divide up variance more EQUALLY --> They will explain EXACTLY THE SAME total variance as the initial factors, even though the variabhles will related to the rotated factors differently than relate to the initial factors
Term
Factors are defined directly in ____.
 Definition
confirmatory factor analysis
Term
What is one similarity and one difference b/n multiple correlation and factor analysis?
 Definition
They relate a linear combination of variables to a criterion

Different: in MR, the predictors and criterion are separate entities; in FA, the predictors (factors) are at least partially defined by the criteria (variables)
Term
If a subtest correlates substantially with other measures given the same or similar names, it should possess ____ validity.
 Definition
Convergent validity
Term
What is MOST important for FA?
 Definition
Selection of variables!
Term
What are criteria for defining variables?
 Definition
1) More variables in a set that a given variable correlates with and hte higher the general level of correlation, the better
--> Higher the level of intercorrelation, the easier it is to determine patterning of correlations

2) Variables should be reliable
--> problem of attenuation ue to unreliability BC affects measures of relationship like r

3) Analysis should contain variables with known properties called marker variables

4) Large sample sizes should be used to ensure that groupings are not simply effects of sampling error
Term
All the variance is considered ____ in component models. They estimate the unique variance to be ___ for every variable.
 Definition
Systematic
.0
Term
___ is highly desirable in component analysis.
 Definition
Intercorrelations
Term
Weightings are called ____, which may be viewed as ____.
 Definition
pattern (b) elements; may be viewe as regression weights
Term
Each variable has ___ in factor analysis.
 Definition
its own unique factor
Term
What is the average communality?
 Definition
Proportion of variance accounted for
Term
What is a limitation to communality?
 Definition
The communality of each variable and therefore the proportion of variance accounted for generally increases as the number of factors increases for the same reason that all multiple correlations are biased with respect to number of predictors (EVEN WHEN THE ADDITIONAL FACTORS ARE MEANINGLESS)
Term
What is the multiple correlation for components and common factors?
 Definition
Components: 1.0 BC components are linear combinations of the other variables

Common factors: LESS THAN 1.0 BC they are broader than the variables that define them

**Multiple Correlations reflect the proportions of variance in the factors that are explained by the variables
Term
True/False. Adding a variable that is highly correlated with other variables in a grouping does NOT add appreciably to the factor definition.
 Definition
True.
Term
What should you do if the factors have low multiple correlations?
 Definition
Add additional variables.
Term
What are 2 things that are UNAFFECTED by rotation?
 Definition
1) Individual h2 values
2) Overall indices of fit (e.g. proportion of variance accounted for)
Term
Name 4 general effects of alternative ways of defining data.
 Definition
1) Standardizing measures eliminates the effects of differences in the unit of measurement
2) Expressing the measures as deviation scores eliminates the effects of differences in location of the variables (means) from the analysis but allows differences in variance to play a role
3) Expressing the measures as raw scores allows difference in both location and variance to affect the results
4) Dividing raw scores by the variance of the particular variable over subjects would eliminate differences in variance but allow differences in location to remain
Term
Pattern weights are ___.
 Definition
Beta weights
Term
Initial solutions in factor analysis are nearly always ___.
 Definition
Orthogonal
Term
If deviation scores are used in the analysis INSTEAD of standardized scores, _______. If raw scores are used, then _____.
 Definition
covariances are estimated instead of correlations

mean sums of products are estimated
Term
What is contained in the diagonal elements of R?
 Definition
communality estimates

Component analysis: 1.0
Common factor analysis: < 1.0
Term
True/False. Communality estimates ARE NOT THE SAME as communalities.
 Definition
True.
Communality estimates: rij in R
Communalities: h2
Term
What does the beta weight for a predictor equal?
 Definition
its correlation with the criterion (validity) when the predictors are UNCORRELATED
Term
The number of factors to be retained is suggested by ____.
 Definition
the first set of structure elements
--> if highly correlated with 1st factor, may not need other factors
--> if moderately correlated, may need several factors
--> if correlations are near 0, may not be any factors in the data
Term
What is the method of successive extraction used for?
 Definition
the obtaining the uncorrelated (orthogonal) factors of exploratory factor analysis
Term
What is the method of extracting simultaneous factors?
 Definition
>1 factor may be extracted at each step, but the factors extracted at that step will usually be correlated

May be orthogonal or oblique

Characteristic of confirmatory approaches
Term
What is a vector?
 Definition
1) A line segment having both:
(a) direction (orientation) and
(b) length (magnitude) in geometry

2) A set of numbers, such as test scores, in algebra
Term
The more different the vectors, the ___ the angle and the ___ the cosine.
 Definition
larger (up to 90o) angle
smaller the cosine
Term
Two unrelated vectors form ___ angle. Their cosine is ___.
 Definition
90o
0
Term
The cosine of the angle between the two vectors viewed geometrically is totally equivalent to their ______ if they are defined algebraically as 2 sets of numbers.
 Definition
correlation
Term
The method of successive extraction is the basis for obtaining the ______.
 Definition
Uncorrelated (orthogonal) factors of exploratory factor analysis.
Term
Angles between 90o and 180o have ___ and therefore ____.
 Definition
negative cosines
negative correlations
Term
How do you calculate the length of the common factor?
 Definition
Square root of its communality
Term
When vectors are all of unit length, _____. When they are different lengths less than 1 they are _____.
 Definition
Component analysis

Common factor analysis
Term
If there are more than two factors, the vectors (would/would not) have unit lenght because they would also project into other dimensions.
 Definition
Would not
Term
What is a common goal of factor analysis?
 Definition
To find the minimum rank of a matrix of correlations by a suitable choice of diagnoal elements (communality estimates)
Term
If the factor model fit perfectly, it would partition the total variance of any variable (1.0 in standard form) into 3 terms: __, __, and ___.
 Definition
Variance due to measurement error +
Specific variance +
Common variance
Term
What is the measurement error of variable Xi?
 Definition
Squared standard error of measurement from reliability theory
= 1 - coefficient alpha
Term
If the reliability of variable X1 is .80, then the measurement error variance would be ___ and the systematic variance is ___.
 Definition
.20
.80
Term
What is systematic variance?
 Definition
Systematic variance is the sum of common variance and specific variance
Term
What is specific variance?
 Definition
Specific variance is nonrandom but CANNOT be explaind by relationships with other varaibles in the model
Term
What is unique variance?
What is a standardized variable's unique variance?
 Definition
h2 + u2
Specific variance + error variance

1-h2
Term
What is the difference b/n Factor analysis and Component analysis WRT Components of variance?
 Definition
Factor: partitioning variables into common and unique variance

Component analysis: explains common variance, with linear combinations of the variables
--> Unique variance becomes residual not explained by obtained factors
Term
Salients are generally referred to as ____.
 Definition
Magnitude
Term
What are the 7 types of factors?
 Definition
1) General: all measures are salients
2) Group: Some, but not all are salients
3) Common: General and Group are called common factors BC what they measure is common to more than one variable
4) Unipolar: when all salients have the same sign
5) Bipolar: Some salients are positive, some are negative
6) Singlet: Factor with only one salient (e.g. masculinity/feminity on MMPI)
7) Null: No salients
Term
Name 3 general approaches to Condensing Variance in Exploratory Factor Analysis.
 Definition
1) Factors are defined before analyzing the data.
--> Centroids: when a factor (usually a component) is the equally weighted sum of the variables
2) Optimize some property of the sample data.
--> PrC maximizes the amt of variance that can possibly be explained

3) Use sample data to predict the results in a population
--> Maximum likelihood EFA stresses statistical inference, rather than assuming an indefinitely large sample
Term
What are 4 ways you use a correlation matrix WRT factoring?
 Definition
1) If correlations are high enough to warrant factoring
2) Common groupings in the data
3) Signs and sizes within groupings
--> Sizes of the correlations defines how strongly the factor (grouping) is defined
4) Correlation b/n groupings to decide about the type of rotation to use
--> If low (<.3), use orthogonal solution, however, if not, use oblique
4)
Term
Pattern elements are to ____ as structure elements are to ____.
 Definition
Regression weights

Correlations
Term
h2 in orthogonal solution equals ___.
 Definition
sum of squared structure (or pattern) elements
Term
Are absolute residual correlations bigger in a Component Solution or a Common Factor Solution?
 Definition
Component SOlution
Term
In PrC, what are Eigenvalues?
 Definition
Proportion of variance accounted for by each PrC
Term
PrC must account for _______ than the number of centroids.
 Definition
more variance
Term
Each ___ defines the total variance explained by the PrC.
 Definition
Eigenvalue
Term
All component Eigenvalues are either ___ or ____ because they are interpretable as variances.
 Definition
0 or positive
Term
The number of ___ represents the # of PrC factors neeed to explain all the variance in a correlation matrix.
 Definition
positive nonzero eigenvalues
Term
The sum of the diagonal elements in R are called the ____. This equals ___.
 Definition
trace
the sum of the eigenvalues
Term
The product of all eigenvalues in R equals its ___.
 Definition
Determinant
used in FA as a multivariate measure of variance
Term
When each variable in the population is perfectly correlated with every other variable, each element in R is ___.
 Definition
1.0
Term
Measurements are reliable to the extent that they are ____.
 Definition
repeatable
Term
Any random influence which causes different measurements of the same variable to vary is a source of ____.
 Definition
measurement error
Term
A long test with a _____ among items is ALWAYS a highly reliable test.
 Definition
positive average correlation
Term
Correction for Attenuation
 Definition
Used to examine the correlation between two variables as the reliability of each is changed to a designated level and not simply made perfect as was previously assumed
Term
Increase test reliability by using:
 Definition
1) more items
2) using Spearman-Brown prophecy formula
Term
What does variation within a test do?
 Definition
lowers average correlation among items, but average correlation is still sufficient to estimate reliability
Term
What are 9 sources of variation in a test?
 Definition
1. Item sampling: Individuals have probability of correctly answering each item, depending on their a) true score & b) difficulty of the item
2. More items  less error
3. Error due to sampling is predicable from average correlation
4. Coefficient alpha is appropriate measure of relaitbility for any type of item
5. Guessing: lowers correlations b/n items and overall test reliability
6. Accidentally marking one answer instead of another
7. Misreading a question, due to confusing wording
8. Fatigue on long tests
9. Random (not systematic) grader errors
Term
What is item sampling?
 Definition
Individuals have probability of correctly answering each item, depending on their
a) true score &
b) difficulty of the item
Term
Three sources of error that cause domain-sampling model to overestimate actual correlation b/n forms:
 Definition
1. Systematic differences in content of the 2 tests
a. Items composed, not randomly sampled
2. Systematic Effects: from subjectivity of scoring, due to different standards among judges
3. Change in the subject in the attribute being measured
a. More important with mood-related measures than ability mreasures
Term
Alternative form correlations may be higher than ____.
 Definition
within-test estimates of reliability
Term
Defined Internal Consistency.
 Definition
Estimates of reliability based on the average correlation among items within a test
Term
What is Coefficient alpha?
 Definition
represents both number of items and their average correlation
1. Usually provides good estimate of reliability BC sampling of content is usually major sources of measurement error for static constructs
2. Sets upper limit for reliability of tests constructed in terms of domain-sampling model based on observed correlations
a. If a is low: testis too short or items do not have much in common
b. Measurement problem  choose different items
Term
Alternative Forms: If correlation b/n AF is markedly lower than coefficient a (e.g. .20 or more), measurement error is present (due to 3 sources of error):
 Definition
1) systematic differences in content,
2) subjectivity of scoring,
3) variation in trait over time
Term
i. If average correltion within 2 test formats is substantial (e.g. .20), but average cross correlation b/n items on two forms is low (e.g. .10), then _____.
 Definition
tests reliably differ in content and thus measure different traits
Term
If correlation b/n tests over a 2-wk interval is less than correlation for tests taken on same day, ____.
 Definition
scoring is probably reliable but trait is temporally unstable (desired)
Term
What does unreliability of scoring mean?
 Definition
trait does not exist in a manner consistent over judges;
Raters are inconsistent
Term
WRT Alternative forms, Coefficient a is good estimate of reliability on measures that ____.
 Definition
domain of content is easily specified and ppl are stable over time (e.g. aptitude and achievement tests)
Term
If alternative form cannot be constructed, then ___.
 Definition
Domain of content cannot be defined;
Cannot accurately communicate what is being measured
Term
What are 4 other estimates of reliability?
 Definition
i. Coefficient alpha
ii. Correlations
iii. Split-half approach
iv. Test-Retest
Term
What is the split-half approach WRT reliability? What is a problem with this approach?
 Definition
test items divided in half; scores on both are correlated
Misleading estimates when items ordered in terms of difficulty
Term
What is Test-Retest WRT reliability? What are the problems of this approach?
 Definition
Same people are retested by the same test after a period of time
- Used instead of alternative-from method to determine reliability
- Problems: (lead to spuriously high correlations b/n tests) BC:
a. Remember answers
b. Repeated work habits
c. Similar guesses
d. Only partly dependent on inter-item correlations  reliability of test is function of average correlation among items
--> Makes it possible to have a measure with no internal consistency stable over time (High retest correlation with low internal consistency)
Term
____ reduces validity of decisions when temporally unstable measure is used to make practical decisions about people and work.
 Definition
Measurement error
Term
Two types of reliability coefficients to be computed and reported:
 Definition
1) Coefficient a: all forms of a test
2) Correlations among alternative forms (reveal sources of measurement error not detectable by coefficient
2)
Term
What are 3 uses of the reliability coefficient?
 Definition
i. Correction for attenuation
ii. Confidence Intervals
iii. Effect of Dispersion on Reliability
Term
WRT Correction for attenuation, ii. If relevant measures are modestly reliable, observed correlations will ___ correlations among traits.
 Definition
UNDERESTIMATE
Term
Correction for attenuation formula:
 Definition
r’xy = rxy / (√(r’ xxr’yy) / (√r xxryy)
r’xy = estimated correlation b/n variables x and y if their reliabilities are changed
r’xx = changed reliability for variable x
rxx = obtained reliability for variable x
Term
WRT Corrections for Attenuation, the Reliability coefficient:
 Definition
Estimates extent to which obtained correlations b/n variables are attenuated by measurement error
Term
Corrected correlations are seldom dramatically different from observed correlations, therefore (3 things):
 Definition
1. Low correlations have nothing to do with reliability
2. Increase # items to make a test more reliable
3. Tests usually correlate poorly BC measure different things NOT due to measurement error
Term
What is the formula for confidence intervals for obtained score?
 Definition
Omeas = ox √(1-rxx)
Term
CI should be obtained around a person’s ______ not their _____.
 Definition
true score, not their test scores
Term
WRT CIs, high obtained scores are biased ___, low obtained scores biased _____.
 Definition
upward; downward
Term
What are Unbiased scores?
 Definition
Average scores people would obtain if they were administered all possible tests with a aonstand # of items from a domain
T’ = rxxx
X = deviation score
Rxx = reliability
T’ = estimated true scores
Term
CI should be centered around the ___.
 Definition
estimated (regressed) observed score
Term
WRT CIs, Less error in estimating ____ from an ___ than the converse.
 Definition
true scores; observed score
Term
_____ may be used to measure change over time.
 Definition
Estimated true scores
Term
Standard error of measurement =
 Definition
-estimated standard deviation of obtained scores if any individual is given a large # of tests from a domain

-stable across populations which differ in variability BC the changes in RC and SD are partially offsetting
Term
Reliability coefficient is directly related to _____ for any sample.
 Definition
SD of obtained scores
Term
Reliability coefficient will be ___ in more variable samples.
 Definition
larger
Term
Reliability coefficient
 Definition
ratio of true-score variance to obtained-score variance
Term
5 ways to reduce measurement error
 Definition
1. Writing items clearly
2. Making test instructions easily understood
3. Adhering closely to the prescribed conditions for administering an instrument
4. Making subjective scoring rules as explicit as possible
5. Training raters to do their jobs
Term
If # items know, use ______ to estimate how much reliability will increase if the # of items were increased by any factor k :
 Definition
Spearman-Brown prophecy

Rkk = kr11 / (1+k-1)r11
k = # items on shorter test / # items on longer test
rkk = estimated reliability of the shortened test
r11 = reliability of longer test
Term
Reliability approaches ____ as test length increases, so long as average correlation of items in a domain is _____.
 Definition
1.0 ; positive
Term
What formula would you use to determine how many times you have to increase the length of a test?
 Definition
K = rkk(1-r11) / r11 (1-rkk)
Rkk = desired reliability
R11 = reliability of existing test
K = number of times test would have to be lengthened to obtain a reliability of rkk
Term
If average correlation among items in a domain is _____, the correlations b/n samples of items will be ____ and # of items needed to achieve acceptable reliability will be ____.
 Definition
very low; small; very large
Term
If sig correlations are found, ____ will estimate how much the correlations will increase when reliabilities of measures are increased.
 Definition
corrections for attenuation
Term
4 Limitations on the Reliability Coefficient’s Utility
 Definition
i. Relibility estimates based upon observed correlations  affected by similarities of the item distributions
ii. Look at item’s distributional properties (e.g. p values) and correlation with total test score to avoid eliminating items spuriously
iii. Hetergeneities in item distributions  undersestimate worth of an item
iv. Coefficient a = lower bound of population reliability (true variance: total variance)
Term
If tests had an acceptable reliability, but they were mutually uncorrelated, Coefficient a would be ___.
 Definition
0; however linear combination NOT necessarily 0
Term
Reliability of variable in linear combination =
 Definition
true-score variance/obtained variance

True-score variance = Obtained score variance x reliability
Term
What do negative elements in a linear combination effect? What do they not effect?
 Definition
ii. Does not affect logic BC still 3 items
iii. DOES affect Denominator (denominator is variance of linear combination)  Minus sign reverses signs of correlations
iv. IF S3 correlated negatively with X1 and X2 minus sign in the linear combination would increase denominator over what it would have been had it been added
Term
Larger variance of linear combination -->
 Definition
greater reliability
Term
WRT linear combinations, Reliability of a sum (CAN/CANNOT) be estimated by coefficient alpha.
 Definition
CANNOT
Term
Within an ANOVA design:
1. True variance is estimated from:
2. Error variance is estimated from:
3. Ratio is identical to:
 Definition
1. Difference in MSs
2. MS within groups
3. Alpha
Term
What does Generalizability Theory allow you to do?
 Definition
To evaluate both random sampling error that arises within a domain and systematic error that might arise BC different judges evaluate different attributes
Term
Tests may be evaluated by standards of (3 things)
 Definition
1. Content
2. Construct
3. Predictive validity
Term
How are Predictive and construct validation similar?
 Definition
Both involve correlating measure with a criterion
Term
What is content validation?
 Definition
Begin with domain of content that defines what is to be measured (who the test is applicable, test plan that defines how it is to be measured)

Administer --> item-analysis to define each item’s difficulty; discrimination --> how highly each relates to total test score
Term
What is Construct Validation?
 Definition
• Begin with hypothesis that implies domain of content
Scales content should be homogeneous; difficulty BC methods used to infer trait are heterogeneous
Term
Difference b/n one-time use and repeated use tests:
 Definition
1. Greater potential legal scrutiny
2. Various non-psychometric considerations (greater test security)
3. Several cycles of refinement before use
Term
Achievement tests: created through
 Definition
content validation
Term
Ability tests: created through
 Definition
construct-validation
Term
Domain of Content and Test Plan should describe (6 things):
**Should be done BEFORE creating test
**Define an appropriate domain of content is ESSENTIAL
 Definition
1. Types of items to be employed with examples
2. Approximate # of items to be employed in each section and subsection
3. How long the test will take to administer
4. How it will be administered
5. How it will be scored
6. Types of norms or other referencing that will be obtained
Term
Term papers measure ___.
 Definition
measure divergent thought and creativity
Term
Construct good items and have clarity by (4 things):
 Definition
1. Phrasing of items
2. Relate to domain
3. Points the knowledgeable student toward what is demanded
4. Avoid trivial Qs
Term
Thorndike’s construction for ALL items:
 Definition
1. Make complexity of items appropriate to students’ level
2. Define task & directions as clearly as possible
3. Inform students about grading standards
4. Write itmes simply & straightforward
5. Know mental processes to be used
6. Use novel material or organization to prevent reproduction of lectures
7. Vary complexity and difficulty of items  improves ability to discriminate
8. Make questions Independent
9. Avoid negatively phrased items
10. Never use double negatives
Term
What are 6 recommendations for short answer tests?
 Definition
1. Omit only key words
2. Do NOT leave too many blanks
3. Put blanks near end of Q
4. Avoid specific determiners such as “all” and “none”
5. Avoide ambiguous determiners such as “frequently” and “sometimes”
6. Have each item express a single ideal
Term
What are 10 recommendations for MC items?
 Definition
1. Be sure stem clearly formulates problem
2. Include as must of item’s content in stem as possible
3. Include only necessary material
4. Use novel material and examples
5. Be sure distracters are plausible
6. Use “none of the above” or “all of the above” sparingly
7. Make alternatives of approximately equal length & parallel grammatical construction
8. Randomize location of correct alternative
9. Make sure each alternative agrees with stem
10. Try to eliminate any factor that makes the correct alternative stand out
11. Formulate incorrect alternatives so that they detect common ways in which students may be misinformed
Term
If you have low coefficient alpha, what are 3 things you can do?
 Definition
a. Construct more items
b. Get responses from larger group of subjects
c. Perform complete item analysis
Term
What does the Spearman-Brown prophecy indicate?
 Definition
indicates adding items to increase reliability obeys a law of diminishing returns; difficult to improve reliability of a test that is already reliable substantially by adding more items
Term
WRT Test length, what are 3 things to consider?
 Definition
1. Reflect time available
2. Desired reliability
3. More variable the population, smaller # of items needed to achieve a given reliability
Term
What are 3 things to consider WRT pilot sample of subjects?
 Definition
1. Pilot sample needs to be similar to target population
2. Conditions of pilot study should resemble eventual use
3. Rule: at least 200 normative subjects
Term
WRT item analysis, content validity relies on _____ grounds.
 Definition
on rational, not empirical
Term
What is IA?
 Definition
statistical data regarding how subjects responded to each item and how each item relates to overall performance
Term
How is IA useful?
 Definition
a. Be suspicious of items if distracter is chosen more often than correct alternative  instructions or item are misleading
b. Distracters hardly ever chose are too transparently incorrect
c. Proportion choosing correct alternative or item p value is the classical index of item difficulty (can also apply to sentiments)
i. Items with extreme p values should be excluded BC do NOT discriminate among individuals
Term
What is the classical index of item difficulty?
 Definition
p-value
Items with extreme p-values should be excluded BC do NOT discriminate among individuals
Term
Driver’s test: example of test designed for _____.
 Definition
mastery learning
Term
Tests designed for mastery learning have ______ when instruction has the desired effect
 Definition
low internal consistency
Term
General achievement tests: _____ BC difficult to teach enough in a short period of time.

Classroom tests: _____.
 Definition
temporally stable

temporally unstable
Term
Content-validated tests: (DO/DO NOT) need to correlate with any other measure nor have very high internal consistency.
 Definition
DO NOT
Term
Item analysis must:
 Definition
1. Describe how each item relates to overall test performance
2. Provide discrimination among discrimination indices
Term
Best items are most ___.
 Definition
discriminating
Term
Less ambiguous, NOT extremely difficult make individual scores on final test ___.
 Definition
more reliable
Term
Item response theory uses ___ and correlates items with this estimate.
 Definition
a estimate of trait magnitude (theta θ) 
Term
What are discrimination indices (3 things)?
 Definition
a. Covariance b/n an item and total score
b. Average correlation b/n a given item and all other items
c. Proportion of people passing the item in the top half of the class – the proportion of people passing the item in the bottom half of the class
Term
WRT IA, negative items may mean (3 things):
 Definition
1. Bad wording
2. Sampling error
3. Miskeying
Term
Content validation uses ___ as a final decision-making about whether to keep an item.
 Definition
Use human judgment as final decision to reject or include an item
Term
Construct and predictive validity use __ to make final decision about whether to keep an item.
 Definition
Item analyses
Term
WRT item selection, ___ is the primary criterion.
 Definition
Discrimination index (E.g. corrected item total r)

-a. Items with high item-total r values have more variance relating to what the items have in common & add more to test’s reliability than low values
Term
____ are used as a secondary criterion for item inclusion.
 Definition
P values

-Corrected and uncorrected item-total r values are biased toward items with intermediate p values
Term
If correlation is low, but intended reliability is high, what can you do?
 Definition
-Add sets of 5 – 10 items until desired reliability is obtained
-Key to successful item selection is re-administration of test in a new sample
Term
# items needed to be added depends on:
 Definition
1. item-total r values
2. the reliability of the 1st set of items
Term
WRT adding items, Obtained reliabilities are ____ than predicted reliabilities.
 Definition
lower
Term
Why may items fail?
 Definition
a. Items may be from a poorly defined domain where correlations among items are uniformly low
i. Reliability grows slowly as the # of items increases
b. Items may be factorially complex (multidimensional) so that clusters of items have relatively ihg correlations with one another, but low correlations with members of other clusers
i. Range of inter-item correlations will be large
c. Some items may have high correlations with one another, but other items may have near zero correlation with all other items  adding items will do no good in this case
i. Detect by noticing marked decrease in item-total correlations at some point
Term
Most likely to find a ____ of items in the initial set that correlate well with total score, _____ that do not correlate at all, _____ in the middle.
 Definition
small to modest #
large number
moderate block
Term
What are norms?
 Definition
Statistical data that provide a frame of reference to interpret an individual’s scores relative to scores of others

1. Less essential when measure is intended for use in group research, rather than individual decisions
2. Expressed in percentiles (percentage of persons in the normative sample at or below a particular score) and z-scores
3. MORE meaningful to think of grades as reflecting JUDGMENTS based on the instructor’s conception of the various categories
4. Scores CRITERION-referenced when implications for relevant bxs (e.g. score of 50 on admission test  .75 chance of completing the program)
5. Scores DOMAIN-referenced when relate to the domain being measured
Term
True/False. Measures designed through content validation need NOT correlate with any external criterion to be valid
 Definition
True.
Term
Tests developed for construct validity CANNOT be developed without a _____.
 Definition
theory that dictates the properties of that measure
Term
Personality traits involve _____.
Abilities tests involve ____.
 Definition
sentiments
judgments
Term
Measurement properties for content homogeneity should include:
 Definition
1. Coefficient a reliability
2. Temporal stability
3. Homogeneity of content: content is homogeneous when it has little measurement error (high coefficient a) and measures ONLY one attribute
a. Implies measures are unidimensional and unifactorial
Term
How do you know when content is homogeneous?
 Definition
When it has little measurement error (high coefficient a) and measures ONLY one attribute

-Implies measures are unidimensional and unifactorial
Term
Heterogeneous Content if ___.
 Definition
if average correlation among items and average item-total correlation is low
Term
WRT content validation, you want:
 Definition
1. Average correlation with total scores is high
2. Spread of correlations about this average is small
Term
WRT Factor Analysis, correlations and variance among _____ items are usually higher than _____.
 Definition
multicategory (e.g. Likert)
dichotomous items
Term
Mean and standard deviation for multicategory items provides info about ____ and ______.
 Definition
Item difficulty for judgements
Endorsement level for sentiments
Term
______ is preferred discrimination index for content-validated tests.
 Definition
Corrected item-total correlation
Term
To improve item distributional characteristiscs(WRT IA and Selection):
 Definition
a. Use multicategory format (Likert scale)
b. Change modifier (eg. Dropping “very”)
Term
What are 2 problems iwth Empirical (criterion-oriented) approaches to test construction?
 Definition
i. Difficult to determine which items fall on which scales
ii. Items may correlate with the criterion because they correlate with each other
Term
A construct is less likely to require norms than a content-validated measure BC of ___.
 Definition
Differences in their probable used

--> Items derived from content validation involve ad hoc requirements that are of pragmatic, rather than theoretical import
Term
What is EFA?
 Definition
Interested in discovering factors by exploration
i. Defines factors in terms of “best fit”  “most variance accounted for”
ii. Step-wise: 1st reduce items into # of factors, next rotate
Term
Name and describe Three uses of Linear combinations:
 Definition
1. Effect Indicators: linear combinations in which the observables (DVs) are the results of the factor (IV)
a. observables (DVs) = error free
b. factor (IV) = contain error
2. Components: Simple linear combinations of observables and observables in their own right
a. IF one term is known, the other is indeterminate
3. Causal Indicators: Linear combinations in which the factor depends upon the observables
a. Factor becomes the criterion (SES & wealth)
b. Identifies error solely with the factor
c. Observables and predictors can contain error
Term
Common factor analysis uses which type of linear combination?
 Definition
EFFECT INDICATORS
Term
S model is equivilant to S curve when ___.
 Definition
you multiply it by 1.7
Term
WRT correlations and validity, when should it be high, and when should it be low?
 Definition
Correlation should be low if you are measuring two different things; high if you are measuring similar things
Term
What is the formula for Linear Combination of Variables?
 Definition
simple linear regression: y =b0 + b (x) e
Term
WRT linear combinations, what are components?
 Definition
Components : linear combinations of observables and therefore observables in their own right
Term
Can you ever know a true score?
 Definition
No, because it is a theoretical idea.
Term
WRT FA, if it is reliabile, then ____ and ____.
 Definition
-Reliabile --> DECREASE MAGNITUDE

- ATTENUATION: R XY = RXY / SQRT (R XX R YY )

- AS RELIABILTIY GOES DOWN, MORE ATTENUATION
Term
Numbers in the matrix are ___.
 Definition
factor loadings: correlation of item with unknown factor!!
Term
variance can be decomponsed into ___.
 Definition
Variance due to measurement error + specific variance + common variance
Term
What is uniqueness?
 Definition
Measurement error + Specific variances = u2
Term
What is common variance?
 Definition
h2 aka communality

- means variance that is shared across the factors (BC may share variance across different constructs)

- reflects correlation with factors
Term
Factor analysis is frequently described as partitioning variables into ___.
 Definition
common and unique varianace
Term
Principle Components is a __ procedure which ____ thereby _____.
 Definition
- Variance MAXIMIZING PROCEDURE
- Analyzes all the variance: h2 + u2
- MAXIMIZES EXPLAINED VARIANCE BC OF THE DIAGNOAL CORRELATION MATRIX (1.0 IS IN THE DIAGNOAL

- USES ALL THE VARIANCE);
Term
___________ maximizes the variance.
 Definition
Principle Components
Term
Variance of a standard score = __.
 Definition
1
Term
Communality is the sum of ______.
 Definition
sum of the squared factor loadings across the factors
Term
Communality = sum of
 Definition
sum of squared factor loadings across the factors
Term
There will ___eigenvalues to number of items.
 Definition
equal
Term
Principle axis analyzes ___.
 Definition
Only variance shared across the items
Term
Factor Transformation Matrix is a function of ___ and ___.
 Definition
sin and cosine
Term
Initial eigenvalues are based on ___
 Definition
unrotated matrix (initial item values) are used in extraction
Term
% of variance = __
 Definition
Total Extraction sums of squared loadings /# components
Term
Cumulative % = ___.
 Definition
% variance accounted for
Term
What are 3 approaches to condensation?
 Definition
o Centroid analysis: Defining a factor’s content in advance as sum of variables in analysis
o Principle Components & Principle axis Analysis: Maximizing a property of the sample data by accounting for the most possible variance
o Maximum likelihood Analysis: Estimating population parameters
Term
Eigenvalues that exceed 1.0 may be ___.
 Definition
misleading
Term
1- communality = ___.
 Definition
uniqueness
Term
What is a varimax rotation?
 Definition
Makes vectors interpretable by cleaning up factors/constructs

-->High/low on different components
Term
Varimax assumes the items are ___.
 Definition
UNCORRELATED
Term
Validation is an iterative process, therefore you should ____.
 Definition
re-validate on a separate sample
Term
Pattern matrix = rel of ___.
 Definition
item and component holding constant all other items

-like standardized regression coefficient BC partials out contribution of other variables
Term
Structure is like a ____.
 Definition
regular bivariate correlation.
Term
Varimax assumes factors are ___.

Oblique rotation: assumes factors are ____.
 Definition
Uncorrelated

Correlated
Term
Stability of a factor solution is better when ____.
 Definition
done across groups and industries
Term
You don't have convergence when _____. If this happens, you know ____.
 Definition
there are too many iterations
something is wrong with the model (e.g. bad items, small sample size, etc.)
Term
# variables you have in the variance/covariance matrix is given by what formula?
 Definition
P(p+1)/2
Term
How is Coefficeint alpha calculated?
 Definition
proporetion of true score/observed score

True score/error = observed score!!
Term
4 reasons for differences in intercepts
 Definition
• Possible reasons for differences in intercepts:
• bias in test
• Bias in criterion
• Reliability of test
• Omitted variables (when you omit a relevant 3rd or 4th variable  SPECIFICATION ERROR!!)
Term
Diffrence in validity coefficient (rxy) may be cited as evidence of ___.
 Definition
DIFFERENTIAL VALIDITY
Term
Slope bias means ___.
 Definition
DIFFERENTIAL VALIDITY  means the correlation of the selection tool and performance will vary across groups; if steeper slope, better --? Means validitiy foefficient is different between groups
Term
Intercept bias is heavily influenced by ____.
 Definition
mean
Term
What are sources of bias?
 Definition
1. Real mean differences in attributes
2. Differences are function of the test
3. Content of test is familiear to some groups, but not others
4. Method of Presentation (e.g. written v. video, etc.)
5. Interaction with administrator
Term
What are remedies for bias?
 Definition
1. Change content of test
2. Employ multiple methods of assessment
3. Change method
Term
What are 3 ethical positions WRT bias?
 Definition
o Unqualified Individualism: use tests to select MOST qualified individuals that can be found  Indifferent to race or gender of applicants
o Quotas: using non-psychometrically; explicitly recognize race and gender differences (e.g. if 20% African Americans in the population, then select 20% African Americans for company)
o Qualified individualism: compromise b/n unqualified individualism and quotas
Term
When is adverse impact legal?
 Definition
o 4/5s of selection ratio (20 whites hired/100 white applicants)
o IF there is an adverse impact, validation study may be necessary to prove the test is jjob-related; IF SO, ADVERSE IMPACT IS LEGAL
Term
What are times power tests designed to measure?
Who uses them?
 Definition
measure designed to assess power but administered with a time limit, normally imposed for administrative pruposes (e.g. classroom availability)
o THIS IS TYPICALLY WHAT IS USED AT UNIVERSITY – LEVEL (combination of speed and power, but MAINLY POWER)
Supporting users have an ad free experience!