Term
| What is a time series design? What does it look like? |
|
Definition
- A series of observations made on the same variable consecutively over time. -O O O O O O O O O O X O O O O O O O O O O O |
|
|
Term
| What are autocorrelations? |
|
Definition
- A characteristic of time-series designs, when the value of one observation is related to the value of previous observations that may be one, two, or three lags away. Estimating this ______ usually requires a large number of observations, typically 100, to facilitate correct model identification. - Statistically, one needs enough observations to identify the model, and this identification depends on the amount of error in the data, any periodicity effects, the timing of the intervention within the series, and the number of lags to be modeled. Sometimes fewer observations are used to show how an abbreviated time series can rule out many more threats to validity than is possible in cases in which there are only a few pretest and posttest time points. |
|
|
Term
| Threats to validity in a time series design |
|
Definition
Internal: History: Possibility that forces other than the treatment under investigation influenced the dependent variable (e.g., Detroit study of banning alcohol sales to pregnant women). Instrumentation: Change in admin. procedures can lead to a change in how records are kept (person who wants to make perf. look good will change bookkeeping procedures). Selection: If composition of experimental group changes abruptly at the time of the intervention (if treatment causes or even requires attrition from measurement framework; hence, effect might do to difference of Ps from the pre and posttreatment series). Statistical Conclusion Validity: Low power, violated test assumptions, and unreliability of measurement; this stuff weakens the causal logic of the design (for Detroit alcohol ban study, problem with lack of specificity at what point the intervention began and its diffusion through the units, pregnant women, exposed to the treatment). Construct Validity: Inadequate explication of constructs or confounding of constructs; when using archival records, consideration of observed results due to evaluation apprehension, demand characteristics, or some similar threat must be considered.
External: Disaggregation (investigating external validity via using available data on the background characteristics of units to stratify them into subgroups, say, males and females, or different age groups) can be problematic because if it is not done correctly it can lower statistical power (e.g., if last age category of archival research is "over 65" and one is interested in studying the so called "old old" (over 75), one cannot do so).
|
|
|
Term
| Interpreting the results of a time series design |
|
Definition
1. A Change in form (level or intercept) 2. A change in slope of the series at the point of interruption (the treatment). 3. Weak or delayed effects (law banning sale of alcoholic bevs to pregnant women took effect 7 months later). |
|
|
Term
| Common problems with time series designs |
|
Definition
1. Many treatments are implemented slowly and diffuse through a population, so that the treatment is better modeled as a gradually diffusing process rather than as occurring all at once. 2. Many effects occur with unpredictable time delays that differ among populations and over time. 3. Many data series are much shorter than the 100 observations recommended for statistical analysis. 4. Many archivists are difficult to locate or reluctant to release data. 5. Archived data might involve time intervals between each data point that are longer than one needs; some data may be missing or look suspicious; and there may be undocumented definitional shifts. |
|
|
Term
| Regression Discontinuity; what is it? |
|
Definition
- An experiment in which units are assigned to conditions based on exceeding a cutoff on an assignment variable. |
|
|
Term
|
Definition
Any measure taken prior to treatment, in which the units scoring on one side of the cutoff are assigned to one condition and those on the other side to another. |
|
|
Term
|
Definition
- It may be chosen on substantive grounds (professional opinion about who needs a medical treatment). - Statistical power and the estimation of interactions are both facilitated if the cutoff is the mean of the distribution of assignment variable scores (not possible if Ps trickle into the study slowly over time). ... |
|
|
Term
| Function ("functional form") |
|
Definition
The characteristics of the true relations among variables, represented graphically by the shape of the relations (e.g., is it a curve?) and represented statistically by a model that may include nonlinear terms (e.g., power and interactions) or other transformations. |
|
|
Term
| Advantages & Disadvantages to Randomized Designs |
|
Definition
- It ensures that alternative causes are not confounded with a unit's treatment condition. - It reduces the plausibility of threats to validity by distributing them randomly over conditions. - It equates groups on the expected value of all variables at pretest, measured or not. - It allows the researcher to know and model the selection process correctly. - It allows computation of a valid estimate of error variance that is also orthogonal to treatment. - - - - - - - - - - - - - - - - - - - - - - - - Randomization does not prevent units from maturing or regressing. - Doesn't prevent events other than treatment from occurring after the study begins. - Pretests can still cause a testing effect. - Changes in instrumentation can still occur. |
|
|
Term
|
Definition
- When you are assigning subjects to conditions, make sure you know what your subject is (a person, team, animal, etc.), know your unit of analysis; do not say you are randomizing participants to assignment when you are randomizing at the wrong units (e.g., when assigning teams, make sure you are assigning participants at the team level not the individual level). |
|
|
Term
|
Definition
These designs use two or more IVs (called factors), each with at least two levels.
|
|
|
Term
| Advantages and disadvantages of factorial designs |
|
Definition
- They often require fewer units - They allow testing combinations of treatments more easily. - They allow testing interactions. - - - - - - - - - - - More subjects (you have to have a larger sample) - Rapid statistical and analytical complications as the number of IVs increases. |
|
|
Term
|
Definition
- In a ____ design, each level of each factor is exposed to (crossed with) all levels of all other factors (e.g., educational experiment in which some students in ea. classroom are exposed to treatment and some to control, factor is ____ with classroom).
- In a ____ design, some levels of one factor are not exposed to all levels of the other factors (e.g., classrooms receive treatment but no control condition, classrooms are _____ within treatment conditions). |
|
|
Term
| The 3 principles of quasi-experimental designs |
|
Definition
1. Identify plausible effects to internal validity. 2. Primacy of control by design 3. Coherent Pattern Matching (aka coherence or pattern matching)
|
|
|
Term
| One Shot Case Study Design |
|
Definition
X O This design obtains one posttest observation on a respondent who experienced a treatment, but there are neither control groups nor pretests. Internal: History, Maturation, Selection, and Mortality. External: Interaction of Selection and X. |
|
|
Term
How to improve on the one-shot case study.
|
|
Definition
Use multiple substantive posttests.
|
|
|
Term
| One Group Pretest-Posttest Design |
|
Definition
O1 X O2 A single pretest observation is taken on a group of respondents, treatment then occurs, and a single posttest observation on the same measure follows. - Internal: History, Maturation, Testing, Instrumentation, possible regression problem, and interaction of selection and maturation. - External: Interaction of testing and x; interaction of selection and x; possible reactive arrangements.
|
|
|
Term
How do you fix problems with one-group pretest-posttest design?
|
|
Definition
Use a double pretest or use a nonequivalent dependent variable.
|
|
|
Term
Removed & Repeated Treatment Designs
|
|
Definition
O1 X O2 O3 X O4
- This design adds a third group to the one-group pretest-posttest design and then removes the treatment before a final measure is made. Internal: Statistical conclusion validity; design may be unethical or correlated with measures of aggression, satisfaction, or performance; possible resentful demoralization and compensatory rivalry. O1 X O2 X O3 X O4 - The introduction, removal, and reintroduction of a treatment over time to study how treatment and outcome covary over time. - Internal: cyclical maturation, construct validity, history . - External and statistical conclusion validity.
|
|
|
Term
How to improve removed and repeated treatment designs
|
|
Definition
- Using repeated treatment design. - Better with transient effects, unobtrusive treatments, a long delay b/w initial treatment and its reintro, with no confounding temporal cycles of the treatment w/treatment's intro, removal, and reintro; good when reintros of treatment are frequent and randomly distributed across time, creating a randomized experiment to which time blocks are the unit of assignment.
|
|
|
Term
Posttest only design with nonequivalent groups
|
|
Definition
- Add a control group to the one-group posttest-only design. Used if treatment begins before the researcher is consulted so that pretest observation are not available on the scale used at posttest. NR X O1 - - - - - - - - - - NR X O2 Internal: Not really any unless differential testing effects occur (but it's rare); b/c you are not using pretest, it can entail large costs for detecting selection biases.
|
|
|
Term
How to improve upon static group comparison
|
|
Definition
Use an indepenndent pretest sample: NR O1 X O2 - - - - - - - - - - - - - - - NR O1 X O3
|
|
|
Term
Matching & Stratifying non-equivalent groups (including using cohorts as controls)
|
|
Definition
- Synonymous with blocking, sometimes more specific to imply blocks in which units are exactly equal (rather than just similar) on a matching variable. - Decreases odd of selection biases...
- The process of creating homogenous groups of units in which each group has more units than there are experimental conditions. -Problems with ______: Undermatching and concern for producing result that is further away from the right answer than if _____ hadn't been used at all.
|
|
|
Term
|
Definition
- A study that contrasts units with an outcome of interest to those without the outcome to identify retrospectively the predictors or causes of the outcome. - Internal: Difficulty of operationalization and selection of cases; definitions and measures may change over time (instrumentation), attrition, fallible sources such as memory or records (hence, difficulty of classification of whether Ps were exposed to the treatment). Improving study relates to fixing problems or preventing these possible problems from occurring in the study...
|
|
|
Term
| Nonequivalent control group design (untreated control group design with dependent pretest and posttest samples) |
|
Definition
- Most common of all quasi-experiments; in the most basic form, uses a treatment group and an untreated comparison group, with both pretest and the joint use of a pretest and a comparison group makes it easier to examine certain threats to validity. Internal: Selection and selection will combine with other threatss additively and interactively: Selection-maturation Selection-instrumentation Selection-regression Selection-history |
|
|
Term
Ways to improve the nonequivalent control group design.
|
|
Definition
1. Using a "double pretest": O1 O2 X O3 - - - - - - - - - - - - - - - - O1 O2 O3 This allows the researcher to understand possible biases in the main treatment analysis, if "treatment effects" emerge in the analysis of O1--O2, similar biases may exist in the analysis of O2--O3. 2. Using swtiching replications: O1 X O2 O3 - - - - - - - - - - - - - - - - - - - - O1 O2 X O3 Given the contextual differences between the first and second treatment, the second introduction of the treatment is a modified replication, probing both an internal and external validity issue of whether this new context changes the treatment effect. 3. Using a reveresed treatment control group O1 O2 X+ O3 - - - - - - - - - - - - - - - - - - - - O1 O2 X - O3 X+ represents a treatment expected to produce an effect in one direction and X- represents treatement expected to produce an effect in opposite direction. This design can have special construct validity advantages. If treatment X+ improves scores and X- decreases scores of the comparison group, a statistical interaction should result suggesting a treatment effect. |
|
|
