Qualitative Research 

Firsthand observation of real world phenomena. 

Three qualities: focus on natural inquiry, researcher as instrument of data collection, report focuses on narrative data vs. #'s (ex. case studies) 

Systematic Review 

Process of collecting, analyzing, and critiquing data from studies including in the review

Limitations 

Aspects of study that a researcher knows may influence the results of the study or generalizability of the study (little control). Examples: small sample size

Latent purpose

Purpose that emerged as a result of the study being conducted (ex. the instruments used in the study were tested as reliable and valid and can be used with similar populations) 

Control Variable

A variable that does not change or is held constant in the study to analyze the relationship between other variables 

Independent Variable 

Variable that affects or is presumed to affect the dependent variable under study

Ex. (Intervention or treatment being given in experiment) 

Dependent Variable 

What is measured in an experiment and what is affected during the experiment (is DEPENDENT on the independent variable) 

Outcome study: DV= outcome measure, IV= treatment

0₁

Researcher collect data at one single (and only) entry point, no true intervention or treatment offered 

Ex. observing a wk meeting, answering a survey

XO₁

Defined treatment effect or intervention was given to a sample. Data collected about impact or non-impact. 

Ex. students take a stats course (X) and are given a final exam (O₁₎

O₁ X O₂

Data collected before main treatment (O₁₎  and X occurs for time it takes to administer (hrs, days, wks, mos, etc.) and a post-test (O₂₎ occurs

Ex. Daylong stress seminar, before participants fill out survey and after they fill out the same survey. 

Multiple Time Series Design with an Intervention 

O₁ X O₂ O₃ O₄

Pre-test and post tests occur before and after the main treatment, determining the long-term impact of the treatment effect 

Substance abuse program (mandatory), behaviors survey before (0_{1) ,} treatment/program (X) , a 1-month, 3-month, 6-month follow up (O₂ O₃ O₄₎

Prevalence

Number of cases of a disorder observed (new or previously exisisting) in a specific period of time  

Incidence 

Proportion of new cases of the target event or behavior in the population studied during a specific time interval 

Sample drawn in such a way that every person in population has an equal and independent chance of being selected 

(Ex. School adminstrator, 4 schools, opinions, list of all students, randomnly selects 150)

Convenience Sampling

A sample that is easily accessible, does not represent the entire population, may be chosen because of time, availability, financial constraints 

Ex. studying effects of DV, sample residents at local DV shelter

Systematic random sampling

Regular sampling using a known interval with a randomized start 

Ex. School administrator, high school, 200 students sample, etc. 

Stratified random sampling

process of selecting a sample in such a way that identified subgroups in the population are represented in the same proportions in which they exist in the population 

Ex.  undergrad students, subgropups (classes), etc. 

Non-Probability Based Sampling

Subgroup of study participants selected by researcher, unknown in they represent the larger population 

Five Types of Non-Probability Sampling

Purposive Sampling 

Sample selected by researcher because prior knowledge suggest sample is representative or sample has needed information 

Snowball Sampling

Asking research participants to ask others they know to participate in the study 

Evaluation Research

Sytematic inquiry to describe or assess the impact of specific programs or interventions by determining its activities or outcomes

Reliability 

Extent to which a measurement instrument gives consistent results 

Ex. An instrument used to measure depression has been tested for reliability through applying instrument in numerous studies (similar populations) and produced consistent/similar results

Validity

The extent to which results respond to "true" positino of attribute being measured

Internal Validity

The validity of the attribution that X causes Y

External Validity 

The stability or generalizability of the X-Y relationship

Threats to Internal Validity (Experimental) 

History, Maturation, Testing, Instrumentation, Statistical Regression, Selection, Mortality, Selection (maturation)

Threats to Internal Validity (Theoretical)

Multi-dimensional independent variable 

Experimenter Expectancy 

Subject Expectancy

Threats to External Validity

Generalizability,

Multiple Treatment Interference, 

Pre-test Sensitization

Selection- Treatment Interaction

Bias

Having a tendency, prejudice, or preference toward a particular perspecgive (six example categories)

Selection Bias

Threat to Internal validity of the study's findings

Occurs when participants for comparisons groups are selected differently 

Ex. Study on spending habit differences in men and women (30% women vs. 70% men)

any entity that can take on different values

Ex. variable of gender, values- male, female, intersex, etc.

Respondent Bias (Subject Expectancy) 

When a respondent answers questions based on what they think the questions wants versus their true beliefs, 

threat to internal validity 

Hypothesis 

A prediction of study outcomes, 

based on literature, theory, an educated guess, or observation of the phenomenon

Guesstimate

A scientific hunch about any part of a research study based empirically on prior experience, literature, best practices, etc.

combination of the word guess and estimate

Sample Size

Small = less than 30

Large= over 250

Normal= in between

Empiricism

Pursuit of knowledge purely through experience by means of observation and experimentation (ex. empirically based practice)

Parametric Stats/Tests

Test of significance, appropriate when the data represents an interval or ratio scale of measurement and when specific assumptions have been met 

Ex. ANOVA (differences among means), Independent or dependent T-test

Non-Parametric Stats/Tests

Used when the data represent a nominal or ordinal level scale or when assumptions required for parametric tests can not be met, used with small sample size, biased samples, etc. 

example - Chi-square (compares frequencies), Cohen's d (effect size)

Underrepresentative Sample

Intervention

Independent variable in an evaluation or research study (what is designed to be studied)

Time Effect

Historical event at the time of the study that happens to all participants and affects the results 

(Ex. 9/11...)

History (internal validity)

A specific event occuring between the 1st and 2nd observation that may affect the dependent variable score change

(ex. math program admin. outside of school to 1st graders did not cause outcome, the first graders observed watch SS daily and the show presented math concepts/problems discussed in class)

Maturation 

possibilty that results are due to changes over time with participants (ex. human development, fatigue)

testing knowledge increases from participants in parenting program, each week. Participants show no increases or retaining of knowledge. May have been due to fatigue and inability to review parenting program material on a consistent basis.

Testing Effect 

Internal threat, pretest influences the post-test 

Pretest on healthy eating habits caused participants to eat better simply due to taking the test...

American Evaluation Association 

Guiding Principles for Evaluators

Systematic Inquiry (data-based), competence, integrity/honesty, respect for people, responsibilities for general and public welfare