The variable under the experimenter's control

Argues causality is always qualitative because we make inferences

In psychology we often cant measure things like we would in the hard sciences

In psychology we often don't know directionality

We often base inferences on theoretical models

very concerned with how we approach cause and effect

Three conditions needed for cause and effect:

• contiguity (events happen together, but correlation =/= causation)
• temporal precedence: the cause happens before the effect
• Constant conjunction: the cause had to be present whenever the effect was obtained

a cause referes to something both necessary (needed) and sufficient (all you need) to make an effect happen

Another way to talk about correlation.  When there are muliple factors we can control for, we don't know what is necessary and sufficient

Three conditions necessary for inferring cause:

• cause must happen before effect
• cause and effect have to be related
• other explainations must be eliminated

Three ways to infer cause

• The method of agreement - the effect is present whenever the cause is present
• The method of disagreement - the effect is absent when the cause is absent
• The method of concominant variation (best method) - the effect is present whenever the cause is present AND the effect is absent when the cause is absent (the logic behind control groups)
• USE WATERBOTTLE EXAMPLE

Wrote about theoretical underpinnings of how we go about proving thi ngs

Confirming a cause is logically impossible.  What we need to do it confirm a disconfirming occurence.  This allows us to throw away our original hypothesis

Proving something isn't possible, but ruling out other possibilities is

We never prove a theory, we reject evidence that refutes our theory

Type I error - rejection of the null hypothesis when it's actually true

Type II error - failing to reject the null hypothesis when it's false (not as worried about this, because we're not rejecting the truth)

p = .05 means there's a 5% chance we're wrong about rejecting the null

When we're talking about people, by definition the null will usually include at least some people.  

When we start off, we have the belief that the null is false (because something is always correlated with something else, some of the time)

As we improve instrumentation, design, larger sample sizes, our alternative moves farther away

Power: the ability to detect a significant difference.  Power can be detected in several ways:

• can improve the logical structure of experiment
• reduce the variance from error
• can increase sample size

There is a 50/50 chance we will falsely reject the null because we're using non-directional tests

We use p=.05 either way.  

Because there is so much inter connection in the phenomena we're studying, it's like a coin toss

We should use point prediction (Tx for depression will cause participant's Beck Inventory Scale score to be in the 20-23 range), not null hypothesis testing (the two groups are different)

Meehl wants to increase precision of the design of the study to make better inferences.  He wants our science to be able to build one study after another because we're not just rejecting the null.

David Lykken

It's important to add replication to be more certain of our findings

The types of replication:

• Literal - doing exactly what was done before 
• Operational - using the same methods and sample, with a different experimenter and different place
• Constructive/conceptual - not using same methods, but using the same theoretical and conceptual underpinnings

Useful when looking at meta-analyses for overall effect sizes

Wrote a book on power analyses

The logic of null hypothesis testing needs to be interpreted in to probability statements

Modus Tullens: an error in interpreting if A then B, if not A then not B statements.  When we reject a null hypothesis we are making probability statements.  Therefore, when we take a probability statement and dont put it in if-then format, we are making a Modus Tullens error.  

A priori probability - chances of an event occuring before testing (at baseline)

Posterior probability : the probability of an event occuring after experimentation

P(Data|Ho) =/= P(Ho|Data) - if data is true, then Ho is true

These are not the same, because there are different levels of probability for each (not testing absolutes)

Wants us to use effect sizes in the form of CIs 

Pros and Cons of hypothesis testing 

Recommendations:

• report actual p-values, not just >.05
• Treat p=.05 as interesting, but don't put more value on significance than you should
• Very small p-values instill confidence that we may have something
• Report results from every single test you run, significant or not
• Report all secondary analyses (n, effect sizes, means, CIs, etc)

It's important to put a hypothesis into operational terms

will help determine what type of research design you want to use

Moderation

typically have to do with "under what conditions" questions

usually categorical, but dont have to be

ex: is this association different for males vs females?

Religion x gender would be a significant interaction

Look at interactions before main effects, because they're the most complex

Mediation

Mediation questions typically have to do with mechanisms.  The deeper process, how the problem unfolds, and through what ways and processes in which one variable leads to another

Ex: without conflict avoidance, there would be no interaction between parental experiental avoidance and children's adherence behaviors

making an observation, and then doing research to explore the phenomena 

ex: Pavlov discovering dogs drooled with just a bell, not food

Look for inconsistencies, reading literaure, etc

Frame your research question with the background literature, inconsistencies, etc

the definition of a specific process within the context of the field, on the basis of specific operations used in the experiment

The more ways you think about conceptually operationalize your variable, the stronger your experiment will be

Comparison between two different groups

treatment A vs treatment B

can be effected by assignment bias, researcher/particiapant expectancy bias, practice effects

comparison of groups at two time points.  Mix of between group and intrasubject design.  Good when it is difficult to find subjects.  However, practice effects and carryover effects can be potential confounds.  We can distract or give time in order to minimize carryover effects.

Pre-test/post-test design

used for individual difference measures.  Rely on a single subject or small number of subjects, when subjects are very rare.

Ex: assigning one group to treatment as usual, and another group to treatment as usual and treatment B

Using a sample to illustrate another sample

ex: using certain brain simulation in rats to demonstrate an autism sample.  "One man's analog is the next man's research pool."

ex: recruiting a psych 101 student for psychotherapy research, instead of real therapy patients.  In analog samples, we see tighter controls, making internal validity stronger.

Thinking about analogs shouldn't be yes or no, we should look at it dimensionally

Based off probability

The law of probability says if we randomized, groups will be even on the variables we don't care about

Small groups - higher probability there will be variability within each group, but still want to use random assignment.

The difference between an experiment and a quasi-experiment

Parametric tests - assumption that people are randomly assigned

Internal consistency - the extent to which you can attribute changes to the dependent variable

Blocking - rank order participants based on a particular characteristics, and then evenly distribute/randomize the highest qualities down through the lowest qualities

Matching - want to match two (or however many groups there are) people with the same level of characteristic, and randomly assign them to group.  Useful when people come in sets - twins, husband/wife, grad school cohorts

Regression to the mean - if we have two different groups that have different means, we can't choose overlap of participants (Kazdin) due to regression to the mean of extreme scores, which would be a confound.  Instead, we take the two differing groups and randomly assign each to the conditions. 

·         In general, we don’t like post-test only design  Ex: when we’re measuring the amygdala (cant change amygdala size) Ex: testing effects

·         Ex-post facto post-test design – different conditions but no control group.  Ex: when unethical to not give treatment

·         Pretest-post-test controlled group design – we like this one, especially with random assignment.  Want to make sure the two groups are similar before treatment.   Don’t want to run multiple t-tests because it would inflate type-I errors.  Could do a 2x2 ANOVA.  If we get a significant interaction, we do a post-hoc test.

·         Soloman four group design – tries to see if pre-test influenced (changes) the post-test design (phenomenon being studied).  Requires many many participants, etc.  We don’t usually see them because they’re very expensive. 

·         Proxy measures – if you think your measure changes the phenomenon (like taking an IQ test) you can use a different measure at pretest as long as it highly correlates with your post-test measure. 

·         Time series designs – introduce measure, introduce intervention, introduce another measure, withdraw intervention, introduce measure.  Makes a design completely within group (reduces between subjects variance, reduces number of subjects needed).   There may be treatment effects.  

Help rule out alternative hypotheses

·      Only the independent variable should be different between the experimental group and control. 

·      Must pick control group carefully based on what we want to study

·      Quasi-experimental design – variables occur naturally, nothing is being manipulated. 

·      Design – framework of an experiment, setting IVs

·      Methodology – measures, DVs, sampling, limitations, inclusion and exclusion criteria

o   How do we make the distinction if we’re controlling for variables with design or methodology?

§  In general, design controls for primary research question, methodology controls for other things.  However, this is not always the case. 

·      When we think about control groups, we should always ask ourselves ‘what are we controlling for?’  We can control in design or in methodology

o   Waitlist

§  can be unethical to withhold treatment

o   No treatment

§  can be unethical to withhold treatment

§  People have less incentive to participate, differential attrition

§  Internal validity concerns

o   Placebo

§  Controls for treatment effects, expectation, hope that treatment will make things better. 

§  The Declaration of Helsinki – Experimentation during Nazi times.  The notion that you will recruit people who are sick and promise them a cure, but not help them.  The Declaration of Helsinki was modified to say that placebo control groups shouldn’t be used unless there is an extraordinary reason.  Instead, we would use treatment as usual.  Raises the bar, because it will be harder to get significant results. 

§  In medicine, the design is treatment as usual plus something, and treatment as usual. 

o   Yoked

§  Executive monkey study – one group was trained to do an activity.  Every time they succeeded in completing a task, they got a raisin.  The yoked control group got a raisin every time the experimental group got a raisin.  They then switched to giving a shock every time the executive monkey gets a shock for failing the task. The executive monkeys (without control) developed the monkey version of hypertension and diabetes.  When we yoke, we are controlling for individual differences we think are relevant.  Slow monkeys got shocked a lot, as did their yokes.  Fast monkeys didn’t get shocked a lot, either did their yokes.  Much more sophisticated way than accounting for random error in statistics.  

·      Used in early physiological psychology.  Normal curves and SDs were first starting to be used.  Focused on group differences. 

·      Broca – worked in the 1860’s with patients who had cognitive deficiencies

·      Boring – used a lot of single subject design

·      Wundt – worked with children with learning disabilities. 

·      Cattell – 16 Pf – designed to come up with 16 traits that capture every human individual difference. 

·      Fisher – 1930’s.  Came up with control vs experimental groups, derived from agriculture. 

·      In the 30’s, 40’s, 50’s, case studies were mainly being used in applied psychology. 

·      Eysenck – pulled insurance records and did a meta analysis.  Send psychotherapy was ineffective.  First one in psychology to say ‘lets collect data.’

·      Measurement

·      Extraneous variables

·      Intervention related – need to get the intervention variance large enough that it overcomes the extraneous variable variance.  We manage this by repeat assessment. 

·      We need to monitor variability and do replication.  

Within Series Design – ABAB design

·      Elements – what you’re doing at any given time

·      Starts with baseline – data before intervention is given.  Want to allow baseline data to stabilize before we give the intervention.  Ideally, three time points makes a baseline.  If the baseline is still unstable, you extend it. 

·      Phases – an intervention.  Looking at effectiveness, can change or add a different intervention if the previous one isn’t working, etc.  We don’t want to add one intervention after another without going back to baseline, because we wouldn’t be able to tell if carryover effects were occurring.

·      Withdrawals – want to control for extraneous variables by taking away the intervention.  Want to see if the intervention is effecting the target behaviors.  Doesn’t need to be as long as the intervention stage, because it doesn’t  make sense to have maladaptive behaviors go back to where they were. 

·      Reversals –

·      Can’t use an ABAB design when interventions involve learning or skills… because we can’t take the skills back once they’re taught.

·      ABAB –baseline intervention baselineintervention

·      ABAC –baselineintervention baseline intervention 2

·      BAB – intervention baseline intervention – when behaviors are aversive (severe head banging in autism), sometimes we need to change the intervention.  We do a very brief baseline, and then go back to intervention.  Allows you to demonstrate the efficacy of what you’re doing. 

·      Change in criterion design: different criterion throughout the B phase.  One criterion goes for ex. 3 days, then two days, then six days, then another two.  Same intervention, but if you exceed X number of cigarettes during this criterion, you’ll have to send $50 to a group you hate.   It will be X cigarettes, then x + 3, then x – 1, etc.  The criterions are not predictable.  Criteria guided behavior – the intervention is influencing behavior.  We show that the criterion guide behavior, and then fade the criterion until the behavior is extinct.  This design tends to be used with health behaviors.  

Within group/intersubject design

baseline, treatment 1, baseline (withdrawal of treatment), treatment 2, baseline (withdrawal of treatment), treatment 1 and 2

Treatment b produces a positive change, c produces a non significant change, and bc creates a positive change comparable to b

How do we find the mechanism behind the pattern of results?  If it works and how it works?  The graph shows us IF it works.  In order to look at HOW it works, we include other measures (in reading comprehension where b is peer tutoring, and c is independent reading, other measure would include evaluating visual attention, social skills, self efficacy, etc).  If it works – design.  How it works – methodology.

In between group – If it works – one group receives treatment, the other doesn’t.  (ABAB)

·      If it works (design)– baseline for both groups.  One group gets treatment 1 the other gets treatment 2, back to baseline for both groups

·      O X(Tx1) O           o=baseline x=treatment

·      O X(Tx2) O

·      How it works (methodology) – ex CBT vs supportive therapy was used - which assessments used, patient ratings, patient learning, specific skills acquired.  

Multiple baseline design – useful to combine within group and between group design. 

History or maturation effect that is controlled by design and methodologyWe’re measuring levels of depression

Baseline levels of depression is 100 for each participant, order is randomly assigned

We first start with monitoring A (baseline), then introduce medication (B) to participant 1 while part 2 and 3 are still at baseline.  We then introduce treatment (B) to participant 2 while part 3 is still at baseline.  Last, introduce part 3 to treatment (B), so all participants have treatment.

EX: anxiety w/ exposure

The varying lengths of baseline are part of the design

Compared to an ABAB design, you can answer more questions with a multiple baseline design.  Don’t have to work about time effects or order effects, because it is built in.  We can use this design for interventions where somebody is learning something.  With learning, we can’t do a withdrawal phase, so this design works well. 

Alternating treatments design 

no baseline.  (B) and (C) only.  EX: used when you’re looking at your behavior and it’s impact on somebody else.  In situations where you’re not able to replicate a true experiment, but can have some control over a natural setting.  In situations when you’re not possible to have a baseline – ex: for a natural disaster.  Don’t have a baseline, but can track effects of treatment.  Need to use a rapidly alternating treatment with somebody who is very very distressed, within or across meeting periods.  Should be able to see change over time separately from the other treatments being used. 

ATD tends to be something that is not heavily relied on, but can be used in a wide varieties of ways.  The easiest time series design.  

Simultaneous Treatments Design 

Scheduled treatment intervals used when the research question involves interest in client preference.  Heavily used in school systems (ex: with autistics kids).  They get to choose if they want a cookie or a toy for reinforcement. 

·      Client preference is becoming a higher order goal for clients in applied settings

Simultaneous Treatments Design 

Scheduled treatment intervals used when the research question involves interest in client preference.  Heavily used in school systems (ex: with autistics kids).  They get to choose if they want a cookie or a toy for reinforcement. 

·      Client preference is becoming a higher order goal for clients in applied settings

·      Internal Validity

o   How well can we make inferences of causation between the independent variable and the dependent variable?

§  Rule out rival hypotheses

§  Control for 3^rd variables  

• History – things that happen in the middle of treatment
• §  Maturation – any natural changes that happen over time.   Problem is that people mature at different rates.

§  Attrition – people who drop out might be significantly different in their characteristics than those who remain in the study

§  Instrumentation – using slightly different instruments can influence validity.  Different instruments can be using different mechanisms. 

§  Testing effects – initial testing can change outcome of second test.  Individual might try to change answers to reflect themselves in a positive light when they know they’re being observed.

§  Regression to the mean – extreme scores statistically tend to regress to the mean with repeated testing

§  Selection bias – The people chosen for sample could be inadvertently different than the general population.  If samples aren’t randomly assigned, might introduce third variables (ex: some people might be more likely to volunteer for a study than others)

·      External Validity

o   How well can we generalize the findings of a study to other people or situations?

o   Threats to external validity

§  Settings – In a laboratory setting, may influence if you can generalize results across settings

§  Sample characteristics – how large, what groups, how many people in each group, is it a representative sample of the population, etc

§  Stimulus characteristics –

§  Novelty effects – if there is an effect other than the  treatment effect.  Something new that changes the individual’s reaction to the stimulus.  Just because it’s so far away from normal. 

§  Multiple treatment exposure – what type of interference might be going on with other treatments, and how that will effect your ability to generalize.

§  Timing of measurement – what order will you present multiple treatments in, pre/post measurement?  How many times?  At what time?

§  Sensitization – doing a pretest and posttest might sensitize participants being sensitized to a test 

·      Construct Validity

o   How well we are measuring what we think we’re measuring

Threats to construct validity

§  Mixed measure – claiming to tap into one thing, but really there are multiple things mixed in that are involved in results

§  Conceptualizing constructs

§  Single measures/operations – if only measuring something one way, how sure can you be that you’re tapping into what you want to tap in to?

§  Attention/contact - Is it just contact with the participants that is giving you results, or the actual construct you’re trying to measure?

§  Stimulus sampling - May not really be tapping into your construct completely

§  Experimenter effects – the experimenter (inadvertently or not) may push things in the way they want them to go because of their investment in the construct. 

§  Cues Experimental situations - Will experimenter’s demeanor change they way participants respond to measures?

§  Convergent validity

ú  To what extent are different measures of a construct that are meant to relate to each other actually related?

ú  Measures of same construct correlated well with one another

§  Discriminant validity

ú  To what extent are measures of different constructs that are not supposed to relate to each other actually unrelated

ú  Measures of different constructs do not correlate with one another

§  Campbell and Fisk 1959

ú  Construct validity requires evidence of BOTH convergent and discriminant validity

ú  Close attention needs to be paid to the method(s) used to measure constructs

ú  Multtrait-multimethod matrix

ú  Multtrait-multimethod matrix

·      The MTMM matrix is used to help a researcher evaluate convergent and discriminant validity

·      Requires at least two constructs that are proposed to be “different” and at least two different ways to measure the proposed constructs

o   Multiple traits

§  Proposed consturcts

ú  Some people would argue that they might be the same, while others would argue that they do differ

ú  Looking for related, but different

o   Multiple methods

§  Measures that will be used

ú  Should differ as much as possible

ú  “method variance” – want high method variance, to ensure the methods are not highly correlated with one another

·      Reliability Diagonal

o   Essentially a correlation between the measure and itself

§  Would typically be a perfect correlation (r=1.0)

§  Instead, usually substitute an estimate of reliability

ú  Test-retest, internal consistency, etc.

·      Validity diagonal

o   This is how you demonstrate convergent validity

o   Needs to be high to show that you are measuring the same thing (construct)

§  Must have r>0

§  Also needs to be higher than the heterotrait-heteromethod triangles

·      Heterotrait-monomethod trianges

o   These trianges show how much method variance is present

o   Share only method, not trait or concept

o   If correlations are high, it is because the method is responsible for the correlation

§  Methods factor

·      Heterotrait-heteromethod triangles

o   Share neither method, nor trait

§  Ideally, correlations here should be zero

·      MTMM discriminant validity

o   Correlations of the validity diagonal need to be greater than the correlations of BOTH the geterotrait-monomethod triangles and the hetertrait-heteromethod trianges

§  Validty r> heterotrait-monomethod r AND validity r > heterotrait-heteromeothd r

·      MTMM is essentially a pattern matching approach to determining convergent and discriminant validity

·      Practically: different researchers can ultimately arrive at different conclusions

·      Statistical Conclusion Validity

o   How correct are or how reasonable can we feel about the conclusion of a study?

o   Appropriate use of statistics to infer whether the IV and DV covary.

o   Statistics inferences regarding:

§  Whether presumed cause and effect covary

§  How strongly they covary

§  Fishing for effects – looking first at obvious effects

§  Procedure variability – how closely they’re followed may lead to variability

§  Low power/effect size – increased chance for type II error

§  Subject heterogeneity – how alike/different are subjects?  What characteristics are you looking for?  If everybody is the same/very different, how does this change your ability to look to results/covariation?

§  Measures reliability – how well can you use a measure to draw statistical conclusions if your measure isn’t reliable?

§  Error rates – error in study (procedure, measurement, treatment fidelity) can limit ability to draw strong statistical conclusions 

·      Cultural Sensitivity (Okazaki & Sue, 1995)

o   Race, Ethnicity, Culture

§  What do they mean?

§  Which ways are you operationalizing these things?

ú  Race as a social construct, vs race as physiological differences

§  How alike are members?

ú  Individual differences vs. group charactistics

§  Methodologial issues

ú  Population focus

·      Selection of participants?

·      Comparisons between groups?

ú  Sampling

·      Self-report?

·      Subjective call by researchers?

·      Small sample sizes? – how are things weighted compared to the general population?

·      Recruitment?

ú  Equivalent measures?

·      Biases?

·      How was the measure normed?  How appropriate is the measure for use on the sample you’re looking at?

·      How assess?   One method better than another?

·      Data interpretation?

§  Guidelines

ú  Assumptions should be made explicit – clearly define use of ethnicity in the study.

ú  Fully discuss sampling methodology – can allow replication in the future as well combining with other studies (metaanalyses) to make statements about the sample overall

ú  Maximize the significance despite small sample sizes

·      Availability for metaanalyses

·      Ensure you can draw strong inferences between the covariation between things - statisical conclusion validity

ú  Use consultants – to gain cultural context

ú  Findings should generate Ho for further research.