Methodology

- the diverse principles, procedures, and practices that govern research

- a way of thinking

- alerts us to the issues that effect how we examine and interpret phenomena

Research Design

The plan or arrangement used to examine the question of interest.

Hypothesis

- educated guess about the results of a study based on the principal research question which identifies an IV and DV(s) and sometimes a directional correlation between the two

- "if-then" statement

- the "if" is the IV and the "then" is the resulting variable

Q: If results are consistent with a hypothesis, has it been proven?

A: No, data can be taken as proof of a hypothesis only if no conceivable alternative hypothesis can account for the results or if the predicted relations would be obtained if and only if the hypothesis were true.

Parsimony

Parsimony directs us to select the simplest version or account of the data among the alternatives that are available.

Plausible rival hypotheses

An interpretation of the results of an investigation on the basis of some other influence than the one the investigator has studied or wishes to discuss.

Plausible rival hypotheses vs. Parsimony

Parsimony: adopting the simpler of 2 or more explanations that account equally well for the data

...whereas...

Plausible rival hypotheses: seeks to ask whether there any other plausible interpretations we can make of the findings than one advanced by the study

Q: What is the role of methodology in addressing whether plausible rival hypotheses are relevant to interpretation of data?

A: Methodological practices are intended to rule out or to make implausible competing interpretations of the results.

Study's findings vs. conclusions

Findings refer to the results that are obtained

...whereas...

Conclusions refer to the explanation of the basis of the finding

Q: What does Kazdin mean by saying that the goal of research is to make convincing hypotheses?

A: Convincing relates to the quality of the research design and those features that permit sound inferences to be drawn.

A well-designed study is persuasive to the investigator as well as to others in the scientific community.

Operational definition

Refers to defining a concept on the basis of the specific operations used in the experiment.

3 limitations to operational definitions...

(1) Operationalizing an abstract notion, may greatly simplify the concept of interest or focus only on part of that construct.

(2) An operational def. may include features tha are irrelevant or not central to the original concept.

(3) An operational def. pertains to the use of single measures to define a construct.

Q: How is multiple operationism useful in representing constructs?

A:

- Researchers often wish to include many different constructs.

- For research it is usually better to examine multiple constructs and few measures.

3 major types of variables used in psychological research...

(1) environmental or situational variables

(2) instructional variables

(3) subject or individual difference variables

Environmental or situational variables

What is done to, with, or by the subjects. Providing qualitatively different conditions to the subjects.

ie.) Treatment vs. no treatment

OR

more treatment for some subjects and less for others

Variations in what the participants are told or lead to believe through verbal or written statements about the experiment and their participation.

Attributes or characteristics of individual subjects. Encompasses characteristics to which the subjects may be exposed (ie. environment, context, living conditions).

3 major types of studies in psychology...

(1) True experiments

(2) Quasi-experiment

(3) Case-control designs

True experiments

Consist of invetigations in which the arrangements permit maximum control over the IV or manipulation of interest.

ie.) assign subjects to different conditions randomly.

Strongest basis for drawing inferences.

Quasi-experiment

Those designs in which the conditions of true experiments are approximated.

ie.) school intervention program to prevent drug abuse / teen pregnancy.

No random assignment and factors vary greatly over conditions.

Case-control designs

Selects subjects who vary in the characteristics or experience of interest. (ie. differences between smokers and non-smokers OR males and females)

Comparison groups are important for drawing conclusions here.

Laboratory vs. Applied research

Laboratory: Investigations conducted in a specific research setting (ie. psychology department, research space where sugjects are brought to engage in some lab procedure). Devoted to basic questions under highly-controlled conditions.

...whereas...

Applied: ie.) a clinic setting where patients are seen for treatment; Evaluate different types of treatment or different populations who have one type of problem vs. another.

Analogue vs. Clinical research

Analogue: Non-clinical group. Carefully defined research question under well-controlled conditions. Research important for clinical applications (of particular treatments for example). Bears little resemblance to clinical work.

...whereas...

Clinical: Clinically diagnosed syndromes group. Psychotherapy research. ie.) studies of therapeutic processes under well-controlled conditions. Process in lab should closely resemble phenomenon in daily life

Efficacy vs. Effectiveness research

Efficacy: Treatment outcomes obtained in controlled psychotherapy studies conducted under lab & quasi-lab conditions.

...whereas...

Effectiveness: Treatment outcomes obtained in clinic settings where usual control procedures are not implemented (pations seek Tx, many present with multiple clinical problems).

Cross-sectional vs. Longitudinal research

Cross-sectional: makes comparisons between groups at a given point in time (ie. behavior assessed concurrently by evaluating children of different ages).

...whereas...

Longitudinal: makes comparisons over an extended period, often involving several years (ie. how behaviors change in a given sample over time).

Generating vs. Testing hypotheses

 ????

Studies can sometimes do both:

~A study may test a hypothesis, but the data is gathered and used to extend the description of the phenomenon in ways that beg for further theory.

OR

~A study may begin with a careful description & end with a model or conceptual view that can be tested in subsequent studies.

Internal Validity

- The extent to which an experiment rules out alternative explanations of the results.

- Internally valid when: the results can be attributed with little or no ambiguity to the effects of the IV (thing that someone actively changes).

10 threats to internal validity

(1) H -History

(2) M - Maturation

(3) T - Testing

(4) I - Instrumentation

(5) R - Statistical Regression

(6) S - Selection biases

(7) A - Attrition

(8) C - Combination of selection & other threats

(9) D - Diffusion or imitation of treatment

(10) S - Special treatment or reactions of controls

Internal threat: History

- Any event (besides the the IV) occurring in/outside the experiment that may account for results.

- Events common to all subjects in their everyday lives.

Internal Validity: Maturation

Changes over time from processes within the subjects (id. growing older, stronger, more tired/bored).

Internal Validity: Testing

Effects that taking a test one time may have on subsequent performance on the test (ie. practice & familiarity with the test).

Internal Validity: Instrumentation

Changes in the measuring instruments/measurement procedures over time.

(ie. In assessing behaviors, the scoring criteria that therapists use may change over time, leading to changes in the D.V. not caused by changes in actual behavior)

Internal Validity: Statistical Regression

Tendency for extreme scores on any measure to revert (regress) toward the mean of a distribution when the measurement device is re-administered.

(ie. less extreme scores @ second testing)

Internal Validity: Selection biases

Systematic differences between groups before any experimental manipulation/intervention on the basis of the selection or assignment of subjects to groups.

(Random assignment of subjects is used to lessen selection biases)

Internal Validity: Attrition

Loss of subjects / dropping out.

(Usually when investigation spans days, weeks, months, years...).

Internal Validity: Combination of selection and other threats

Influences affecting one group and not another.

(Most common: history and maturation)

Internal Validity: Diffusion or imitation of treatment

Equalization of performance of treatment and control groups due to accidental provision of treatment to control.

Internal Validity: Special treatment or reactions of controls

Compensation for the control group (for not providing the intervention)--ie. special attention, more monitoring, special privileges--can act as an unintended intervention in its own right and cause control subjects to react differently.

External Validity

The extent to which the results of an experiment can be generalized beyond the conditions of the experiment to other populations, settings, and circumstances.

8 threats to External Validity...

External Validity: Sample characteristics

The extent to which results can be generalized to others who vary in age, race, ethnic background, education, etc.

(ie. animal research-->humans?

university students-->general public?)

External Validity: Stimulus characteristics & settings

The extent to which results extend across the stimulus characteristics of the investigation.

 - features of study with which intervention or condition may be associated.

(ie. settings, experimentors, interviewers, other factors re. experimental arrangement)

External Validity: Reactivity of experimental arrangements

Influence of subjects' awareness that they are being studied in an investigation.

External Validity: Multiple treatment interference

Some experimental designs expose subjects to more than one experimental condition. Conclusions drawn about a treatment when it is evaluated in the context of other treatments.

External Validity: Novelty effects

Possibility that the effects of an intervention may in part depend upon their innovativeness or novelty in the situation.

External Validity: Reactivity of assessment

Any conditions of assessment that differ from those to which the investigator may wish to generalize.

- Obtrusive: if subjects are aware that their performance is being assessed.

- Reactive: if the awareness leads a person to respond differently from how they would usually.

External Validity: Test sensitization

Pretests are sometimes given before treatment. The pretest may sensitize the subjects so they are affected differently by the invervention. Individuals may be more or less responsive to a treatment.

ie.) subjects' awareness of what investigator is looking for

External Validity: Timing of measurement

Results may be dependent on the point in time at which assessment devices are administered.

Q: Why is internal validity often considered more important than external validity?

A:

- must have an unambiguous finding before you can ask about its generality

- must have internal validity before you can have external validity

Q: Why is external validity still important? (even though it's said that internal validity is generally more important)

A: Internal validity shows what can happen when an experiment is set up a certain way, but it does not show that the intervention would work outside of the experimental situation.

10 steps to the research process...

Steps to the research process: examples of Adopting a research strategy

ie. cross-sectional, longitudinal, etc.

Steps to the research process: Maximization of the research effect-->3 components

(1) The logic of this can be illustrated by the F statistic:

F = systematic effects (due to IV) + error variance (confounding var's)

error variance (chance factors)

(2) Musti minimize varation due to unsystematic factors (error) and control for error due to confounding or extraneous variables.

(3) Minimizing error (eliminating threats...)

Reliability

The extent to which the measures assess the characteristics of interest in a consistent fashion. In relation to measures, consistency may refer to internal consistency (how the items relate to each other), consistency between different parts or alternate forms of the same measure, and consistency in performance on the measure over time (test-retest reliability for a given group of subjects).

There are many different types of reliability.

Deception

- Can be entirely misrepresenting the nature of the experiment

- being ambiguous and not specifying all or many important details

Criteria to be met before deception is deemed acceptable

- Do possible risks (to individuals) outweigh potential benefits (to individuals/society)?

- Info to be revealed by experiment of high importance?

- Is deception essential to achieve the research goals? (special procedures to protect subjects are provided to reduce any lingering effects of the experience)

-  less deceptive or nondeceptive methods of investigation could not be used to obtain info?

- Low degree of deception and low potential for magnitude for harmful effects?

Debriefing

Providing a description of the experiment and its purposes after the experiment is completed.

To reveal any deception and/or correct any deceptive information given in experiment.

 Q: Why might debriefing fail to meet its intended purpose?

A:

- effectiveness depends on characteristics of the subject, nature of the deception, time interval between deception and debriefing

- does not erase false impressions established during the experiment

- effects of deception may linger after debriefing

2 conditions which protect participants' rights to privacy...

(1) Anonymity

Participants' rights to privacy: Anonymity

Ensuring the identity of subjects and their individual performance are not revealed.

Participants' rights to privacy: Confidentiality

Information will not be disclosed to a 3rd party without the awareness and consent of the participant.

3 major elements of Informed Consent...

(1) Competence: individual's ability to make well-reasoned decision and to give consent meaningfully.

(2) Knowledge: understanding the nature of the experiment, the alternatives available, and the potential risks and benefits.

(3) Volition: basis of agreeing to participate. Consent must be provided free from constraints or duress.

*Consent may be revoked at any time*

Intervention research issues:

2 safeguards that researchers should put in place to protect their clients when conducting a treatment outcome study...

(1) When possible, a new treatment (or evidence-based treatment) should be compared with a standard treatment rather than to a placebo.

(2) Provide all subjects with the more (or most) effective treatment in the project after they have completed the treatment control condition to which they were assigned.

Construct Validity

Addresses the presumed cause or explanation of the causal relation between the intervention/experimental manipulation (IV) and the outcome (DV).

4 threats to construct validity...

(1) A -Attention and contact with clients

(2) C - Cues of the experimental situation

(3) E - Experimentor expectancies

(4) S - Single operations and narrow stimulus sampling

Construct Validity: Attention and contact with the clients

Different attention to clients along the experimental and control groups may be the basis for group differences and threaten the CV.

Construct Validity: Cues of the experimental situation

Seemingly ancillary factors associated with the intervention that may contribute to the result.

-demand characteristics- sources of influence such as information conveyed to prospective subjects prior to their arrival to the experiment.

Construct Validity: Experimentor expectancies

Possibility that the expectancies, beliefs, and desires about the results on the part of the experimentor influence how the subjects perform.

Construct Validity: Single operations and narrow stimulus sampling

An experimental manipulation/intervention includes features that the investigator considers irrelevant to the study, but these features may introduce ambiguity in interpreting findings.

Statistical Conclusion Validity

Refers to the facets of the quantitative evaluation that influence the conclusions we reach about the experimental condition and its effect.

Null hypothesis

Specifies that there are no differences between groups.

Statistical tests are completed to evaluate whether the differences that are obtained are reliable or beyond what one is likely to find due to chance fluctuations.

What is alpha?

α

The probability of rejecting the null hypothesis when that null hypothesis is true.

(Incorrect decision) 

What is beta?

β

The probability of accepting the null hypothesis when it is false.

(Incorrect decision)

Type I Error

α

Incorrect decision

REJECT THE NULL----NULL IS TRUE

(Reject null hypothesis which is true)

a "miss"

ie.) Treatment does not work

Power

1-β

Correct decision

REJECT THE NULL----NULL IS FALSE

(Reject null hypothesis which is false)

a "hit"

Type II Error

β

Incorrect decision

ACCEPT THE NULL---NULL IS FALSE

(Accept null hypothesis which is false)

a "miss"

ie.) inappropriate dismissal of treatment

Last type of outcome (no name) in group differences  / null hypothesis table

Correct decision

ACCEPT THE NULL---NULL IS TRUE

(Accept null hypothesis which is true)

a "hit"

Effect size

The magnitude of the difference between 2 or more conditions or groups and is expressed in standard deviation units.

5 threats to Statistical Conclusion Validity...

The extent to which an investigation can detect differences between groups when differences exist within the population.

 Statistical Conclusion Validity: Variability in the procedures

 Variation in relation to experimental setting, delivery of procedures, and homogeneity of the subjects.

 (may need to control for this potential threat)

The extent to which the measure assesses the characteristics of interest in a consistent fashion.

The more (statistical) tests that are performed, the more likely a chance difference will be found, even if there are no true differences between conditions. 

Statistical Conclusion Validity: Subject Heterogeneity

Dimensions and characteristics along which subjects can vary (ie. gender, background, race and ethnicity, marital status).

These factors may influence outcome---there will be greater variability in the subjects' reactions to measures and to the intervention (decreases effect size).

6 Sources of bias...

(problems with internal, construct, external validity)

Sources of bias: "Loose protocol effect"

The failure to specify in detail the rationales, script, and activities of the experimentor.

- procedures should be explicit and standardized for the experimentor

- procedure can be recorded (audio/video) in advance

Sources of bias: Experimentor expectancy effects

Influence of the experimentor's beliefs and desires about the results on how the subject performs. The effects are considered to be unintentional because the experimentor may not do anything on purpose to influence subjects' responses.

- keep experimentors naive/blind with respect to the purpose of the experiment and to evaluate the extent to which this is accomplished.

Sources of bias: Experimentor characteristics

...such as age, race, gender, ethnicity, level of anxiety, friendliness, and prestige affect responses given by subjects on self-report and projective tests, measures of intelligence, and lab tasks.

- Useful for investigators to more carefully specify the characteristics of experimentors in their research reports.

Sources of bias: Situational & contextual cues / Demand characteristics

 Cues in the experimental situation that may influence how subjects respond (ie. convincing therapist, coherent tx rationale, appearance of qualified and competent professional-->subjects expect change)

 Evaluating this effect:

- post-experimental inquiry: ask subjects about the purpose of the experiment and how they thought they were "supposed" to act

- preinquiry: subjects asked to imagine themselves in the experimental situation and then respond to measures

- simulators: subjects asked to act as if they receive the experimental condition to try and deceive naive experimenter

Sources of bias: Subject roles

(artifact)

Subjects respond to cues in such a way as to give the experimentor what they want/don't want.

(Subjects become reactive to themselves as participants; behave in perceived "desired"/"undesired" way).

- inform subjects that there are no right/wrong answers, and their resonses are valuable no matter what.

Sources of bias: Data recording, analysis, and fabrication

Errors calculating, computing, or fabricating the data. Errors tend to be in favor of the experimentor's hypotheses.

- individuals recording data should be uninformed of hypotheses

- when possible have subjects enter responses onto a computer

2 Subject-selection biases...

(1) S- The Sample

(2) A - Attrition

Subject-selection biases: The Sample

(Who was selected for the study)

Is it a sample of convenience? Is the range of subjects sampled restricted?

- structuring the situation so that the researcher fosters greater participation and serves a need or interest of the subjects is likely to increase # of volunteers

Subject-selection biases: Attrition

(Who remains in the study)

Loss of subjects during the course of an investigation can effect all facets.

- interviews, mailings, reminders, and monetary rewards have been effective

Random Selection

Important because:

Q: Why is random selection important to consider who is selected to participate in a research study and why they participate?

3 conditions under which it is acceptable to have a restricted sample...

(1)

(2)

(3)

Sample of convenience

Random assignment

Important because: 

Group equivalence

(Q: How can a researcher increase its likelihood?)

Matching

(Q: What is it and how is it done?)

Independent Variable

Dependent variable

Alternative hypothesis