Experimental Design Checklist

Research Question Formulation

Is the Question Well-Formed?

• Specific: Clearly defined variables and relationships
• Answerable: Can be addressed with available methods
• Relevant: Addresses a gap in knowledge or practical need
• Feasible: Resources, time, and ethical considerations allow it
• Falsifiable: Can be proven wrong if incorrect

Have You Reviewed the Literature?

• Identified what's already known
• Found gaps or contradictions to address
• Learned from methodological successes and failures
• Identified appropriate outcome measures
• Determined typical effect sizes in the field

Hypothesis Development

Is Your Hypothesis Testable?

• Makes specific, quantifiable predictions
• Variables are operationally defined
• Specifies direction/nature of expected relationships
• Can be falsified by potential observations

Types of Hypotheses

• Null hypothesis (H₀): No effect/relationship exists
• Alternative hypothesis (H₁): Effect/relationship exists
• Directional vs. non-directional: One-tailed vs. two-tailed tests

Study Design Selection

What Type of Study is Appropriate?

Experimental (Intervention) Studies:

• Randomized Controlled Trial (RCT): Gold standard for causation
• Quasi-experimental: Non-random assignment but manipulation
• Within-subjects: Same participants in all conditions
• Between-subjects: Different participants per condition
• Factorial: Multiple independent variables
• Crossover: Participants receive multiple interventions sequentially

Observational Studies:

• Cohort: Follow groups over time
• Case-control: Compare those with/without outcome
• Cross-sectional: Snapshot at one time point
• Ecological: Population-level data

Consider:

• Can you randomly assign participants?
• Can you manipulate the independent variable?
• Is the outcome rare (favor case-control) or common?
• Do you need to establish temporal sequence?
• What's feasible given ethical, practical constraints?

Variables

Independent Variables (Manipulated/Predictor)

• Clearly defined and operationalized
• Appropriate levels/categories chosen
• Manipulation is sufficient to test hypothesis
• Manipulation check planned (if applicable)

Dependent Variables (Outcome/Response)

• Directly measures the construct of interest
• Validated and reliable measurement
• Sensitive enough to detect expected effects
• Appropriate for statistical analysis planned
• Primary outcome clearly designated

Control Variables

• Confounding variables identified:
  • Variables that affect both IV and DV
  • Alternative explanations for findings
• Strategy for control:
  • Randomization
  • Matching
  • Stratification
  • Statistical adjustment
  • Restriction (inclusion/exclusion criteria)
  • Blinding

Extraneous Variables

• Potential sources of noise identified
• Standardized procedures to minimize
• Environmental factors controlled
• Time of day, setting, equipment standardized

Sampling

Population Definition

• Target population: Who you want to generalize to
• Accessible population: Who you can actually sample from
• Sample: Who actually participates
• Difference between these documented

Sampling Method

• Probability sampling (preferred for generalizability):
  • Simple random sampling
  • Stratified sampling
  • Cluster sampling
  • Systematic sampling
• Non-probability sampling (common but limits generalizability):
  • Convenience sampling
  • Purposive sampling
  • Snowball sampling
  • Quota sampling

Sample Size

• A priori power analysis conducted
  • Expected effect size (from literature or pilot)
  • Desired power (typically .80 or .90)
  • Significance level (typically .05)
  • Statistical test to be used
• Accounts for expected attrition/dropout
• Sufficient for planned subgroup analyses
• Practical constraints acknowledged

Inclusion/Exclusion Criteria

• Clearly defined and justified
• Not overly restrictive (limits generalizability)
• Based on theoretical or practical considerations
• Ethical considerations addressed
• Documented and applied consistently

Blinding and Randomization

Randomization

• What is randomized:
  • Participant assignment to conditions
  • Order of conditions (within-subjects)
  • Stimuli/items presented
• Method of randomization:
  • Computer-generated random numbers
  • Random number tables
  • Coin flips (for very small studies)
• Allocation concealment:
  • Sequence generated before recruitment
  • Allocation hidden until after enrollment
  • Sequentially numbered, sealed envelopes (if needed)
• Stratified randomization:
  • Balance important variables across groups
  • Block randomization to ensure equal group sizes
• Check randomization:
  • Compare groups at baseline
  • Report any significant differences

Blinding

• Single-blind: Participants don't know group assignment
• Double-blind: Participants and researchers don't know
• Triple-blind: Participants, researchers, and data analysts don't know
• Blinding feasibility:
  • Is true blinding possible?
  • Placebo/sham controls needed?
  • Identical appearance of interventions?
• Blinding check:
  • Assess whether blinding maintained
  • Ask participants/researchers to guess assignments

Control Groups and Conditions

What Type of Control?

• No treatment control: Natural course of condition
• Placebo control: Inert treatment for comparison
• Active control: Standard treatment comparison
• Wait-list control: Delayed treatment
• Attention control: Matches contact time without active ingredient

Multiple Conditions

• Factorial designs for multiple factors
• Dose-response relationship assessment
• Mechanism testing with component analyses

Procedures

Protocol Development

• Detailed, written protocol:
  • Step-by-step procedures
  • Scripts for standardized instructions
  • Decision rules for handling issues
  • Data collection forms
• Pilot tested before main study
• Staff trained to criterion
• Compliance monitoring planned

Standardization

• Same instructions for all participants
• Same equipment and materials
• Same environment/setting when possible
• Same assessment timing
• Deviations from protocol documented

Data Collection

• When collected:
  • Baseline measurements
  • Post-intervention
  • Follow-up timepoints
• Who collects:
  • Trained researchers
  • Blinded when possible
  • Inter-rater reliability established
• How collected:
  • Valid, reliable instruments
  • Standardized administration
  • Multiple methods if possible (triangulation)

Measurement

Validity

• Face validity: Appears to measure construct
• Content validity: Covers all aspects of construct
• Criterion validity: Correlates with gold standard
  • Concurrent validity
  • Predictive validity
• Construct validity: Measures theoretical construct
  • Convergent validity (correlates with related measures)
  • Discriminant validity (doesn't correlate with unrelated measures)

Reliability

• Test-retest: Consistent over time
• Internal consistency: Items measure same construct (Cronbach's α)
• Inter-rater reliability: Agreement between raters (Cohen's κ, ICC)
• Parallel forms: Alternative versions consistent

Measurement Considerations

• Objective measures preferred when possible
• Validated instruments used when available
• Multiple measures of key constructs
• Sensitivity to change considered
• Floor/ceiling effects avoided
• Response formats appropriate
• Recall periods appropriate
• Cultural appropriateness considered

Bias Minimization

Selection Bias

• Random sampling when possible
• Clearly defined eligibility criteria
• Document who declines and why
• Minimize self-selection

Performance Bias

• Standardized protocols
• Blinding of providers
• Monitor protocol adherence
• Document deviations

Detection Bias

• Blinding of outcome assessors
• Objective measures when possible
• Standardized assessment procedures
• Multiple raters with reliability checks

Attrition Bias

• Strategies to minimize dropout
• Track reasons for dropout
• Compare dropouts to completers
• Intention-to-treat analysis planned

Reporting Bias

• Preregister study and analysis plan
• Designate primary vs. secondary outcomes
• Commit to reporting all outcomes
• Distinguish planned from exploratory analyses

Data Management

Data Collection

• Data collection forms designed and tested
• REDCap, Qualtrics, or similar platforms
• Range checks and validation rules
• Regular backups
• Secure storage (HIPAA/GDPR compliant if needed)

Data Quality

• Real-time data validation
• Regular quality checks
• Missing data patterns monitored
• Outliers identified and investigated
• Protocol deviations documented

Data Security

• De-identification procedures
• Access controls
• Audit trails
• Compliance with regulations (IRB, HIPAA, GDPR)

Statistical Analysis Planning

Analysis Plan (Prespecify Before Data Collection)

• Primary analysis:
  • Statistical test(s) specified
  • Hypothesis clearly stated
  • Significance level set (usually α = .05)
  • One-tailed or two-tailed
• Secondary analyses:
  • Clearly designated as secondary
  • Exploratory analyses labeled as such
• Multiple comparisons:
  • Adjustment method specified (if needed)
  • Primary outcome protects from inflation

Assumptions

• Assumptions of statistical tests identified
• Plan to check assumptions
• Backup non-parametric alternatives
• Transformation options considered

Missing Data

• Anticipated amount of missingness
• Missing data mechanism (MCAR, MAR, MNAR)
• Handling strategy:
  • Complete case analysis
  • Multiple imputation
  • Maximum likelihood
• Sensitivity analyses planned

Effect Sizes

• Appropriate effect size measures identified
• Will be reported alongside p-values
• Confidence intervals planned

Statistical Software

• Software selected (R, SPSS, Stata, Python, etc.)
• Version documented
• Analysis scripts prepared in advance
• Will be made available (Open Science)

Ethical Considerations

Ethical Approval

• IRB/Ethics committee approval obtained
• Study registered (ClinicalTrials.gov, etc.) if applicable
• Protocol follows Declaration of Helsinki or equivalent

Informed Consent

• Voluntary participation
• Comprehensible explanation
• Risks and benefits disclosed
• Right to withdraw without penalty
• Privacy protections explained
• Compensation disclosed

Risk-Benefit Analysis

• Potential benefits outweigh risks
• Risks minimized
• Vulnerable populations protected
• Data safety monitoring (if high risk)

Confidentiality

• Data de-identified
• Secure storage
• Limited access
• Reporting doesn't allow re-identification

Validity Threats

Internal Validity (Causation)

• History: External events between measurements
• Maturation: Changes in participants over time
• Testing: Effects of repeated measurement
• Instrumentation: Changes in measurement over time
• Regression to mean: Extreme scores becoming less extreme
• Selection: Groups differ at baseline
• Attrition: Differential dropout
• Diffusion: Control group receives treatment elements

External Validity (Generalizability)

• Sample representative of population
• Setting realistic/natural
• Treatment typical of real-world implementation
• Outcome measures ecologically valid
• Time frame appropriate

Construct Validity (Measurement)

• Measures actually tap intended constructs
• Operations match theoretical definitions
• No confounding of constructs
• Adequate coverage of construct

Statistical Conclusion Validity

• Adequate statistical power
• Assumptions met
• Appropriate tests used
• Alpha level appropriate
• Multiple comparisons addressed

Reporting and Transparency

Preregistration

• Study preregistered (OSF, ClinicalTrials.gov, AsPredicted)
• Hypotheses stated a priori
• Analysis plan documented
• Distinguishes confirmatory from exploratory

Reporting Guidelines

• RCTs: CONSORT checklist
• Observational studies: STROBE checklist
• Systematic reviews: PRISMA checklist
• Diagnostic studies: STARD checklist
• Qualitative research: COREQ checklist
• Case reports: CARE guidelines

Transparency

• All measures reported
• All manipulations disclosed
• Sample size determination explained
• Exclusion criteria and numbers reported
• Attrition documented
• Deviations from protocol noted
• Conflicts of interest disclosed

Open Science

• Data sharing planned (when ethical)
• Analysis code shared
• Materials available
• Preprint posted
• Open access publication when possible

Post-Study Considerations

Data Analysis

• Follow preregistered plan
• Clearly label deviations and exploratory analyses
• Check assumptions
• Report all outcomes
• Report effect sizes and CIs, not just p-values

Interpretation

• Conclusions supported by data
• Limitations acknowledged
• Alternative explanations considered
• Generalizability discussed
• Clinical/practical significance addressed

Dissemination

• Publish regardless of results (reduce publication bias)
• Present at conferences
• Share findings with participants (when appropriate)
• Communicate to relevant stakeholders
• Plain language summaries

Next Steps

• Replication needed?
• Follow-up studies identified
• Mechanism studies planned
• Clinical applications considered

Common Pitfalls to Avoid

• No power analysis → underpowered study
• Hypothesis formed after seeing data (HARKing)
• No blinding when feasible → bias
• P-hacking (data fishing, optional stopping)
• Multiple testing without correction → false positives
• Inadequate control group
• Confounding not addressed
• Instruments not validated
• High attrition not addressed
• Cherry-picking results to report
• Causal language from correlational data
• Ignoring assumptions of statistical tests
• Not preregistering changes literature bias
• Conflicts of interest not disclosed

Final Checklist Before Starting

• Research question is clear and important
• Hypothesis is testable and specific
• Study design is appropriate
• Sample size is adequate (power analysis)
• Measures are valid and reliable
• Confounds are controlled
• Randomization and blinding implemented
• Data collection is standardized
• Analysis plan is prespecified
• Ethical approval obtained
• Study is preregistered
• Resources are sufficient
• Team is trained
• Protocol is documented
• Backup plans exist for problems

Remember

Good experimental design is about:

• Asking clear questions
• Minimizing bias
• Maximizing validity
• Appropriate inference
• Transparency
• Reproducibility

The best time to think about these issues is before collecting data, not after.