Outline

Causal Inference

• Bayesian vs. Frequentist

Basic Probability and Statistics

• Probability Theory
• Potential Outcomes Framework
• Graphical Models
• False Positives and False Negatives
• P-Values
• Power
• Confidence Interval
• CTR Variance

Designing Experiments

• Why experiment?
• Landscape
• Product Development Cycle
• Measuring Impact
• What does the data look like
• How to randomize
• What could go wrong
• What to measure
• How to measure
• Compare
• Design and Monitor
• Interpret and Recommend Action
• A/A Test

Overall Evaluation Criteria (OEC)

Multiple Testing

• False Positives and Multiple Testing
• Omnibus test

Tiered Metrics

Central Limit Theorem and Handling Violated Assumptions

• When not to A/B Test
• What to do when you cannot A/B test
• How to check assumptions
• Bootstrap and Delta Method
• One-sided and Two Sided tests
• High Skew Remedies
• Small Samples
• Data Quality Checks

Sequential Tests & Variance Reduction

• Peeking and Sequential Testing
• Getting Results Faster
• Variance Reduction
• CUPED (vs Normal AB) and Crossover
• Fast Surrogates
• Pre-experiment Imbalance

Multi-armed Bandits

• vs. AB
• vs. Contextual Bandits

Marketplace and Network Effects