Challenges of choosing metrics in data science interviews: actionability vs statistical power in marketplaces

In marketplace experiments, the most actionable metrics are often the ones with more variance, making them harder to power. This tradeoff is particularly important when using geo-based experiments where you can't necessarily or cheaply add sample. Suppose Airbnb wants to test highlighting cheaper listings; this should increase bookings while likely reducing AOV. The net effect of a higher conversion rate and reduced AOV depends on elasticities and profit margins. Revenue or contribution margin are the metrics that matter the most (should be the most actionable), but these have far more varianc