FREE Resources on Experimentation for Two-sided Marketplace

I have experienced the pain in developing and operating three two-sided platforms in the space of integrative and natural medicine, and the thrill in witnessing an advised family e-commerce brand grow on two-sided platforms during the “Platform Revolution” and roller-coaster ride during the pandemic. However, this post will be solely focus on the experimental aspects on such platforms from a researcher or data scientist or engineer or economist’s angle.

A list of talks on this topic I watched with paper references FYI:

1. “Experimental Design in two-sided platforms: An Analysis of Bias” by Ramesh Johari. Experiments in marketplaces (a.k.a. platform, we had the “platform revolution”) have the issue of interference, when facing demand and supply imbalance on the platform (e.g., competition between listings), leading to biased estimates, with potential bias as much as the treatment effect. Prof. Johari introduced a two-sided randomization design and associated estimators that reduce bias in large market settings, simulations suggest the approach offer bias reduction without significantly increase variance. Check out the proposed framework below:

From ~40 minutes onwards, Prof. Johari showed a theorem with intuition, explaining that in highly demand-constrained markets, CR is unbiased, while LR is biased; conversely, in highly supply-constrained markets, LR is unbiased, while CR is biased. Moreover, by a design based on two-sided randomization (TSR) where both customers and listings are randomized to treatment and control, appropriate choices of such designs can be unbiased in both extremes of market balance, and yield low bias in intermediate regimes of market balance.

Note that the same speaker and one of the authors, Prof. Johari had an another talk on quality selection in two-sided markets earlier in 2019 here.

2. Another of his talks on interference in experimental design in online platforms can be listened here:

Many experiments (“A/B tests”) in online platforms exhibit interference, where an intervention applied to one market participant influences the behavior of another participant. This interference can lead to biased estimates of the treatment effect of the intervention. In this talk the speaker and his teammates first focused on such experiments in two-sided platforms where “customers” book “listings”. They developed a stochastic market model and associated mean field limit to capture dynamics in such experiments, and used their model to investigate how the performance of different designs and estimators is affected by marketplace interference effects.

3. Paper “Experimental Design in Two-Sided Platforms: An Analysis of Bias” can be accessed from here. The corresponding paper presentation at EC ’20: Proceedings of the 21st ACM Conference on Economics and Computation Virtual Conference in July 2020 can be watched here:

1-minute flash video of the presentation

4. “Experimentation and Interference in a Two-Sided Marketplace”, by Nick Chamandy from Lyft using examples in a ridesharing marketplace, the video of his talks on YouTube is shown here and below:

5. If you are interested in any technical development and trends over time like me, please feel free to time travel back over a decade earlier, listen to the same speaker in the first few videos and co-author of the paper talked about designing online market platforms in 11 minutes:

6. “Detecting Interference in A/B Testing with Increasing Allocation” with initial 2022 version here and 2023 version published in digital library of ACM at https://dl.acm.org/doi/pdf/10.1145/3580305.3599308.

A few basic reads:

7. Multi-armed Bandit Experiments describes the statistical engine behind Google Analytics Content Experiments, a new version of the official blog of Google Marketing Platform can be found here.

8. “Beyond A/B Testing: Multi-armed Bandit Experiments, a study of Google Analytics’ stochastic k-armed bandit test with Thompson sampling and Monte Carlo simulation. Check out “opposing view” here: Why multi-armed bandit algorithm is not “better” than A/B testing. Stitch fix has a blog on this “Multi-Armed Bandits and the Stitch Fix Experimentation Platform” here, too.