Sample Ratio Mismatch Calculator

1. What is Sample Ratio Mismatch (SRM)?
2. Using the Sample Ratio Mismatch calculator
3. What if SRM is detected?

    What is Sample Ratio Mismatch (SRM)?

Sample Ratio Mismatch (SRM) is a statistically significant discrepancy between the target sample ratios between test groups and their observed sample ratios ^[1]. It is a term most frequently used in online controlled experiments a.k.a. online A/B testing. For example, an A/B test with target sample ratios 0.5 0.5 (50% expected in each test group) in which the data shows 10,000 users in the control and 11,000 in the treatment group exhibits very strong sample ratio mismatch as can be verified using this sample ratio mismatch calculator.

The presence of sample ratio mismatch almost surely signifies that the test is biased either for or against the control and/or one or more of the variants. It means the results are untrustworthy and should not be acted upon. Upon discovering SRM in an experiment the results will likely need to be discarded and the reason for the SRM found and dealt with before the A/B test can be relaunched. Despite the sample ratio mismatch check being a simple application of the chi-square goodness-of-fit test, discovering the root cause and even identifying which way the bias goes is no simple matter. More on SRM debugging below.

Note that even if a test passes a sample ratio mismatch checker it doesn't mean the experiment has no biasing issues. It is entirely possible that a bias is present, but it was too small and the chi square test simply doesn't have enough statistical power to pick it up with the required level of certainty ^[1].

    Using the Sample Ratio Mismatch calculator

To use this checker, all you need to know is the sample size of each test group, control included, and their target sample ratios - the ratios you'd expect to see absent any random fluctuation in the assignment. The sample size would usually be the number of users, sessions, pageviews, impressions. It should be the unit you randomize on. Enter the sample size in the 'Sample sizes' field of the sample ratio mismatch calculator above.

If the target allocation is equal among all groups (usually the most efficient approach) simply leave the target allocation radio button with "Equal" selected. Otherwise switch to "Unequal" and then proceed to enter the sample ratios either as fractions (e.g. 0.4 0.2 0.2 0.2) or percentages (e.g. 40% 20% 20% 20%). Note that these should be provided in the same order as the sample sizes above and that they should sum to 1 (100%). Our calculator will alert you to it if they do not.

The delimiter in both the sample size and sample ratio fields is space or new line, so copy/pasting data from a spreadsheet file into the calculator should work just fine. Thousand separator commas will be automatically stripped, so no need to worry about them.

    What if SRM is detected?

While some random variability in the observed allocation between test groups is to be expected, if the p-value produced by our SRM calculator is low enough it can be treated as meaning that the SRM check has failed and the test data cannot be trusted. How low a p-value is low enough may vary from test to test, but usually it has to be lower than 0.05 and oftentimes 0.01 before a sample ratio mismatch is considered statistically significant. Our calculator uses a threshold of 0.01, but also reports the p-value so you can adopt your own threshold as necessary.

Once a mismatch is detected, the next step is to try and identify the root cause of the bias. This is a difficult process of stepwise elimination which doesn't always come to fruition. There is no single way to address a detected mismatch as each testing setup is unique and thus faces unique challenges and potential issues. The debugging capabilities may be more or less limited depending on what the A/B testing platform supports.

Here is a list with the main types of possible causes for an observed sample ratio mismatch in an online A/B test following loosely Fabijan et all ^[2]. It should be a good starting point on where to find the culprit if our sample ratio mismatch calculator detected the presence of SRM.

• Group assignment bias (technical issues other than the randomization device, e.g. misconfiguration forcing users into a test group without randomization)
• Randomizer bias
• Experiment execution bias (variant delivery, experiment initiation issues)
• Telemetry bias (e.g. biased loss or delay of the data for one or more test groups)
• Experiment log processing bias (bias in data processing post-collection)

This list is by no means comprehensive but should be a useful starting point in debugging a sample ratio mismatch. Fields other than online A/B testing where SRM checks are relevant will need to find other resources.