Frequentist Calculator
Calculate your A/B test results using the frequentist approach
How to Use This A/B Test Calculator
- Enter the number of participants for Variation A and B in the respective fields.
- Input the number of conversions for each variation.
- The calculator will automatically update the results as you enter the data.
- Review the Executive Summary for a quick overview of your test results.
- Examine the Conversion Rate Distribution and Improvement Distribution charts for visual representation.
- Check the Additional Insights section for more detailed statistical information.
Key Terms and Definitions
- Conversion Rate
- The percentage of participants who complete a desired action (e.g., making a purchase).
- Z-Score
- A measure of how many standard deviations below or above the population mean a raw score is.
- P-Value
- The probability of obtaining test results at least as extreme as the observed results, assuming the null hypothesis is true.
- Confidence Level
- The probability that the true value of a parameter falls within a specified range of values.
- Statistical Significance
- The likelihood that a relationship between two or more variables is caused by something other than chance.
- Observed Power
- The probability of detecting a statistically significant difference when there is a true difference between the variations. It indicates the test's ability to avoid Type II errors (false negatives).
- False Positive Rate (Type I Error)
- The probability of rejecting the null hypothesis when it is actually true. It's typically set at 5% (0.05) for most statistical tests.
- False Negative Rate (Type II Error)
- The probability of failing to reject the null hypothesis when it is actually false. It's related to the statistical power of the test.
- Sample Ratio Mismatch (SRM)
- A check to ensure that the traffic split between variations is as expected. An SRM indicates a potential issue with the randomization process in an A/B test.