In this tutorial, we explore how to calculate Minimum Detectable Effect (MDE) using Python. This guide introduces you to the concept of MDE in A/B testing, providing a step-by-step approach with code examples, making it easy to determine the smallest change that can be statistically detected in your experiments.
Minimum-Detectable-Effect-(MDE)-Using-Python
The Approach can be discussed below:
Step 1: Import Libraries
import numpy as np from statsmodels.stats.power import TTestIndPower
Step 2: Set Parameters and Take user input for sample size
# Step 1: Set Parameters
baseline_rate = 0.1 # Example: 10% conversion rate
alpha = 0.05 # Significance level (typically 0.05)
power = 0.8 # Desired power (typically 0.8)
# Take user input for sample size
sample_size = int(input("Enter the total sample size: "))
The parameters needed to calculate the MDE are set by the code. Baseline_rate is the current conversion rate, power is the intended likelihood of detecting an impact if one exists, and alpha is the significance threshold. The line of code prompts the user to enter a value for the total sample size, `sample_size`. This value represents the total number of observations or participants in the A/B test.
Step 3: Initialize Power Analysis
analysis = TTestIndPower()
Using the ‘ statsmodels.stats.power ‘ module, this line of code builds an instance of the ‘ TTestIndPower class ‘, which is then stored in the variable analysis.
Step 4: Calculate Effect Size
effect_size = analysis.solve_power(effect_size=None, nobs1=sample_size/2, alpha=alpha, power=power, alternative='two-sided')
Using a two-sample t-test, The following arguments are used with the analysis instance’s solve_power method: effect_size=None, nobs1=sample_size/2, alpha=alpha, power=power, and alternative=’two-sided’. The effect_size variable holds the outcome.
Step 5: Convert Effect Size to MDE
mde = effect_size * np.sqrt((baseline_rate * (1 - baseline_rate)) * (2/sample_size))
Using the previously determined effect size, The formula takes the sample size ({2/sample_size}) and multiplies {effect_size} by the square root of the variance of the baseline conversion rate ({baseline_rate * (1 – baseline_rate)}). The result, which is kept in {mde}, is the least conversion rate difference that the test can consistently identify.
Step 6: Print MDE
print(f"Minimum Detectable Effect (MDE): {mde * 100:.2f}%")
Enter the total sample size: 2000 Minimum Detectable Effect (MDE): 0.12%
Full Code:
import numpy as np
from statsmodels.stats.power import TTestIndPower
# Step 1: Set Parameters
baseline_rate = 0.1 # Example: 10% conversion rate
alpha = 0.05 # Significance level (typically 0.05)
power = 0.8 # Desired power (typically 0.8)
# Take user input for sample size
sample_size = int(input("Enter the total sample size: "))
# Step 2: Initialize Power Analysis
analysis = TTestIndPower()
# Step 3: Calculate Effect Size
effect_size = analysis.solve_power(effect_size=None, nobs1=sample_size/2, alpha=alpha, power=power, alternative='two-sided')
# Step 4: Convert Effect Size to MDE
mde = effect_size * np.sqrt((baseline_rate * (1 - baseline_rate)) * (2/sample_size))
# Step 5: Print MDE
print(f"Minimum Detectable Effect (MDE): {mde * 100:.2f}%")
Enter the total sample size: 2000 Minimum Detectable Effect (MDE): 0.12%