A/B Testing 101 in R

A/B testing is a popular method for optimizing online content such as landing pages, marketing campaigns, and apps. Hoping to become more familiar with this concept and its applications, I recently finished a DataCamp course regarding A/B testing, and this article is to document some useful functions for future references.

The data I used for this article was adapted from the dataset provided by the course, which records visiters' interactions with a website following one simple word change to the site's title — from "Tools for Better Data Viz" into "Tips for Better Data Viz". First, let's do a quick summary of the data, and visualize the results using ggplot2 for easier interpretation.

From the plots, we can clearly see the difference in proportion of visitors that clicked like under the two conditions, but how significant is the difference exactly? Moreover, are the other two variables truly unaffected by the change? We can answer these questions using the following techniques.

Statistical Tests

For continuous dependent variables, we can use simple t-tests or linear regression to help us calculate the significance of the change. T-tests are only viable for testing when independent variables have only two factor levels, while linear regressions can be used with more flexibility. In our case, both methods will provide the same results. Just as we expected, the change has no effect on the time spent on the homepage.

As for binary variables (clicked or not), we can utilize logistic regressions to test the significance. Logistic regressions can be applied by calling the glm function in R and setting the "family" parameter as "binomial". As shown in the tidied summaries down below, the proportion of visitors that clicked like significantly increased under the title "Tips for Data Viz".

Power Analyses

Before conducting experiments, it's crucial to predetermine the required sample size for our test, as well as the appropriate power, alpha, and effect size. In the DataCamp course, the instructor introduced two functions for power analyses — "sSizeLogisticBin" in the powerMediation package, and "pwr.t.test" in pwr.

As its name implies, "sSizeLogisticBin" is used specifically for logistic regressions with binary predictors. "p1" is the initial value for the current control condition, "p2" is the expected value expected value for the test condition, while the "B" parameter stands for the proportion of the data from the test condition, which is ideally 0.5.

On the other hand, "pwr.t.test" is very useful for conducting t-tests, mostly used for computing the power and the proper sample size of tests. Here, I used the conventional value for small effect sizes, which is 0.2. Normally, this value would be determined by calculating the mean difference between two groups, and then dividing the result by the pooled standard deviation. With just a simple line of code, we can obtain lots of information regarding our t-tests. One thing worth noting though, is that the "n" displayed in the output means the number of observations for "each group", and the default type of t-test is a two-sample test with a two-sided alternative hypothesis. This of course, can also be easily modified by tweaking the arguments.

Conclusion

A/B testing may seem pretty simple on paper, but there are many different variations we can play around with using R, so it still requires a lot of practice to become a pro. Hopefully, by digging deeper into statistical inferences, I'll be able to deploy A/B tests with more finesse, and continue striving to become a more polished business analyst!