A/B testing with BayesAB

Assume we have an A/B testing data set here. It is about the conversion rate of the same landing page related to two posts on Facebook with different head pics.

Let us take a look at a quick F-test example.


library(bayesAB)

A_sample<-c(4,96)
B_sample<-c(45,1280)

fisher.test(data.frame(A_sample,B_sample))

##
##  Fisher's Exact Test for Count Data
##
## data:  data.frame(A_sample, B_sample)
## p-value = 0.7727
## alternative hypothesis: true odds ratio is not equal to 1
## 95 percent confidence interval:
##  0.3031585 3.3528400
## sample estimates:
## odds ratio
##   1.185033

The test said that this is no significantly difference. We launched a power test here to check the sample size we need.


power.prop.test(p1=4/96,p2=45/1280,power=0.8,sig.level = 0.05)
##
##      Two-sample comparison of proportions power calculation
##
##               n = 13678.32
##              p1 = 0.04166667
##              p2 = 0.03515625
##       sig.level = 0.05
##           power = 0.8
##     alternative = two.sided
##
## NOTE: n is number in *each* group

To make a significant comparison, we need 13678*2 observations totally.

Oftentimes, performance manager who conducts A/B test will have to do a power test at first to determine sample size, and then interface with a Product Manager to explain the final result, which can be tricky in terms of interpretability.

ab-infographic.png

Advantages of BayesAB testing

“P(A > B) is 8%”, or “When the null hypothesis that A and B are equal is true, the probability that we have confidence to reject that A and B are equal is equal to 3%”. I guess which one is easy to understand is straightforward.

An informative prior of  bayesAB function can help us avoid many common issues in regular A/B testing without power test.

Furthermore, an informative prior will help with the low base-rate problem (when the probability of a success or observation is very low). By indicating this in your priors, your posterior distribution will be far more stable right from the onset.


A_s<-rbinom(96,1,4/96)
B_s<-rbinom(1280,1,45/1280)
AB1 <- bayesTest(A_s, B_s,
priors = c('alpha' = 1, 'beta' = 1),
distribution = 'bernoulli')
summary(AB1)

## Quantiles of posteriors for A and B:
##
## $Probability
## $Probability$A_probs
##           0%          25%          50%          75%         100%
## 0.0008017126 0.0348436245 0.0480286692 0.0639101508 0.1990413449
##
## $Probability$B_probs
##         0%        25%        50%        75%       100%
## 0.01738483 0.03152084 0.03488182 0.03840652 0.06333131
##
##
## --------------------------------------------
##
## P(A > B) by (0)%:
##
## $Probability
## [1] 0.7414
##
## --------------------------------------------
##
## Credible Interval on (A - B) / B for interval length(s) (0.9) :
##
## $Probability
##         5%        95%
## -0.4267753  1.7670381
##
## --------------------------------------------
##
## Posterior Expected Loss for choosing B over A:
##
## $Probability
## [1] 0.009537639

We can plot the result with plot(AB1):

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s