## Tuesday, 28 January 2014

### Binomial testing with buttered toast

Rasmus' post of last week on binomial testing made me think about p-values and testing again. In my head I was tossing coins, thinking about gender diversity and toast. The toast and tossing a buttered toast in particular was the most helpful thought experiment, as I didn't have a fixed opinion on the probabilities for a toast to land on either side. I have yet to carry out some real experiments.

Suppose I tossed 6 buttered toasts and observed that all but one toast landed on their buttered side.

Now I have two questions:
1. Would it be reasonable to assume that there is a 50/50 chance for a toast to land on either side?
2. Which probability should I assume?
Suppose the toast is fair, so that the probability for landing on the buttered (B) or unbuttered (U) side is 50%.

Then, the probability of observing one ore more B (right tail event) is:
(gt <- 1-1/2^6)
# [1] 0.984375

and the probability of observing one or fewer B (left tail event) is:
(lt <- 1/2^6*choose(6,1) + 1/2^6)
# [1] 0.109375

while the probability of either extreme event, one or fewer B (or U), is:
2*min(c(gt, lt))
# [1] 0.21875

In summary, if the toast has an equally probability to fall on either side, then there is 22% chance to observe one or fewer B (or U) in 6 tosses. That's not that unlikely and hence I would not dismiss the hypothesis that the toast is fair, despite the fact that the sample frequency is only 1/6.

Actually, the probabilities I calculated above are exactly the p-values I get from the classical binomal tests:
## Right tail event
binom.test(1, 6, alternative="greater")
## Left tail event
binom.test(1, 6, alternative="less")
## Double tail event
binom.test(1, 6, alternative="two.sided")
Additionally I can read from the tests that my assumption of a 50% probability is on the higher end of the 95 percent confidence interval. Thus, wouldn't it make sense to update my belief about the toast following my observations? In particular, as unlike with a coin, I am far less convinced that a 50/50 probability is a good assumption to start with. Arguably the toast is biased by the butter.

Here the concept of a conjugate prior becomes handy again. The idea is to assume that the parameter $\theta$ of the binomial distribution is a random variable itself. Suppose I have no prior knowledge about the true probability of the toast falling on either side, then a flat prior, such as the Uniform distribution would be reasonable. However, the beta distribution with parameter $\alpha=1$ and $\beta=1$ has the same property and is a conjugate to the binomial distribution with parameter $\theta$. That means there is an analytical solution, in this case the posterior distribution is beta-binomial with hyperparaemters:
$\alpha':=\alpha + \sum_{i=1}^n x_i,\; \beta':=\beta + n - \sum_{i=1}^n x_i,$
and the posterior predictor for one trial is given as
$\frac{\alpha'}{\alpha' + \beta'}$
so in my case:
alpha <- 1; beta <- 1; n <- 6; success <- 1
alpha1 <- alpha + success
beta1 <- beta + n - success
(theta <- alpha1 / ( alpha1 + beta1))
# [1] 0.25
My updated believe about the toast landing on the unbuttered side is a probability of 25%. That's lower than my prior of 50% but still higher than the sample frequency of 1/6. If I would have more toasts I could run more experiments and update my posterior predictor.

I get the same answer from Rasmus' bayes.binom.test function:
> library(BayesianFirstAid)
> bayes.binom.test(1, 6)

Bayesian first aid binomial test

data: 1 and 6
number of successes = 1, number of trials = 6
Estimated relative frequency of success:
0.25
95% credible interval:
0.014 0.527
The relative frequency of success is more than 0.5 by a probability of 0.061
and less than 0.5 by a probability of 0.939

Of course I could change my view on the prior and come to a different conclusion. I could follow the Wikipedia article on buttered toast and believe that the chance of the toast landing on the buttered side is 62%. I further have to express my uncertainty, say a standard deviation of 10%, that is a variance of 1%. With that information I can update my belief of the toast landing on the unbuttered side following my observations (and transforming the variables):
x <- 0.38
v <- 0.01
alpha <- x*(x*(1-x)/v-1)
beta <- (1-x)*(x*(1-x)/v-1)
alpha1 <- alpha + success
beta1 <- beta + n - success
(theta <- alpha1 / ( alpha1 + beta1))
# [1] 0.3351821
I would conclude that for my toasts / tossing technique the portability is 34% to land on the unbuttered side.

In summary, although their is no sufficient evidence to reject the hypothesis that the 'true' probability is not 50% (at the typical 5% level), I would work with 34% until I have more data. Toast and butter please!

### Session Info

R version 3.0.1 (2013-05-16)
Platform: x86_64-apple-darwin10.8.0 (64-bit)

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods
[7] base

other attached packages:
[1] BayesianFirstAid_0.1 rjags_3-12           coda_0.16-1
[4] lattice_0.20-24

loaded via a namespace (and not attached):
[1] grid_3.0.1  tools_3.0.1

1. Rasmus Arnling Bååth29 January 2014 10:41

I've also been thinking of good example of binary "processes" which are not 50%/50%, coins are such a bad example. Toast is good example!

I (almost) found some data on this. Though it does not mentions the exact numbers it seems like out of almost 10000 trials around 6200 toasts fell with the buttered side down: http://www.telegraph.co.uk/news/uknews/1331810/Breakfast-at-Murphys-or-why-the-toast-lands-butter-side-down.html. For quick beta stuff I often use the rbeta function in R that samples from a beta distribution. This would, for example, result in the median posterior and a 95% equitail credible interval:

quantile( rbeta(99999, 1 + 1, 1 + 6), c(0.05, 0.5, 0.975)

2. Thanks Rasmus. That's the article the Wikipedia is referring to.
But why simulate when you can have the answers directly via the qbeta function:
qbeta(c(0.025, 0.5, 0.975), 1 + 1, 1 + 6)

3. Rasmus Arnling Bååth30 January 2014 00:18

True, true. I guess I'm more of a random guy. Livin' on the edge, YOLO, and that kind of stuff.

4. regarding illustrative binary processes that aren't 50:50, what about the thumbtack flipping experiment, Pr(land on flat side). I saw that somewhere ....

5. Hello Markus,

I'm trying to find the optimum values (a and b) for a function's minumum value or a constant minimum level. I used the optim and optimize function but I can't find the optimal pairs of the parameters of a and b. My function is some more complicated and I create this function as min.RSS(a)(b) (two functions are nested) when I used the optim function and I received the following error from this code: error in optim(c(1,1), min.RSS) : cannot coerce type 'closure' to vector of type 'double'