Basically, we’re dealing with a Bernoulli distribution, and we want to estimate p from a given number of samples (which is easy), but we also want to know how sure we can be in our estimate (not so easy.)
Using Jeffrey’s interval we can draw up a Beta distribution for the probabilities for our probability.
In R, you can do this:
hit = 50
try = 1000
library(stats)
library(magicaxis)
x = exp(seq(log(1e-6*exp(1)),1,length.out=1e3))/exp(1)
d = dbeta(x, hit + 1/2, try - hit + 1/2)
plot(x,d,type="l",log="x",ylab = "", xlab="", xaxt="n", yaxt="n")
magaxis(grid=TRUE)
And you get this:
EDIT: I replaced the original image to add another curve for if you had 5 hits out of a 100 tries. Your “best guess” would still be 5%, but you would be a lot less sure about it. EDIT2: sorry i got randomly curious so i added the case for 1 out of 20 (red) as well
apparently, the most likely outcome is that we have less than 5% of the coins allocated
I used a logarithmic horizontal axis because it makes more sense for small p values, e.g. when you got 5 hits out of a 100,000 tries:
