Bayesian reasoning: Bayesian inference with Bayes’ theorem extended to multiple tests or pieces of evidence

by Jan Malakhovski, version 1.0.3, created , published , last updated

Bayes' theorem allows you to compute a probability of a hypothesis assuming a piece of evidence. But what do you do when you have several (hopefully independent) tests all pointing into different directions with different probabilities? Applying the classical formula of Bayes' theorem expressed using the law of total probability multiple times in a row for each piece of evidence naively will produce an incorrect result! The simplest way to do this properly is to switch to using Bayes' theorem in odds form, multiply likelihood ratios, and then convert back to probabilities. What? Why? This article defines everything, derives all the relevant formulas from scratch, and then demonstrates how to use them with some examples.

Table of Contents

Introduction

Consider the following. You think you might have caught some awful illness, you get a lab test done, the advertisement materials say the test is 99% accurate (as in, 99% tests of sick people come back positive and 99% tests of non-sick people come back negative), it comes back positive. What is the probability you are actually sick?

Without more information, it’s actually impossible to say! It depends on the probability of being sick with that illness in the first place.

For demonstration purposes, let’s say that this prior probability is 1%. If we make 10000 people take that lab test:

Therefore, if the above lab test came back positive, the probability of you being sick is only 50% and the “99% test accuracy” claim is misleading.

Moreover, 1% prior probability of being ill is rather high for an average illness. For example, in 2022 HIV/AIDS — one of the most common wold-wide diseases — had about 0.5% of the world-wide population infected. So, a “99% accurate” positive HIV test — which is what most of such modern tests claim to be — has a probability of being true of about 33% (49.549.5+99.5\frac{49.5}{49.5 + 99.5}).

Most laypersons are quite surprised the first time they hear about this, even though this computation is a simple and widely-known corollary of a well-known Bayes’ theorem (which is going to be discussed below) and it is immediately relevant when a person tests positive for an awful illness. It seems, however, that at least some people are starting to appreciate it. A fair fraction of people I mentioned this to at least heard something about this before, which makes me feel optimistic about this.

As a side-note, personally, I think it’s absurd that all relevant probabilities don’t simply get automatically computed and listed on all lab test results and printouts given to doctors and patients. Doing that would prevent a lot of unneeded stress. I kind of understand why they do not, but I think the assumptions behind that reasoning are wrong. The first lab that starts printing these probabilities while explaining them properly will simply capture the market of customers that care about precision, not get infinite numbers of complaints. Personally, I would rather use a more expensive lab that tracks its own accuracies — via patient diagnosis outcome tracking and/or publicly shared videos of them performing equipment calibration — than a cheaper one that does not.

(If some the following does not make sense, don’t worry, it will all be explained in the rest of this article and followup articles in this series.)

But what if you have several tests or pieces of evidence? How do you combine all those probabilities? If you work through the math carefully, you’ll notice that you can’t simply apply the classical formula of Bayes’ theorem expressed using the law of total probability multiple times (even though, this seems like a reasonable thing to do after reading the corresponding Wikipedia article), because it will start accumulating errors in the updated priors.

But if you could do it right somehow, then this kind of reasoning, Bayesian reasoning, could be applied to almost anything.

For example, when lab testing, it would be quite useful to have a way to combine multiple test results performed by independent labs using different equipment and methods (and, thus, different accuracies) to produce a total probability of the hypothesis under all test results.

This article is going explain this use case.

Similarly, when you read scientific studies, for each study and each hypothesis it tests, a p-value (or a confidence interval, which is a more verbose way to represent the same thing) informs you about the probability of that study having those observations given that its hypothesis is false. Meanwhile, Bayes’ theorem computes what you actually want to know: the probability of that hypothesis being true (or false) given the observations. But it only really becomes useful in practice if you can combine results of several independent studies testing the same hypothesis.

The next article in this series (coming soon) is going to explain this use case.

Similarly, when you read news articles, you could also use the same method to approximate probabilities of stated claims being true from trustworthiness probabilities of different sources/journalists/reporters/podcasters/news organizations/etc, topics, geographical locations, stock market changes, etc. Though, this use case is more complex, since you can’t assume different pieces of evidence to be independent here. This is essentially what intelligence agencies are supposed to do for their governments, when they aren’t collecting evidence. (What they actually do most of the time is sabotage evidence collection by others and power seek instead.)

A future article in this series (coming not so soon) is going to explain this use case.

So, this seems very useful. How can you do this probability math properly?

Eliezer Yudkowsky’s writings repeatedly mention that one needs to, to paraphrase, “switch to using Bayes’ theorem in odds form and then simply multiply likelihood ratios”. When I read that the first time, the problem with using the classical Bayes’ theorem multiple times in a row was not obvious to me, so I was really confused. What? Why? So, I’ve read his explanation of how to do this and, later, a supposedly better explanation linked by him, which replaced my “Odds form? Likelihood ratios? What? Why do I even need this?” questions with a smaller set of “Can I have a proper complete formula for doing this, please? Why do I even need this?” questions. I then supposedly managed to piece that formula together by myself from the above articles, the relevant Wikipedia article, another one, and a bunch of web searches. The resulting formula indeed looked quite useful for simplifying long computations, if nothing else. So, tried to do compute some examples using that formula, and it produced obviously wrong results.

That discouraged me from pursuing this line of research for a bit, but then I read more of Yudkowsky’s writings and it seemed like such a useful mental tool to have. If only I could make it work!

So, one day I decided to sit down and carefully work out all the relevant probability math from scratch myself. Hopefully, I though, it would make obvious where my previous mistakes were. (It did.) Then, later, I married that to David Colquhoun’s writings on Bayesian reasoning over scientific results, and the combined result became one of the most re-visited items on my “personal wiki” and one of my most useful mental tools ever. Which is why I decided to share this.

Probabilities, like a type theorist would see them

The basics

In the context of probability theory I like to think of random events as being Boolean predicates/conditions over the states of the world Event=WorldStateBoolA,B,C,H,E,...:Event\begin{aligned} &Event = WorldState \to Bool\\ &A, B, C, H, E, ... : Event \end{aligned} and probabilities as being ratios P(A)=N(A)N(All)P(A)=\frac{N(A)}{N(All)} where NN is a function that returns the number of the states of the world for which its argument evaluates to TrueTrue N:EventNatN : Event \to Nat and AllAll is the predicate that admits all possible world states All:EventAll=λs.True\begin{aligned} &All : Event\\ &All = \lambda s . True \end{aligned}

In other words:

Having defined the above, now we can define:

Also of note is the fact that P(AB)=N(AB)N(All)=N(A)+N(B)N(AB)N(All)=P(A)+P(B)P(AB)P(A \cup B) = \frac{N(A \cup B)}{N(All)} = \frac{N(A) + N(B) - N(A \cap B)}{N(All)} = P(A)+P(B)-P(A \cap B) because the size of intersection of AA and BB denoted N(AB)N(A \cap B) gets counted twice in N(A)+N(B)N(A) + N(B).

Conditional probability

We can define probability of “AA assuming BB”, also known as “conditional probability of AA under assumption BB”, denoted as P(A|B)P(A\vert B), as P(A|B)=P(AB)P(B)=N(AB)N(All)N(All)N(B)=N(AB)N(B)P(A\vert B) = \frac{P(A \cap B)}{P(B)} = \frac{N(A \cap B)}{N(All)}\frac{N(All)}{N(B)} = \frac{N(A \cap B)}{N(B)} I.e., it’s a probability of AA in a smaller world were BB is always true.

When I was studying this the first time in university, I remember being confused. Shouldn’t it be defined as follows? A|B=λs.B(s)A(s)=λs.¬B(s)A(s) !incorrect!A \vert B = \lambda s . B(s) \implies A(s) = \lambda s . \lnot B(s) \lor A(s) \text{ !incorrect!} It’s not! If nothing else, with this definition P(A|B)P(A\vert B) would still have N(All)N(All) in its denominator, which would be incorrect.

In other words, _|_\_\vert\_ is not an operator, A|BA\vert B is not an expression, P(A|B)P(A\vert B) is an atomic definition. The proper way to think about this is to consider P(A)P(A) to be a syntax sugar for P(A|All)P(A\vert All) P(A)=P(A|All)=N(AAll)N(All)=N(A)N(All)P(A) = P(A\vert All) = \frac{N(A \cap All)}{N(All)} = \frac{N(A)}{N(All)}

Of special note is the fact that the above definition of P(A|B)P(A\vert B) trivially implies P(AB)=P(A|B)P(B)P(A \cap B) = P(A\vert B)P(B) We are going to use this observation quite a lot in what follows.

Law of total probability

For a set of events BiB_i for i=1..ni=1..n, which are mutually exclusive P(BiBj)=0 when ijP(B_i \cap B_j) = 0 \text{ when } i \ne j and collectively exhaustive P(B1B2Bn)=1P(B_1 \cup B_2 \cup \dots \cup B_n) = 1 the following formula, called the law of total probability, holds P(A)=i=1nP(ABi)=i=1nP(A|Bi)P(Bi)P(A)=\sum_{i=1}^n P(A \cap B_i)=\sum_{i=1}^n P(A\vert B_i)P(B_i)

This can be specialized to the following case. For an event BB and its negation ¬B\lnot B, the following holds P(A)=P(A|B)P(B)+P(A|¬B)P(¬B)P(A) = P(A\vert B)P(B) + P(A\vert \lnot B)P(\lnot B) We are going to use this observation multiple times in what follows.

Chain rule

Note that, since \cap operator is associative, P(AB)=P(A|B)P(B)P(A \cap B) = P(A\vert B)P(B) can be applied to an arbitrary number of events by taking A=A1A = A_1 and B=A2AnB = A_2 \cap \dots \cap A_n, which gives P(A1A2An)=P(A1|A2An)×P(A2An)P(A_1 \cap A_2 \cap \dots \cap A_n) = P(A_1\vert A_2 \cap \dots \cap A_n) \times P(A_2 \cap \dots \cap A_n)

That expansion can then be repeated for P(A2An)P(A_2 \cap \dots \cap A_n) and other similar sub-expressions. This observation gives us the following formula, called the chain rule P(A1A2An)=P(A1|A2An)×P(A2|A3An)××P(An1|An)×P(An)P(A_1 \cap A_2 \cap \dots \cap A_n) = P(A_1\vert A_2 \cap \dots \cap A_n) \times P(A_2\vert A_3 \cap \dots \cap A_n) \times \dots \times P(A_{n-1}\vert A_n) \times P(A_n)

Hypothesis testing terminology

Consider the following matrix of outcome frequencies when testing a single hypothesis against its negation using a single one-bit-of-information test procedure

Hypothesis is true Hypothesis is false
Test is positive True positives N(TP)N(TP) False positives N(FP)N(FP)
Test is negative False negatives N(FN)N(FN) True negatives N(TN)N(TN)

For the original disease testing example from the beginning of introduction this matrix would look as follows

Sick Healthy
Test is positive 99 99
Test is negative 1 9801

Most testing procedures specify these matrices by giving the following three probabilities:

Given these three values and the number of trials N(All)N(All) we can compute the contents of each cell of the above matrix by simple multiplications.

Classical Bayes’ theorem

The derivation for the classical form of the Bayes’ theorem can now be done as follows. By definition P(A|B)=P(AB)P(B)P(A\vert B) = \frac{P(A \cap B)}{P(B)} therefore, taking into the account that \cap operator is commutative P(AB)=P(A|B)P(B)=P(B|A)P(A)P(A \cap B) = P(A\vert B)P(B) = P(B\vert A)P(A) Throwing away the P(AB)P(A \cap B) part and dividing both leftover parts by P(B)P(B) gives us the following formula, called the Bayes’ theorem P(A|B)=P(B|A)P(A)P(B)=P(A)×P(B|A)P(B) (AKA bayes)P(A\vert B) = \frac{P(B\vert A)P(A)}{P(B)} = P(A) \times \frac{P(B\vert A)}{P(B)} \text{ (AKA bayes)} That’s it.

In practice, it’s useful to treat the AA and BB in the above as a hypothesis and its evidence P(hypothesis|evidence)=P(hypothesis)×P(evidence|hypothesis)P(evidence)P(hypothesis\vert evidence) = P(hypothesis) \times \frac{P(evidence\vert hypothesis)}{P(evidence)} which, with its denominator expanded using the law of total probability, becomes =P(hypothesis)×P(evidence|hypothesis)P(evidence|hypothesis)P(hypothesis)+P(evidence|¬hypothesis)P(¬hypothesis)= P(hypothesis) \times \frac{P(evidence\vert hypothesis)}{P(evidence\vert hypothesis)P(hypothesis) + P(evidence\vert \lnot hypothesis)P(\lnot hypothesis)} which is just =prior×sensitivitysensitivity×prior+(1specificity)×(1prior)= prior \times \frac{sensitivity}{sensitivity \times prior + (1 - specificity) \times (1 - prior)}

This gives the classical way to compute the answer to the first example above P(sick|positive)=0.01×0.990.99×0.01+(10.99)×(10.01)=0.5P(sick\vert positive) = 0.01 \times \frac{0.99}{0.99 \times 0.01 + (1 - 0.99) \times (1 - 0.01)} = 0.5

Note, however, that we can multiply both parts of the above fraction by the number of trials N(All)N(All) P(hypothesis|evidence)=N(All)×sensitivity×priorN(All)×sensitivity×prior+N(All)×(1specificity)×(1prior)P(hypothesis\vert evidence) = \frac{N(All) \times sensitivity \times prior}{N(All) \times sensitivity \times prior + N(All) \times (1 - specificity) \times (1 - prior)} which then becomes =N(TP)N(TP)+N(FP)= \frac{N(TP)}{N(TP) + N(FP)} which is the formula I used in the examples at the beginning of introduction.

The TP-FP formula is most useful for quick mental ballpark calculations for a single test. The prior-sensitivity-specificity formula is most helpful for automated calculations.

But what we actually want to derive is a formula for a single hypothesis with multiple pieces of evidence P(H|E1E2En)=?P(H\vert E_1 \cap E_2 \cap \dots \cap E_n) = ? where, for brevity, hypothesishypothesis was renamed to simply HH and evidenceevidence became a set of EiE_i for i=1..ni=1..n, where nn is the number of tests.

So, let’s derive that now.

Classical Bayes’ theorem with multiple tests

The classical formula for Bayes’ theorem with multiple tests can be derived similarly to the previous version by applying the chain rule to P(HE1E2En)P(H \cap E_1 \cap E_2 \cap \dots \cap E_n) as follows P(HE1E2En)=P(H|E1E2En)×P(E1E2En)=P(H|E1E2En)×P(E1|E2En)×P(E2|E3En)××P(En)\begin{aligned} P(H \cap E_1 \cap E_2 \cap \dots \cap E_n) &= P(H\vert E_1 \cap E_2 \cap \dots \cap E_n) \times P(E_1 \cap E_2 \cap \dots \cap E_n)\\ &= P(H\vert E_1 \cap E_2 \cap \dots \cap E_n) \times P(E_1\vert E_2 \cap \dots \cap E_n) \times P(E_2\vert E_3 \cap \dots \cap E_n) \times \dots \times P(E_n) \end{aligned} Then, by shuffling elements under \cap, since that operator is both commutative and associative, and doing it again we get P(HE1E2En)=P(E1E2EnH)=P(E1|E2EnH)×P(E2|E3EnH)××P(En|H)×P(H)\begin{aligned} P(H \cap E_1 \cap E_2 \cap \dots \cap E_n) &= P(E_1 \cap E_2 \cap \dots \cap E_n \cap H)\\ &= P(E_1\vert E_2 \cap \dots \cap E_n \cap H) \times P(E_2\vert E_3 \cap \dots \cap E_n \cap H) \times \dots \times P(E_n\vert H) \times P(H) \end{aligned} Combining both the same way we did before, we get P(H|E1E2En)=P(H)×P(E1|E2EnH)×P(E2|E3EnH)××P(En|H)P(E1|E2En)×P(E2|E3En)××P(En)P(H\vert E_1 \cap E_2 \cap \dots \cap E_n) = P(H) \times \frac{P(E_1\vert E_2 \cap \dots \cap E_n \cap H) \times P(E_2\vert E_3 \cap \dots \cap E_n \cap H) \times \dots \times P(E_n\vert H)}{P(E_1\vert E_2 \cap \dots \cap E_n) \times P(E_2\vert E_3 \cap \dots \cap E_n) \times \dots \times P(E_n)}

When all tests are independent from each other P(EiEj)=P(Ei)P(Ej) when ijP(E_i \cap E_j)=P(E_i)P(E_j) \text{ when } i \ne j then, by definition of P(A|B)P(A\vert B), this implies P(Ei|Ej)=P(Ei) when ijP(E_i\vert E_j) = P(E_i) \text{ when } i \ne j which simplifies the above formula to P(H|E1E2En)=P(H)×i=1nP(Ei|H)i=1nP(Ei)=P(H)×i=1nP(Ei|H)P(Ei) (AKA naive-bayes)P(H\vert E_1 \cap E_2 \cap \dots \cap E_n) = P(H) \times \frac{\prod_{i=1}^{n} P(E_i\vert H)}{\prod_{i=1}^{n} P(E_i)}=P(H) \times \prod_{i=1}^{n} \frac{P(E_i\vert H)}{P(E_i)} \text{ (AKA naive-bayes)}

This, by the way, is almost the exact same formula Naive Bayes classifier uses, except Naive Bayes ignores the i=1nP(Ei)\prod_{i=1}^{n} P(E_i) constant for simplicity, since it’s a classifier and the actual probability value is unimportant there.

Note, however, that the law of total probability turns the above formula into P(H|E1E2En)=P(H)×i=1nP(Ei|H)P(Ei|H)P(H)+P(Ei|¬H)P(¬H)P(H\vert E_1 \cap E_2 \cap \dots \cap E_n) = P(H) \times \prod_{i=1}^{n} \frac{P(E_i\vert H)}{P(E_i\vert H)P(H) + P(E_i\vert \lnot H)P(\lnot H)} which is =prior×i=1nsensitivityisensitivityi×prior+(1specificityi)×(1prior)= prior \times \prod_{i=1}^{n} \frac{sensitivity_i}{sensitivity_i \times prior + (1 - specificity_i) \times (1 - prior)}

This formula works, but it’s a bit annoying to use: each time you make a new test, you have to remember to use the original priorprior in the denominator or recompute the whole thing from scratch.

Or, to say it another way, trying to compute the above by applying single-evidence Bayes’ theorem nn times by doing prior0=priorpriori+1=priori×sensitivityisensitivityi×priori+(1specificityi)×(1priori) !incorrect!\begin{aligned} prior_0 &= prior\\ prior_{i+1} &= prior_i \times \frac{sensitivity_i}{sensitivity_i \times prior_i + (1 - specificity_i) \times (1 - prior_i)} \text{ !incorrect!} \end{aligned} and then taking priornprior_n as an answer produces an incorrect result because each sensitivityisensitivity_i and specificityispecificity_i were given for the original priorprior, not the updated prioriprior_i values.

Luckily, there’s a slightly more general and yet simpler way to do this.

Bayes’ theorem in odds form with multiple tests

Consider a ratio of conditional probabilities of two hypotheses H1H_1 and H2H_2 under the same evidence EE P(H1|E)P(H2|E)=P(E|H1)P(H1)P(E)P(E)P(E|H2)P(H2)=P(E|H1)P(H1)P(E|H2)P(H2)=P(H1)P(H2)P(E|H1)P(E|H2)\frac{P(H_1\vert E)}{P(H_2\vert E)} = \frac{P(E\vert H_1)P(H_1)}{P(E)}\frac{P(E)}{P(E\vert H_2)P(H_2)} = \frac{P(E\vert H_1)P(H_1)}{P(E\vert H_2)P(H_2)} = \frac{P(H_1)}{P(H_2)}\frac{P(E\vert H_1)}{P(E\vert H_2)} or, similarly, for multiple pieces of evidence P(H1|E1E2En)P(H2|E1E2En)=P(H1)×P(E1|E2EnH1)×P(E2|E3EnH1)××P(En|H1)P(E1|E2En)×P(E2|E3En)××P(En)P(H2)×P(E1|E2EnH2)×P(E2|E3EnH2)××P(En|H2)P(E1|E2En)×P(E2|E3En)××P(En)=P(H1)P(H2)P(E1|E2EnH1)×P(E2|E3EnH1)××P(En|H1)P(E1|E2EnH2)×P(E2|E3EnH2)××P(En|H2)\begin{aligned} \frac{P(H_1\vert E_1 \cap E_2 \cap \dots \cap E_n)}{P(H_2\vert E_1 \cap E_2 \cap \dots \cap E_n)} &= \frac{P(H_1) \times \frac{P(E_1\vert E_2 \cap \dots \cap E_n \cap H_1) \times P(E_2\vert E_3 \cap \dots \cap E_n \cap H_1) \times \dots \times P(E_n\vert H_1)}{P(E_1\vert E_2 \cap \dots \cap E_n) \times P(E_2\vert E_3 \cap \dots \cap E_n) \times \dots \times P(E_n)}}{P(H_2) \times \frac{P(E_1\vert E_2 \cap \dots \cap E_n \cap H_2) \times P(E_2\vert E_3 \cap \dots \cap E_n \cap H_2) \times \dots \times P(E_n\vert H_2)}{P(E_1\vert E_2 \cap \dots \cap E_n) \times P(E_2\vert E_3 \cap \dots \cap E_n) \times \dots \times P(E_n)}}\\ &=\frac{P(H_1)}{P(H_2)} \frac{P(E_1\vert E_2 \cap \dots \cap E_n \cap H_1) \times P(E_2\vert E_3 \cap \dots \cap E_n \cap H_1) \times \dots \times P(E_n\vert H_1)}{P(E_1\vert E_2 \cap \dots \cap E_n \cap H_2) \times P(E_2\vert E_3 \cap \dots \cap E_n \cap H_2) \times \dots \times P(E_n\vert H_2)} \end{aligned} which, when all tests are independent, gives us P(H1|E1E2En)P(H2|E1E2En)=P(H1)P(H2)×i=1nP(Ei|H1)P(Ei|H2) (AKA naive-bayes-ratio)\frac{P(H_1\vert E_1 \cap E_2 \cap \dots \cap E_n)}{P(H_2\vert E_1 \cap E_2 \cap \dots \cap E_n)} = \frac{P(H_1)}{P(H_2)} \times \prod_{i=1}^{n} \frac{P(E_i\vert H_1)}{P(E_i\vert H_2)} \text{ (AKA naive-bayes-ratio)}

Note that this formula no longer mentions any P(Ei)P(E_i) values, so the law of total probability is no longer needed here.

Now, let’s define O(A:B|E)=P(A|E)P(B|E)=P(AE)P(E)P(E)P(BE)=N(AE)N(BE)O(A:B)=O(A:B|All)=P(A)P(B)=N(A)N(B)\begin{aligned} O(A : B\vert E) &= \frac{P(A\vert E)}{P(B\vert E)} = \frac{P(A \cap E)}{P(E)}\frac{P(E)}{P(B \cap E)} = \frac{N(A \cap E)}{N(B \cap E)}\\ O(A : B) &= O(A : B\vert All) = \frac{P(A)}{P(B)} = \frac{N(A)}{N(B)} \end{aligned}

These represent relative odds of AA and BB assuming EE and with no assumptions, respectively.

With this we can rewrite the above naive-bayes-ratio formula as O(H1:H2|E1E2En)=O(H1:H2)×i=1nP(Ei|H1)P(Ei|H2) (AKA naive-bayes-odds)O(H_1 : H_2\vert E_1 \cap E_2 \cap \dots \cap E_n) = O(H_1 : H_2) \times \prod_{i=1}^{n} \frac{P(E_i\vert H_1)}{P(E_i\vert H_2)}\text{ (AKA naive-bayes-odds)}

In the literature, it’s common to also define Λ(H1:H2|E)=P(E|H1)P(E|H2)\Lambda(H_1 : H_2\vert E) = \frac{P(E\vert H_1)}{P(E\vert H_2)} which represents the likelihood ratio of observing EE under two different hypotheses H1H_1 and H2H_2.

This allows us to rewrite the above formula as O(H1:H2|E1E2En)=O(H1:H2)×i=1nΛ(H1:H2|Ei) (AKA naive-bayes-likelihoods)O(H_1 : H_2\vert E_1 \cap E_2 \cap \dots \cap E_n) = O(H_1 : H_2) \times \prod_{i=1}^{n} \Lambda(H_1 : H_2\vert E_i) \text{ (AKA naive-bayes-likelihoods)}

This formula is what Bayesian reasoning is all about: you take a ratio of prior probabilities of two hypotheses you want to compare and then repeatedly update your knowledge by multiplying your current odds by likelihood ratios for each piece of evidence.

Using it to compute probabilities

We can still use the above formulas to compute P(H|E1E2En)P(H\vert E_1 \cap E_2 \cap \dots \cap E_n) for a singe hypothesis HH as follows.

Firstly, note that the above definitions of OO imply that O(A:¬A|E)=P(A|E)1P(A|E)O(A:¬A)=P(A)1P(A)\begin{aligned} O(A : \lnot A\vert E) &= \frac{P(A\vert E)}{1 - P(A\vert E)}\\ O(A : \lnot A) &= \frac{P(A)}{1 - P(A)} \end{aligned}

Secondly, note that if q=p1pq = \frac{p}{1 - p} then p=q×(1p)=qq×pp+q×p=qp×(1+q)=q\begin{aligned} &p = q \times (1 - p) = q - q \times p\\ &p + q \times p = q\\ &p \times (1 + q) = q \end{aligned} and thus p=q1+qp = \frac{q}{1 + q}

Combining these gives us the way to convert O(A:¬A|E)O(A : \lnot A\vert E) odds to probabilities P(A|E)=O(A:¬A|E)1+O(A:¬A|E)P(A)=O(A:¬A)1+O(A:¬A)\begin{aligned} P(A\vert E) &= \frac{O(A : \lnot A\vert E)}{1 + O(A : \lnot A\vert E)}\\ P(A) &= \frac{O(A : \lnot A)}{1 + O(A : \lnot A)} \end{aligned}

Thus, if substitute H1=HH_1=H and H2=¬HH_2=\lnot H into the above naive-bayes-odds formula O(H:¬H|E1E2En)=O(H:¬H)×i=1nP(Ei|H)P(Ei|¬H)O(H : \lnot H\vert E_1 \cap E_2 \cap \dots \cap E_n) = O(H : \lnot H) \times \prod_{i=1}^{n} \frac{P(E_i\vert H)}{P(E_i\vert \lnot H)} it becomes O(H:¬H|E1E2En)=P(H)1P(H)×i=1nP(Ei|H)P(Ei|¬H)=prior1prior×i=1nsensitivityi1specificityiO(H : \lnot H\vert E_1 \cap E_2 \cap \dots \cap E_n) = \frac{P(H)}{1 - P(H)} \times \prod_{i=1}^{n} \frac{P(E_i\vert H)}{P(E_i\vert \lnot H)}=\frac{prior}{1 - prior} \times \prod_{i=1}^{n} \frac{sensitivity_i}{1 - specificity_i} and we can compute P(H|E1E2En)P(H\vert E_1 \cap E_2 \cap \dots \cap E_n) as P(H|E1E2En)=O(H:¬H|E1E2En)1+O(H:¬H|E1E2En)P(H\vert E_1 \cap E_2 \cap \dots \cap E_n) = \frac{O(H : \lnot H\vert E_1 \cap E_2 \cap \dots \cap E_n)}{1 + O(H : \lnot H\vert E_1 \cap E_2 \cap \dots \cap E_n)}

So, for instance, for the original disease testing example from the beginning of introduction x=O(sick:¬sick|positive)=P(sick)1P(sick)P(sick|positive)P(¬sick|positive)=prior1priorsensitivity1specificity=0.010.990.9910.99=1x = O(sick : \lnot sick\vert positive) = \frac{P(sick)}{1 - P(sick)}\frac{P(sick\vert positive)}{P(\lnot sick\vert positive)} = \frac{prior}{1 - prior}\frac{sensitivity}{1 - specificity}=\frac{0.01}{0.99}\frac{0.99}{1 - 0.99} = 1 thus P(sick|positive)=x1+x=0.5P(sick\vert positive)=\frac{x}{1 + x}=0.5

To demonstrate the real usefulness of this, let’s say you did three independent HIV tests at different labs, they came back positive, then negative, then positive again, and you want to know your probability of being sick. Let’s say the first test had sensitivity,specificitysensitivity, specificity of 0.8,0.90.8, 0.9 because it was a cheap self-administered test, the second test had probabilities 0.9,0.90.9, 0.9 and was performed by a closest available lab, and the third test had probabilities of 0.99,0.990.99, 0.99 and was performed by sending your blood to a well-known lab.

With that we can compute y=O(sick:¬sick|test1¬test2test3)=0.00510.0050.810.910.90.90.9910.99y=O(sick:\lnot sick\vert test_1 \cap \lnot test_2 \cap test_3)=\frac{0.005}{1 - 0.005} \frac{0.8}{1 - 0.9} \frac{1 - 0.9}{0.9} \frac{0.99}{1 - 0.99} Note how for the second test we multiply by the 1/Λ1/\Lambda since it came back negative.

Therefore P(sick|test1¬test2test3)=y1+y30%P(sick\vert test_1 \cap \lnot test_2 \cap test_3)=\frac{y}{1 + y} \approx 30\%

This seems like a lot to you, so you go and take another very expensive test, which claims sensitivity,specificitysensitivity, specificity of 0.99,0.9990.99, 0.999 and comes back negative, what is the probability of being sick now? z=O(sick:¬sick|test1¬test2test3¬test4)=y×10.9990.99z=O(sick:\lnot sick\vert test_1 \cap \lnot test_2 \cap test_3 \cap \lnot test_4)=y \times \frac{1 - 0.999}{0.99} P(sick|test1¬test2test3¬test4)=z1+z0.045%P(sick\vert test_1 \cap \lnot test_2 \cap test_3 \cap \lnot test_4)=\frac{z}{1 + z} \approx 0.045\%

After some practice this actually becomes pretty easy to do in your head, giving you a nice mental tool for quickly approximating ballpark odds of something under multiple pieces of evidence.

Computing in the presence of non-independent hypotheses

This section is a work in progress.

See Bayesian networks.

Changelog