W14 Notes

Significance testing

06 Theory - Significance testing

Theory 1 - Significance testing

Significance test

Ingredients of a significance test (unary hypothesis test):

• $H_0$ — Null hypothesis event
  • Identify a Claim
  • Then: $H_0$ is background assumption (supposing Claim isn’t known)
  • Goal is to invalidate $H_0$ in favor of Claim
• $R$ — Rejection Region event (decision rule)
  • $R$ is written in terms of decision statistic $X$ and significance level $\alpha$
  • $R$ is unlikely assuming $H_0$. $R$ is more likely if Claim
  • “If $X$ falls in $R$, this Test rejects $H_0$.”
• $P[R|H_0]$ — Able to compute this
  • Usually: inferred from $f_{X|H_0}$ or $P_{X|H_0}$
  • Adjust $R$ to achieve $P[R|H_0]=\alpha$

Significance level

Suppose we are given a null hypothesis $H_0$ and a rejection region $R$.

The significance level of $R$ is:

$$\begin{array}{cc}\alpha & =P[R|H_0]\\ & \\ & =P[\text{reject }{H}_0|H_0\text{ is true}]\end{array}$$

Sometimes the condition is dropped and we write $\alpha =P[R]$, e.g. when a background model without assuming $H_0$ is not known.

Null hypothesis implies a distribution

Usually $S$ is unspecified, yet $H_0$ determines a known distribution.

In this case $H_0$ will not take the form of an event in a sample space, $H_0\subset S$.

At a minimum, $H_0$ must determine $P[R|H_0]$.

We do NOT need these details:

• Background sample space $S$
• Non-conditional distribution (full model): $f_X$ or $P_X$
• Complement conditionals: $f_{X|H_0^c}$ or $P_{X|H_0^c}$

---

In basic statistical inference theory, there are two kinds of error.

• Type I error concludes with rejecting $H_0$ when $H_0$ is true.
• Type II error concludes with maintaining $H_0$ when $H_0$ is false.

Type I error is usually a bigger problem. We want to consider $H_0$ as “innocent until proven guilty.”

	$H_0$ is true	$H_0$ is false
Maintain null hypothesis	Made right call	Wrong acceptance $\text{T}\text{y}\text{p}\text{e}\text{I}\text{I}\text{E}\text{r}\text{r}\text{o}\text{r}$
Reject null hypothesis	Wrong rejection $\text{T}\text{y}\text{p}\text{e}\text{I}\text{E}\text{r}\text{r}\text{o}\text{r}$	Made right call

---

To design a significance test at $\alpha$, we must identify $H_0$, and specify $R$ with the property that $P[R|H_0]=\alpha$.

When $R$ is written using a variable $X$, we must choose between:

• One-tail rejection region: $x$ with $R(x)\le r$ or $x$ with $R(x)\ge r$
• Two-tail rejection region: $x$ with $|R(x)-\mu |\ge c$

Link to original

07 Illustration

Example - One-tail test: Weighted die

One-tail test: Weighted die

Your friend gives you a single regular die, and say she is worried that it has been weighted to prefer the outcome of 2. She wants you to test it.

Design a significance test for the data of 20 rolls of the die to determine whether the die is weighted. Use significance level $\alpha =0.05$.

Solution

Let $X$ count the number of 2s that come up.

The Claim: “the die is weighted to prefer 2” The null hypothesis $H_0$: “the die is normal”

Assuming $H_0$ is true, then $X\sim \mathrm{Bin}(\mathrm{20,1}/6)$, and therefore:

$$P_{X|H_0}(k)=\left(\genfrac{}{}{0px}{}{20}{k}\right){(1/6)}^k{(5/6)}^{20-k}$$

---

⚠️ Notice that “prefer 2” implies the claim is for more 2s than normal.

Therefore: Choose a one-tail rejection region.

Need $r$ such that:

$$\begin{array}{cc}P[X\ge r|H_0]& =0.05\\ & \\ ⟺P[X<r|H_0]& =0.95\end{array}$$

---

Solve for $r$ by computing conditional CDF values:

$k:$	0	1	2	3	4	5	6	7
$F_{X|H_0}(k):$	0.026	0.130	0.329	0.567	0.769	0.898	0.963	0.989

Therefore, choose $r=6$:

$P[X\ge 6|H_0]<0.04$, but $P[X\ge 5|H_0]>0.05$. Final answer:

$$R=\{x|x\ge 6\}$$ Link to original

Two-tail test: Circuit voltage

Two-tail test: Circuit voltage

A boosted AC circuit is supposed to maintain an average voltage of $130\mathrm{V}$ with a standard deviation of $2.1\mathrm{V}$. Nothing else is known about the voltage distribution.

Design a two-tail test incorporating the data of 40 independent measurements to determine if the expected value of the voltage is truly $130\mathrm{V}$. Use $\alpha =0.02$.

Solution

Use $M_{40}(V)$ as the decision statistic, i.e. the sample mean of 40 measurements of $V$.

The Claim to test: $E[V]\ne 130$

The null hypothesis $H_0$: $E[V]=130$

Rejection region:

$$|M_{40}-130|\ge c$$

where $c$ is chosen so that $P[|M_{40}-130|\ge c]=0.02$

---

Assuming $H_0$, we expect that:

$$E[M_{40}]=130,{\sigma }_{M_{40}}^2=\frac{{2.1}^2}{40}\approx 0.110$$

Recall Chebyshev’s inequality:

$$P[|M_{40}-130|\ge c]\le \frac{{\sigma }_{M_{40}}^2}{c^2}\approx \frac{0.110}{c^2}$$

---

Now solve:

$$\frac{0.110}{c^2}=0.02\gg \gg c\approx 2.348$$

Therefore the rejection region should be:

$$M_{40}<127.65\cup 132.35<M_{40}$$ Link to original

One-tail test with a Gaussian: Weight loss drug

One-tail test with a Gaussian: Weight loss drug

Assume that in the background population in a specific demographic, the distribution of a person’s weight $W$ satisfies $W\sim 𝒩({\mathrm{190,24}}^2)$. Suppose that a pharmaceutical company has developed a weight-loss drug and plans to test it on a group of 64 individuals.

Design a test at the $\alpha =0.01$ significance level to determine whether the drug is effective.

Solution

Since the drug is tested on 64 individuals, we use the sample mean $M_{64}(W)$ as the decision statistic.

The Claim: “the drug is effective in reducing weight”

The null hypothesis $H_0$: “no effect: weights on the drug still follow $𝒩({\mathrm{190,24}}^2)$”

Assuming $H_0$ is true, then $W\sim 𝒩({\mathrm{190,24}}^2)$.

⚠️ One-tail test because the drug is expected to reduce weight (unidirectional). Rejection region:

$$M_{64}(W)\le r$$

---

Calculate:

$${\sigma }_{M_{64}}^2\gg \gg \frac{{24}^2}{64}\gg \gg 9$$

⚠️ Standardized $M_{64}(W)$ is approximately normal!

(The standardization of $M_{64}(W)$ removes the effect of $\frac{1}{n}$. As if it’s the summation.)

So, standardize and apply CLT:

$$\begin{array}{c}\frac{M_{64}(W)-190}{\sqrt{9}}\sim 𝒩(\mathrm{0,1}),\\ \\ \gg \gg P[M_{64}(W)\le r]\approx P[Z\le \frac{r-190}{3}]=\mathrm{\Phi }\left(\frac{r-190}{3}\right)\end{array}$$

---

Solve:

$$\begin{array}{c}P[M_{64}(W)\le r]=0.01\\ \\ \gg \gg \mathrm{\Phi }\left(\frac{r-190}{3}\right)=0.01\\ \\ \gg \gg \mathrm{\Phi }\left(\frac{190-r}{3}\right)=0.99\\ \\ \gg \gg \frac{190-r}{3}=2.33\\ \\ \gg \gg r=183.01\end{array}$$

Therefore, the rejection region:

$$M_{64}(W)\le 183.01$$Link to original

Binary hypothesis testing

01 Theory - Binary testing, MAP and ML

Theory 1 - Binary testing, MAP and ML

Binary hypothesis test

Ingredients of a binary hypothesis test:

• $H_0$ and $H_1$ — Complementary hypotheses
  • Maybe also know the prior probabilities $P[H_0]$ and $P[H_1]$
  • Goal: determine which case we are in, $H_0$ or $H_1$
• $A_0$ and $A_1$ — Complementary events of the Decision Rule
  • Directionality: given $H_0$, $A_0$ is likely; given $H_1$, $A_1$ is likely
  • Decision Rule: outcome $A_0$, accept $H_0$; outcome $A_1$, accept $H_1$
  • Usually: $A_i$ written in terms of decision statistic $X$ using a design
  • We cover three designs:
    • MAP and ML (minimize ‘error probability’)
    • MC (minimizes ‘error cost’)
  • Designs use $P_{X|H_0}$ and $P_{X|H_1}$ (or $f_{X|H_0}$, $f_{X|H_1}$) to construct $A_0$ and $A_1$

MAP design

Suppose we know:

• $P[H_0]$ and $P[H_1]$
  • Both prior probabilities
• $P_{X|H_0}(x)$ and $P_{X|H_1}(x)$ (or $f_{X|H_0}(x)$ and $f_{X|H_1}(x)$)
  • Both conditional distributions

The maximum a posteriori probability (MAP) design for a decision statistic $X$:

$$A_0=\text{set of }x\text{ for which:}$$

Discrete case:

$$P_{X|H_0}(x)\cdot P[H_0]\ge P_{X|H_1}(x)\cdot P[H_1]$$

Continuous case:

$$f_{X|H_0}(x)\cdot P[H_0]\ge f_{X|H_1}(x)\cdot P[H_1]$$

And $A_1=\{x\in ℝ|x\notin A_0\}$.

The MAP design minimizes the total probability of error.

ML design

Suppose we don’t know the priors, we know only:

• $P_{X|H_0}(x)$ and $P_{X|H_1}(x)$ (or $f_{X|H_0}(x)$ and $f_{X|H_1}(x)$)
  • Both conditional distributions

The maximum likelihood (ML) design for $X$:

$$A_0=\text{set of }x\text{ for which:}\begin{array}{cc}{P}_{X|H_0}(x)& \ge P_{X|H_1}(x)\text{(discrete)}\\ & \\ f_{X|H_0}(x)& \ge f_{X|H_1}(x)\text{(continuous)}\end{array}$$

ML is a simplified version of MAP. $$ (Set $P[H_0]$ and $P[H_1]$ to $0.5$.)

---

The false alarm error rate is called $P_{FA}$. The miss error rate is called $P_{\text{Miss}}$.

$$\begin{array}{cc}{P}_{FA}& =P[A_1|H_0]\\ & \\ P_{\text{Miss}}& =P[A_0|H_1]\end{array}$$

Total probability of error:

$$P_{\text{ERR}}=P[A_1|H_0]\cdot P[H_0]+P[A_0|H_1]\cdot P[H_1]$$

Wrong meanings of $P_{FA}$

Suppose $A_1$ sets off a smoke alarm, and $H_0$ is ‘no fire’ and $H_1$ is ‘yes fire’.

Then $P_{FA}$ is the odds that we get an alarm assuming there is no fire.

This is not the odds of experiencing a false alarm (no context). That would be $P[A_1{H}_0]$.

This is not the odds of a given alarm being a false one. That would be $P[H_0|A_1]$.

Link to original

02 Illustration

Example - ML test: Smoke detector

ML test: Smoke detector

Suppose that a smoke detector sensor is configured to produce $8\mathrm{V}$ when there is smoke, and $0\mathrm{V}$ otherwise. But there is background noise with distribution $𝒩(0,3^2)$.

Design an ML test for the detector electronics to decide whether to activate the alarm.

What are the three error probabilities? (Type I, Type II, Total.)

Solution

First, establish the conditional distributions:

$$X|H_0\sim 𝒩(0,3^2)X|H_1\sim 𝒩(8,3^2)$$

Density functions:

$$f_{X|H_0}=\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}{f}_{X|H_1}=\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}$$

---

The ML condition becomes:

$$\begin{array}{c}\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\stackrel{?}{\ge }\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\\ \\ \gg \gg -\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2\stackrel{?}{\ge }-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2\\ \\ \gg \gg x^2\stackrel{?}{\le }{(x-8)}^2\\ \\ \gg \gg x\le 4\end{array}$$

---

Therefore, $A_0$ is $x\le 4$, while $A_1$ is $x>4$.

The decision rule is: activate alarm when $x>4$.

---

Type I error:

$$\begin{array}{c}{P}_{FA}=P[A_1|H_0]\gg \gg P[X>4|H_0]\\ \\ \gg \gg 1-P[\frac{X-0}{3}\le \frac{4}{3}|H_0]\\ \\ \gg \gg 1-P[Z\le 1.3333]\gg \gg \approx 0.0912\end{array}$$

Type II error:

$$\begin{array}{c}{P}_{\text{Miss}}=P[A_0|H_1]\gg \gg P[X\le 4|H_1]\\ \\ \gg \gg P[\frac{X-8}{3}\le \frac{4-8}{3}|H_1]\\ \\ \gg \gg P[Z\le -1.3333]\gg \gg \approx 0.0912\end{array}$$

Total error:

$$P_{\mathrm{ERR}}=P_{FA}\cdot 0.5+P_{\text{Miss}}\cdot 0.5\gg \gg \approx 0.0912$$ Link to original

Example - MAP test: Smoke detector

MAP test: Smoke detector

Suppose that a smoke detector sensor is configured to produce $8\mathrm{V}$ when there is smoke, and $0\mathrm{V}$ otherwise. But there is background noise with distribution $𝒩(0,3^2)$.

Suppose that the background chance of smoke is $5\%$. Design a MAP test for the alarm.

What are the three error probabilities? (Type I, Type II, Total.)

Solution

First, establish priors:

$$P[H_0]=0.95P[H_1]=0.05$$

The MAP condition becomes:

$$\begin{array}{c}\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\cdot \text{0}\text{.}\text{9}\text{5}\stackrel{?}{\ge }\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot \text{0}\text{.}\text{0}\text{5}\\ \\ \gg \gg e^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\stackrel{?}{\ge }{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot \frac{0.05}{0.95}\\ \\ \gg \gg -\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2\stackrel{?}{\ge }-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2+\ln \left(\frac{0.05}{0.95}\right)\\ \\ \gg \gg x^2\stackrel{?}{\le }{(x-8)}^2-18\ln \left(\frac{0.05}{0.95}\right)\\ \\ \gg \gg x\le 7.31\end{array}$$

---

Therefore, $A_0$ is $x\le 7.31$, while $A_1$ is $x>7.31$.

The decision rule is: activate alarm when $x>7.31$.

---

Type I error:

$$\begin{array}{c}{P}_{FA}=P[A_1|H_0]\gg \gg P[X>7.31|H_0]\\ \\ \gg \gg 1-P[Z\le 2.4367]\gg \gg \approx 0.007411\end{array}$$

Type II error:

$$\begin{array}{c}{P}_{\text{Miss}}=P[A_0|H_1]\gg \gg P[X\le 7.31|H_1]\\ \\ \gg \gg P[Z\le -0.23]\gg \gg \approx 0.4090\end{array}$$

Total error:

$$P_{\mathrm{ERR}}=P_{FA}\cdot 0.95+P_{\text{Miss}}\cdot 0.05\approx 0.02749$$ Link to original

03 Theory - MAP criterion proof

Theory 2 - MAP criterion proof

Explanation of MAP criterion - discrete case

First, we show that the MAP design selects for $A_0$ all those $x$ which render $H_0$ more likely than $H_1$. This will be used in the next step to show that MAP minimizes probability of error.

Observe this calculation:

$$\begin{array}{cccc}& P[H_i|X=x]& =P[X=x|H_i]\cdot \frac{P[H_i]}{P[X]}& \text{(Bayes’ Rule)}\\ & & & \\ & & =P_{X|H_i}(x)\cdot \frac{P[H_i]}{P[X]}& \text{(Conditional PMF)}\end{array}$$

Recall the MAP criterion:

$$P_{X|H_0}(x)\cdot P[H_0]\ge P_{X|H_1}(x)\cdot P[H_1]$$

Divide both sides by $P[X]$ and apply the above Calculation in reverse:

$$\gg \gg P[H_0|X=x]\ge P[H_1|X=x]$$

This is what we sought to prove.

---

Next, we verify that the MAP design minimizes the total probability of error.

The total probability of error is:

$$P_{\mathrm{ERR}}=P[A_1|H_0]\cdot P[H_0]+P[A_0|H_1]\cdot P[H_1]$$

Expand this with summation notation (assuming the discrete case):

$$\gg \gg \sum _{x\in A_1}{P}_{X|H_0}(x)\cdot P[H_0]+\sum _{x\in A_0}{P}_{X|H_1}(x)\cdot P[H_1]$$

Now, how do we choose the set $A_0\subset ℝ$ (and thus $A_1=A_0^c$) in such a way that this sum is minimized?

Since all terms are positive, and any $x\in ℝ$ may be placed in $A_1$ or in $A_0$ freely and independently of all other choices, the total sum is minimized when we minimize the impact of placing each $x$.

So, for each $x$, we place it in $A_0$ if:

$$P_{X|H_0}(x)\cdot P[H_0]\ge P_{X|H_1}(x)\cdot P[H_1]$$

That is equivalent to the MAP criterion.

Link to original

04 Theory - MC design

Theory 3 - MC design

• Write $C_{10}$ for cost of false alarm, i.e. cost when $H_0$ is true but decided $H_1$.
  • Probability of incurring cost $C_{10}$ is $P_{FA}\cdot P[H_0]$.
• Write $C_{01}$ for cost of miss, i.e. cost when $H_1$ is true but decided $H_0$.
  • Probability of incurring cost $C_{01}$ is $P_{\text{Miss}}\cdot P[H_1]$.

Expected value of cost incurred

$$E[C]=P[A_1|H_0]\cdot P[H_0]\cdot C_{10}+P[A_0|H_1]\cdot P[H_1]\cdot C_{01}$$

MC design

Suppose we know:

• Both prior probabilities $P[H_0]$ and $P[H_1]$
• Both conditional distributions $P_{X|H_0}(x)$ and $P_{X|H_1}(x)$ (or $f_{X|H_0}(x)$ and $f_{X|H_1}(x)$)

The minimum cost (MC) design for a decision statistic $X$:

$$A_0=\text{set of }x\text{ for which:}$$

Discrete case:

$$P_{X|H_0}(x)\cdot P[H_0]\cdot C_{10}\ge P_{X|H_1}(x)\cdot P[H_1]\cdot C_{01}$$

Continuous case:

$$f_{X|H_0}(x)\cdot P[H_0]\cdot C_{10}\ge f_{X|H_1}(x)\cdot P[H_1]\cdot C_{01}$$

Then $A_1=\{x\in ℝ|x\notin A_0\}$.

The MC design minimizes the expected value of the cost of error.

MC minimizes expected cost

Inside the argument that MAP minimizes total probability of error, we have this summation:

$$P_{\mathrm{ERR}}=\sum _{x\in A_1}{P}_{X|H_0}(x)\cdot P[H_0]+\sum _{x\in A_0}{P}_{X|H_1}(x)\cdot P[H_1]$$

The expected value of the cost has a similar summation:

$$E[C]=\sum _{x\in A_1}{P}_{X|H_0}(x)\cdot P[H_0]\cdot C_{10}+\sum _{x\in A_0}{P}_{X|H_1}(x)\cdot P[H_1]\cdot C_{01}$$

Following the same reasoning, we see that the cost is minimized if each $x$ is placed into $A_0$ precisely when the MC design condition is satisfied, and otherwise it is placed into $A_1$.

Link to original

05 Illustration

Example - MC Test: Smoke detector

MC Test: Smoke detector

Suppose that a smoke detector sensor is configured to produce $8\mathrm{V}$ when there is smoke, and $0\mathrm{V}$ otherwise. But there is background noise with distribution $𝒩(0,3^2)$.

Suppose that the background chance of smoke is $5\%$. Suppose the cost of a miss is $50\times$ the cost of a false alarm. Design an MC test for the alarm.

Compute the expected cost.

Solution

We have priors:

$$P[H_0]=0.95P[H_1]=0.05$$

And we have costs:

$$C_{10}=1C_{01}=50$$

(The ratio of these numbers is all that matters in the inequalities of the condition.)

The MC condition becomes:

$$\begin{array}{c}\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\cdot 0.95\cdot 1\stackrel{?}{\ge }\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot 0.05\cdot 50\\ \\ \gg \gg e^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\stackrel{?}{\ge }{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot \frac{2.5}{0.95}\\ \\ \gg \gg -\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2\stackrel{?}{\ge }-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2+\ln \left(\frac{2.5}{0.95}\right)\\ \\ \gg \gg x^2\stackrel{?}{\le }{(x-8)}^2-18\ln \left(\frac{2.5}{0.95}\right)\\ \\ \gg \gg x\le 2.91\end{array}$$

---

Therefore, $A_0$ is $x\le 2.91$, while $A_1$ is $x>2.91$.

The decision rule is: activate alarm when $x>2.91$.

---

Type I error:

$$\begin{array}{c}{P}_{FA}=P[A_1|H_0]\\ \\ \gg \gg P[X>2.91|H_0]\gg \gg \approx 0.1660\end{array}$$

Type II error:

$$\begin{array}{c}{P}_{\text{Miss}}=P[A_0|H_1]\\ \\ \gg \gg P[X\le 2.91]\gg \gg \approx 0.04488\end{array}$$

Total error:

$$P_{\mathrm{ERR}}=P_{FA}\cdot 0.95+P_{\text{Miss}}\cdot 0.05\approx 0.1599$$

---

PMF of total cost:

$$P_C(c)=\{\begin{array}{cc}0.002244& c=50\\ 0.1577& c=1\\ 0.840056& c=0\end{array}$$

Therefore $E[C]=0.27$.

Link to original