Theory

Theory 1 - Significance testing

Significance test

Ingredients of a significance test (unary hypothesis test):

• $H_0$ — Null hypothesis event
  • Identify a Claim
  • Then: $H_0$ is background assumption (supposing Claim isn’t known)
  • Goal is to invalidate $H_0$ in favor of Claim
• $R$ — Rejection Region event (decision rule)
  • $R$ is written in terms of decision statistic $X$ and significance level $\alpha$
  • $R$ is unlikely assuming $H_0$. $R$ is more likely if Claim
• $P[R|H_0]$ — Able to compute this
  • Usually: inferred from $f_{X|H_0}$ or $P_{X|H_0}$
  • Adjust $R$ to achieve $P[R|H_0]=\alpha$

Significance level

Suppose we are given a null hypothesis $H_0$ and a rejection region $R$.

The significance level of $R$ is:

$$\begin{array}{cc}\alpha & =P[R|H_0]\\ & \\ & =P[\text{reject }{H}_0|H_0\text{ is true}]\end{array}$$

Sometimes the condition is dropped and we write $\alpha =P[R]$, e.g. when a background model without assuming $H_0$ is not known.

Null hypothesis implies a distribution

Usually $S$ is unspecified, yet $H_0$ determines a known distribution.

In this case $H_0$ will not take the form of an event in a sample space, $H_0\subset S$.

At a minimum, $H_0$ must determine $P[R|H_0]$.

We do NOT need these details:

• Background sample space $S$
• Non-conditional distribution (full model): $f_X$ or $P_X$
• Complement conditionals: $f_{X|H_0^c}$ or $P_{X|H_0^c}$

---

In basic statistical inference theory, there are two kinds of error.

• Type I error concludes with rejecting $H_0$ when $H_0$ is true.
• Type II error concludes with maintaining $H_0$ when $H_0$ is false.

Type I error is usually a bigger problem. We want to consider $H_0$ as “innocent until proven guilty.”

	$H_0$ is true	$H_0$ is false
Maintain null hypothesis	Made right call	Wrong acceptance $\text{T}\text{y}\text{p}\text{e}\text{I}\text{I}\text{E}\text{r}\text{r}\text{o}\text{r}$
Reject null hypothesis	Wrong rejection $\text{T}\text{y}\text{p}\text{e}\text{I}\text{E}\text{r}\text{r}\text{o}\text{r}$	Made right call

---

To design a significance test at $\alpha$, we must identify $H_0$, and specify $R$ with the property that $P[R|H_0]=\alpha$.

When $R$ is written using a variable $X$, we must choose between:

• One-tail rejection region: $x$ with $R(x)\le r$ or $x$ with $R(x)\ge r$
• Two-tail rejection region: $x$ with $|R(x)-\mu |\ge c$