Theory 1 - Significance testing
Significance test
Ingredients of a significance test (unary hypothesis test):
— Null hypothesis event
- Identify a Claim
- Then:
is background assumption (supposing Claim isn’t known) - Goal is to invalidate
in favor of Claim — Rejection Region event (decision rule)
is written in terms of decision statistic and significance level is unlikely assuming . is more likely if Claim — Able to compute this
- Usually: inferred from
or - Adjust
to achieve
Significance level
Suppose we are given a null hypothesis
and a rejection region . The significance level of
is: Sometimes the condition is dropped and we write
, e.g. when a background model without assuming is not known.
Null hypothesis implies a distribution
Usually
is unspecified, yet determines a known distribution. In this case
will not take the form of an event in a sample space, . At a minimum,
must determine .
We do NOT need these details:
- Background sample space
- Non-conditional distribution (full model):
or - Complement conditionals:
or
In basic statistical inference theory, there are two kinds of error.
- Type I error concludes with rejecting
when is true. - Type II error concludes with maintaining
when is false.
Type I error is usually a bigger problem. We want to consider
| Maintain null hypothesis | Made right call | Wrong acceptance |
| Reject null hypothesis | Wrong rejection | Made right call |
To design a significance test at
When
- One-tail rejection region:
with or with - Two-tail rejection region:
with