Level of Significance (alpha - α) and Critical Value

Even before we look into level of significance and how it helps to derive critical value, let's understand the concept of probability in terms of distribution curve. Recall that probability refers to the likelihood of an event. Maximum area under any distribution curve is always 100% (or 1 in terms of probability). Any section of area from this area will have a probability between 0 and 1. If we have to define a sample data from entire distribution, we say its probability of occurrence will be 0.05 or 0.025 or 0.1 but never more than 1.

Significance level is a threshold value which researcher should set before collecting the data. We set this value beforehand to avoid output based bias which results in determining what criteria we should use. Alpha value (or α) is defined in terms of probability (or chances) under the data distribution. This value (or level) is used to determine whether or not we should reject the null hypothesis. It is the probability of rejecting the null hypothesis when it is true. If our sample data produce values that meet or exceed this threshold, then we have sufficient evidence to reject the null hypothesis; if not, we fail to reject the null (we never “accept” the null). For example, a significance level of 0.05 indicates a 5% risk of concluding that a difference exists (between population statistic and test statistic) when there is no actual difference.

A critical value is derived based on the level of significance and the statistical test. It is a point scale of the test statistic beyond which we reject the null hypothesis.

In other words, critical value is the value of the test statistic that marks the boundary of your rejection region. It is the least "extreme" value of the test statistic that is still in the rejection region (i.e. the value which would cause you to just reject the null hypothesis). Any test statistic that is more extreme (less consistent with the null hypothesis in the direction of the alternative) will be in the rejection region and any that is less extreme (more consistent with the null hypothesis) will not be in the rejection region. Critical values can be used to do hypothesis testing in the following way:

1. Calculate the test statistic

2. Calculate critical values based on significance level

3. Compare test statistic with critical values.

If test statistic is more than critical value, we reject null hypothesis in favor of alternate hypothesis. It is easy to visually determine if we can reject the null hypothesis from a figure similar to given below. But, we need to define a numeric measure if we are working in a complex production environment. This measure is known as p-value.

Summarizing using a graphical example - As in the figure above, shaded region takes 5% of the area under the curve. Any test statistic which falls in that region has sufficient evidence to reject null hypothesis in favor of alternate hypothesis. The rejection region is bounded by a specific z-value called the critical value (Zcrit or Z*).

Manually finding z-score: Calculating z* is same as calculating z-score for any area under the curve. From the normal table, find the z-score corresponding to 5% of the area under the curve which is equal to 1.645.