The significance level in the hypothesis test assesses how far from the center of probability we want to reject the null hypothesis. In other words, we say that the significance level is the probability of rejecting H0 when H0 is true, and hypothetically we want this probability near 0. The significance level is denoted with the letter alpha. Two common value of alpha are 0,05 and 0,01.
Example: In the fair coin hypothesis tests, where we flip 10 coins, the probability of flipping a number of heads equal to 0,1,2,8,9,10 are 0,11 that, is the sum of the probability of all the single events. In this case, if we want to reject the null hypothesis on those number of heads, we can say that we have alpa=0,11
We already know that the critical region is the value that we use to reject H0, in the example is 0,1,2,3,8,9 and 10. Other important terms are the critical values that separate the critical region from the non-critical region. In this case are 3 and 9. We can look the definition above plotted on the image1 (note that the number of flips is written at the right of each bar).
The bar chart above was plotted on an experiment of 1000 flip. The table version of the probability obtained from the experiment is in the image2. Remember that the probability is near the real distribution because this is a sample, but it’s not exactly the same. For instance, the exact probability of 1 head on 10 flips is 0.010. With 1000 experiments, we obtained 0.008.
Another important thing about the critical region is the difference between one tail (left&right) and two tails.
In the image1, we have the critical region at the right and left of the mean value. In this case, we made a two tails test.
If we have a critical region only at the right of the mean, so in the example of image1, only from 8 and above, we have a right tail test.
If we have a critical region only at the left of the mean, so in the example of image1, only from 3 and belove, we have a left tail test.
In the real world we say if for us is a problem to go higher and lower the mean, or only higher, or only lower. In made the test, we need this information to select the correct interval of confidence.
It’s essential to know the difference between Statistical and Practical significance. Statistical significance shows that an effect exists in a study. Practical significance indicates that the result is large enough to be meaningful in the real world. In other words, sometimes, even if an effect arises in the study, this effect is too small to impact the real world.