Severity Calculator

1. What is "severity" and "severely tested"
2. Severity criterion
3. How to interpret severity
4. Examples

    What is "severity" and "severely tested"

Severe testing is a meta-statistical principle in statistical inference which addresses some common issues faced by users of statistical data and statisticians when interpreting p-values (significance levels) and observed power in post-hoc analyses of tests. In particular it makes fallacies related to tests of significance that are overly sensitive or not sensitive enough. This severity calculator calculates SEV of any statistical hypothesis of interest.

The idea is that only statistical hypotheses that have passed severe tests should be inferred from the data. Severity is a post-data assessment similar to the p-value / significance level, and observed (post-hoc) power. that severity evaluates the probativeness of a given test with regards to a particular inference. The higher the severity value is, the more warranted the statistical hypothesis is.

Similarly to how statistical power is evaluated at the point of the minimum detectable effect, severity is evaluated at the point of interest. It is denoted by μ₁, which is equal to (μ₀ + y). You can specify the μ₁ you are interested in infering about in the severity calculator interface.

More formally put: data x₀ in test T provides good evidence for inferring a hypothesis H (just) to the extent that H passes severely with x₀, i.e., to the extent that H would (very probably) not have survived the test so well were H false. ^[1]

    Severity criterion

The severity criterion can be formally expressed as ^[1]:

SEV(T(α), d(x₀), μ ≤ μ₁)

where μ₁ = (μ₀ - y), for some y ≥ 0. (T(α), d(x₀)) can be suppressed where there can be no confusion and the abbreviation SEV(μ ≤ μ₁) can be used.

In its fullest, the severity equation looks like so:

SEV(μ ≤ μ₁) = P(d(X) > d(x₀); μ ≤ μ₁ false) = P(d(X) > d(x₀); μ > μ₁))

The value of SEV depends on the outcome of the statistical test and the value you want to infer about (μ). If the statistic of the original test is in the rejection region, then SEV(μ > μ₁) is calculated, estimating the severity with which one can accept the claim (μ > μ₁). If the statistic of the original test is outside the rejection region, then SEV(μ ≤ μ₁) is calculated, estimating the severity of the test passed by the claim (μ ≤ μ₁).

In a severity evaluation, the original statistical design (sample size, power and significance threshold) are not altered, rather we perform a check to see how well a given inference of interest was probed by the data at hand. In other words, how warranted would such an inference be.

    How to interpret severity

Understanding how to interpret severity is key for putting the results from a severity calculator to good use. Interpreting the severity of a given test in relation to a given inference about μ₁ is straightforward: the higher the severity of the test, the higher its probativeness is relative to that μ₁, and the more warranted the inference, be it (μ ≤ μ₁) or (μ > μ₁). Respectively, the lower the severity, the less warranted the statistical inference is.

This is in contrast with the uninformative nature of a low p-value and/or a non-significant result (by itself), as well as the lack of information about what values other than the original null hypothesis are also excluded at a given significance level, for which the use of confidence intervals is necessary.

    Examples

We have a binomial trial with two groups: control (A) and treatment (B), planned for and executed with 1,000 observations in each group, and we have set the significance threshold at α = 0.05. The observed mean event rate in group A of 12% (0.12) and in group B of 15% (0.15).

We are wondering how well corroborated is the claim that the event rate in the treatment group is higher than 16% (0.16) (μ > 16%)? We plug the numbers in our severity calculator and get SEV(μ > 16%) = 0.256443. That is a very low probability: 1 in 4, so making that statistical inference will be poorly justified in most situations.

On the other hand, if we wanted to check how warranted the conclusion that the event rate in the treatment group is larger than 12.4% (0.124) is, we get SEV(μ > 12.4%) = 0.955558 which means that such a claim is well-probed given the data.

Expanding on the first example, let us imagine that the treatment (B) had a mean of 14% instead of 15%, meaning that the test fails to reject the null at α = 0.05. In this case, how warranted is it to accept the hypothesis that μ < 12% given the data at hand? A severity calculation reveals that SEV(μ ≤ 12%) = 0.091793, meaning that such an inference will be very poorly supported.