The Value of Accepting the Null Hypothesis
1 Background
In standard frequentist models, we cannot formally accept the null hypothesis \(H_0\), but can only reject, or fail to reject, \(H_0\).
Bayesian models allow one to both accept and reject \(H_0\) (Kruschke and Liddell 2018).
Accepting \(H_0\) may be very scientifically valuable, and may have consequences for affirming similarity, universality, or treatment invariance (Gallistel 2009; Morey, Homer, and Proulx 2018).
The ability to accept \(H_0\) may also lead to a lower likelihood of the publication bias that results from frequentist methods predicated upon the rejection of \(H_0\) (Kruschke and Liddell 2018).
Lastly, the ability to accept \(H_0\) means that one is not only looking for statistically credible–or statistically significant–results as substantively important findings, but that results supporting the null hypothesis can also be seen as substantively important findings that contribute to theory and practice.
This handout is written from a Bayesian perspective. However, even from a traditional frequentist statistical perspective, it may be helpful to think about the value of results that are not statistically significant.
A finding of a null result is dependent on having enough statistical power that one might plausibly detect an effect were an effect to exist.
Most of the research teams that I work with are addressing hard problems like: child abuse; harsh parenting; mental health challenges; and substance use.
Many of these issues are complicated, long-standing, and sometimes seemingly intractable.
I think that in any statistical model, some associations end up not being statistically credible, or in frequentist models, not statistically significant. Unfortunately, I think this is often seen as a “failure” of the statistical model, or as a rejection of the conceptual underpinnings of a particular project in its entirety.
I’ve been hoping that statistical modeling might be seen more as a process of discernment: Which results are credible or significant? Which results are not credible or significant? How does this pattern of results help us understand the issues that we are working with more deeply? What does this imply for our future work? What aspects of our current work do we need to strengthen in the future?
After all, we are usually trying to address longstanding and difficult problems. Not every result in every model is going to be statistically credible or significant, and I think finding results that are statistically not credible, or statistically insignificant can be informative.
2 Important Substantive Cases
The Value of Accepting the Null Hypothesis \(H_0\)
| case | description | H_0 | example |
|---|---|---|---|
| Equivalence Testing | Equivalence Of 2 Treatments Or Interventions | $$\beta_1 = \beta_2$$ | The effect of Treatment 1 is indistinguishable from the effect of Treatment 2 (especially important if one treatment is much more expensive, or time consuming than another). |
| Equivalence Testing | Equivalence Of 2 Groups On An Outcome | $$\bar{y_1} = \bar{y_2}$$ or in multilevel modeling $$u_0 = 0$$ | People identifying as men and people identifying as women are more similar than different with regard to psychological processes (Hyde2005). |
| Retiring Interventions | There Is No Evidence That Intervention X Is Effective | $$\beta_{intervention} = 0$$ | Evidence consistently suggests that a particular treatment has near zero effect. |
| Contextual Equivalence | Equivalence of a Predictor Across Contexts (Moderation) | $$\beta_{interaction} = 0$$ or in multilevel modeling $$u_k = 0$$ | Warm and supportive parenting is equally beneficial across different contexts or countries. |
| Family Member Equivalence | Equivalence of a Predictor Across Family Members | $$\beta_{parent1} = \beta_{parent2}$$ | Parenting from one parent is equivalent to parenting from another parent |
| Full Mediation | Association of x and y Is Completely Mediated; No Direct Effect | $$\beta_{xmy} \neq 0$$ $$\beta_{xy} = 0$$ | The relationship of the treatment and the outcome is completely mediated by mechanism m. |
| No Mediation | No Indirect Effect; Association of x and y Is Not Mediated by m | $$\beta_{xmy} = 0$$ $$\beta_{xy} \neq 0$$ | The relationship of the treatment and the outcome is not mediated at all by mechanism m. |
| Theory Simplification | Removing An Association From A Theory | $$\beta_x = 0$$ | There is no evidence that x is associated with y. |
| Theory Rejection | Rejecting A Theory | $$\beta_{theory} = 0$$ | There is strong evidence (contra Theory X) that x is not associated with y. |