Differences In The Means Of Two Groups

Differences May Be Statistically Significant Even With Overlapping Confidence Intervals

1 Background

Intuitively, if the confidence intervals of two variables overlap, we would expect that the two variables would not differ to a degree that is statistically significant.

However, as the brief example below illustrates, it is possible for two variables to have overlapping confidence intervals, yet to be different to a degree that is statistically significant.

library(Statamarkdown)

2 Demonstration

2.1 Set Up The Data

clear all // clear the workspace

set seed 3846 // set random seed

set obs 1000 // 1000 empty observations

generate x1 = rnormal(100, 10) // x1 has mean of 100, sd of 10

generate x2 = rnormal(102, 10) // x2 has mean of 102, sd of 10

list in 1/10 // list out some data

save demo.dta, replace

Number of observations (_N) was 0, now 1,000.

     +---------------------+
     |       x1         x2 |
     |---------------------|
  1. | 110.8965   94.62458 |
  2. | 85.56382   96.55556 |
  3. | 104.4178   106.0406 |
  4. | 90.79031   82.07703 |
  5. | 108.6776   81.65119 |
     |---------------------|
  6. | 114.3565   101.9648 |
  7. | 87.86876   91.34626 |
  8. | 92.02374   121.2981 |
  9. | 103.8483   77.34809 |
 10. | 91.34591     107.17 |
     +---------------------+

file demo.dta saved

2.2 Confidence Intervals Overlap

use demo.dta

ci means x1 x2 // confidence intervals of the two variables overlap

    Variable |        Obs        Mean    Std. err.       [95% conf. interval]
-------------+---------------------------------------------------------------
          x1 |      1,000    99.71427    .3288933        99.06886    100.3597
          x2 |      1,000    101.9414    .3214891        101.3105    102.5722

2.3 t-test Of Mean Differences Is Statistically Significant

use demo.dta

ttest x1 == x2 // t-test finds significant differences between x and x2

Paired t test
------------------------------------------------------------------------------
Variable |     Obs        Mean    Std. err.   Std. dev.   [95% conf. interval]
---------+--------------------------------------------------------------------
      x1 |   1,000    99.71427    .3288933    10.40052    99.06886    100.3597
      x2 |   1,000    101.9414    .3214891    10.16638    101.3105    102.5722
---------+--------------------------------------------------------------------
    diff |   1,000   -2.227102    .4587072    14.50559   -3.127242   -1.326961
------------------------------------------------------------------------------
     mean(diff) = mean(x1 - x2)                                   t =  -4.8552
 H0: mean(diff) = 0                              Degrees of freedom =      999

 Ha: mean(diff) < 0           Ha: mean(diff) != 0           Ha: mean(diff) > 0
 Pr(T < t) = 0.0000         Pr(|T| > |t|) = 0.0000          Pr(T > t) = 1.0000