Code:
. /* Congruent Results */
.
. power twoproportions 0.146, n1(254426) n2(255237) alpha(.05) power(0.8) test(chi2) effect(ratio)
Performing iteration ...
Estimated experimental-group proportion for a two-sample proportions test
Pearson's chi-squared test
H0: p2 = p1 versus Ha: p2 != p1; p2 > p1
Study parameters:
alpha = 0.0500
power = 0.8000
N = 509,663
N1 = 254,426
N2 = 255,237
N2/N1 = 1.0032
p1 = 0.1460
Estimated effect size and experimental-group proportion:
delta = 1.0191 (ratio)
p2 = 0.1488
. display "MDE (in %)) " 100*(r(p2) - r(p1))/r(p1)
MDE (in %)) 1.9056807
.
. prtesti 254426 0.146 255237 .1488, level(95) // use MDE from above
Two-sample test of proportions x: Number of obs = 254426
y: Number of obs = 255237
------------------------------------------------------------------------------
| Mean Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
x | .146 .0007 .1446279 .1473721
y | .1488 .0007044 .1474193 .1501807
-------------+----------------------------------------------------------------
diff | -.0028 .0009931 -.0047465 -.0008535
| under H0: .0009931 -2.82 0.005
------------------------------------------------------------------------------
diff = prop(x) - prop(y) z = -2.8193
H0: diff = 0
Ha: diff < 0 Ha: diff != 0 Ha: diff > 0
Pr(Z < z) = 0.0024 Pr(|Z| > |z|) = 0.0048 Pr(Z > z) = 0.9976
. display "Estimated ratio = " 100*(r(P2) - r(P1))/r(P1)
Estimated ratio = 1.9178082But then he gave me a counterexample, where a statistically significant difference is smaller than the MDE:
Code:
. /* Incongruent Results From Experiments */
. prtesti 254426 0.146 255237 .148, level(95)
Two-sample test of proportions x: Number of obs = 254426
y: Number of obs = 255237
------------------------------------------------------------------------------
| Mean Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
x | .146 .0007 .1446279 .1473721
y | .148 .0007029 .1466224 .1493776
-------------+----------------------------------------------------------------
diff | -.002 .000992 -.0039443 -.0000557
| under H0: .000992 -2.02 0.044
------------------------------------------------------------------------------
diff = prop(x) - prop(y) z = -2.0161
H0: diff = 0
Ha: diff < 0 Ha: diff != 0 Ha: diff > 0
Pr(Z < z) = 0.0219 Pr(|Z| > |z|) = 0.0438 Pr(Z > z) = 0.9781
. display "Estimated ratio = " 100*(r(P2) - r(P1))/r(P1) // SS diff < MDE
Estimated ratio = 1.369863
0 Response to Puzzling Post Hoc Power Calculation
Post a Comment