Code:
. /* Congruent Results */ . . power twoproportions 0.146, n1(254426) n2(255237) alpha(.05) power(0.8) test(chi2) effect(ratio) Performing iteration ... Estimated experimental-group proportion for a two-sample proportions test Pearson's chi-squared test H0: p2 = p1 versus Ha: p2 != p1; p2 > p1 Study parameters: alpha = 0.0500 power = 0.8000 N = 509,663 N1 = 254,426 N2 = 255,237 N2/N1 = 1.0032 p1 = 0.1460 Estimated effect size and experimental-group proportion: delta = 1.0191 (ratio) p2 = 0.1488 . display "MDE (in %)) " 100*(r(p2) - r(p1))/r(p1) MDE (in %)) 1.9056807 . . prtesti 254426 0.146 255237 .1488, level(95) // use MDE from above Two-sample test of proportions x: Number of obs = 254426 y: Number of obs = 255237 ------------------------------------------------------------------------------ | Mean Std. err. z P>|z| [95% conf. interval] -------------+---------------------------------------------------------------- x | .146 .0007 .1446279 .1473721 y | .1488 .0007044 .1474193 .1501807 -------------+---------------------------------------------------------------- diff | -.0028 .0009931 -.0047465 -.0008535 | under H0: .0009931 -2.82 0.005 ------------------------------------------------------------------------------ diff = prop(x) - prop(y) z = -2.8193 H0: diff = 0 Ha: diff < 0 Ha: diff != 0 Ha: diff > 0 Pr(Z < z) = 0.0024 Pr(|Z| > |z|) = 0.0048 Pr(Z > z) = 0.9976 . display "Estimated ratio = " 100*(r(P2) - r(P1))/r(P1) Estimated ratio = 1.9178082
But then he gave me a counterexample, where a statistically significant difference is smaller than the MDE:
Code:
. /* Incongruent Results From Experiments */ . prtesti 254426 0.146 255237 .148, level(95) Two-sample test of proportions x: Number of obs = 254426 y: Number of obs = 255237 ------------------------------------------------------------------------------ | Mean Std. err. z P>|z| [95% conf. interval] -------------+---------------------------------------------------------------- x | .146 .0007 .1446279 .1473721 y | .148 .0007029 .1466224 .1493776 -------------+---------------------------------------------------------------- diff | -.002 .000992 -.0039443 -.0000557 | under H0: .000992 -2.02 0.044 ------------------------------------------------------------------------------ diff = prop(x) - prop(y) z = -2.0161 H0: diff = 0 Ha: diff < 0 Ha: diff != 0 Ha: diff > 0 Pr(Z < z) = 0.0219 Pr(|Z| > |z|) = 0.0438 Pr(Z > z) = 0.9781 . display "Estimated ratio = " 100*(r(P2) - r(P1))/r(P1) // SS diff < MDE Estimated ratio = 1.369863
0 Response to Puzzling Post Hoc Power Calculation
Post a Comment