st: strange -multproc- results

["Feiveson, Alan H. (JSC-SK311)" <alan.h.feiveson@nasa.gov>]

Hi - I have been using -multproc- to control the false discovery rate (FDR) on 27 significance tests. As given in the help file, there are several methods to chose from for controlling the FDR: liu1,liu2,simes,yekutieli, and krieger.


            method()          Step type    FWER/FDR    Definition or source
           userspecified     One-step     Either      pcor() option
           bonferroni        One-step     FWER        pcor=puncor/m
           sidak             One-step     FWER        pcor=1-(1-puncor)^(1/m)
                                                      (or Sidak, 1967)
           holm              Step-down    FWER        Holm, 1979
           holland           Step-down    FWER        Holland and Copenhaver, 1987
           liu1              Step-down    FDR         Benjamini and Liu, 1999a
           liu2              Step-down    FDR         Benjamini and Liu, 1999b
           hochberg          Step-up      FWER        Hochberg, 1988
           rom               Step-up      FWER        Rom, 1990
           simes             Step-up      FDR         Benjamini and Hochberg, 1995 (or
                                                      Benjamini and Yekutieli, 2001
                                                      (first method))
           yekutieli         Step-up      FDR         Benjamini and Yekutieli, 2001
                                                      (second method)
           krieger           Step-up      FDR         Benjamini, Krieger and Yekutieli, 2001

So I tried liu1, liu2, simes, yekutieli, and krieger to see what difference it would make with a specified FDR of 0.05. The two liu's and yekutieli were about the same (4 rejections, critical P-value about 0.002. But the simes and krieger were completely different (simes: 14 rejections, critical p-vlaue = 0.026) and (krieger: 15 rejections, critical p-value = 0.051). The latter two look too good to be true, especially the Krieger, where the critical p-value is actually higher than the specified FDR rate. 

Anyone know what's going on here? Am I doing this correctly? What assumptions are there for Krieger, for example, that do not hold for the first three?

Al Feiveson

If anyone wants to try it - here's the data:

  h	         se	          z	         pv
1761.754	419.83	4.196352	.0000271
.0613758	.0171379	3.58129	.0003419
.0431283	.0134256	3.212402	.0013163
.0503242	.0159218	3.160711	.0015738
.0662939	.0223807	2.962102	.0030555
.0388915	.0133944	2.903562	.0036894
.0793423	.0274955	2.885645	.0039061
.0353006	.0129654	2.722682	.0064754
868.2667	323.8542	2.681042	.0073393
.0491057	.0184865	2.6563	.0079003
893.4875	341.7166	2.614703	.0089305
.0310786	.0131919	2.355878	.018479
.034885	.0150222	2.322223	.0202209
.032349	.0144647	2.236412	.0253248
.0302972	.0139816	2.166937	.0302396
.0295493	.0165115	1.789618	.0735153
-.0201654	.0129031	-1.562831	.1180923
.0255772	.017776	1.438857	.1501909
.0150236	.0122768	1.223738	.2210511
.0187261	.0165989	1.128154	.2592548
-.013579	.0127488	-1.065118	.2868224
.0142208	.0142817	.9957331	.3193798
.0099778	.0117025	.8526207	.3938697
.0071485	.0095379	.7494824	.4535665
.0130484	.021209	.6152284	.5384039
-.0067718	.0137157	-.4937261	.6214996
.0028293	.0090162	.3138045	.7536695

. multproc ,method(liu1) pvalue(pv) puncor(.05)

Method: liu1
Uncorrected overall critical P-value: .05
Number of P-values: 27
Corrected overall critical P-value: .00262649
Number of rejected P-values: 4

. multproc ,method(liu2) pvalue(pv) puncor(.05)

Method: liu2
Uncorrected overall critical P-value: .05
Number of P-values: 27
Corrected overall critical P-value: .00255198
Number of rejected P-values: 4

. multproc ,method(simes) pvalue(pv) puncor(.05)

Method: simes
Uncorrected overall critical P-value: .05
Number of P-values: 27
Corrected overall critical P-value: .02592593
Number of rejected P-values: 14

. multproc ,method(yekutieli) pvalue(pv) puncor(.05)

Method: yekutieli
Uncorrected overall critical P-value: .05
Number of P-values: 27
Corrected overall critical P-value: .00190351
Number of rejected P-values: 4

. multproc ,method(krieger) pvalue(pv) puncor(.05)

Method: krieger
Uncorrected overall critical P-value: .05
Number of P-values: 27
Corrected overall critical P-value: .05102041
Number of rejected P-values: 15


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/