Re: st: Bootstrapped Standard Errors

[Stas Kolenikov <skolenik@gmail.com>]

This gets very complicated, in my opinion. Conceptually speaking, your
standard errors should reflect all sources of uncertainty about the
final treatment effect estimate you obtained. If by virtue of your
sample design, the controls were oversampled, then it would have been
relatively easier for you to find the controls for your treated, thus
reducing your bias and mean squared error. The procedure to get the
standard errors should take into account the fact that the original
data were obtained by random sampling; replacing the sampling weights
with CEM weights eliminates this source of uncertainty, and probably
biases your resulting estimate. If the bootstrap is to mimic the
sampling process, then it should remove some of the observations in
both the control and the treatment group, thus making it more
difficult to find matches (may be as far as having to drop the whole
CEM stratum if it does not have matches, which has a drastic effect on
the estimate and its degrees of freedom), and increasing your bias/MSE
within the bootstrap subsamples. I am not entirely sure I see how to
handle all of these issues and incorporate all different sources of
uncertainty. Following Don Rubin's tradition, Gary King does not
address the question of variance estimation, not even in the JASA
paper, let alone the Stata Journal paper. I guess you would want to
write to Gary King and ask about his advice on your data situation.
You can utilize -bsweights/bs4rw- with the CEM weights as the
baseline, but how far off your resulting standard errors would be from
the true sampling variability is beyond me. If I were you, I would set
up a simulation to see if the standard errors are at least
approximately correct; you would have to make some smart choices
regarding the sample design and the decisions within CEM steps,
though. In the best case, you will see 95% coverage of your confidence
intervals (rarely happened to me in my simulations). In the worst
case, you will have material for a paper titled "Failures of CEM
method: how on earth does one get the standard errors right???".

On Thu, Mar 22, 2012 at 12:22 PM, Danny Dan <danny2011dan@gmail.com> wrote:
> Hello Professor Kolenikov,
>
> I have seen your article in the Stata journal ("Resampling variance
> estimation for complex survey data") and was also trying to use it in
> my work. However, I am little confused about the use of the
> appropriate weights in using your tool as because I am not using the
> usual sampling weights as available in the raw data but using weights
> generated after implementing a matching method.
>
> I am trying to use weights that are generated after coarsened exact
> matching (CEM). After running CEM matching it gives both CEM_STRATA
> and CEM_WEIGHTS (where wt=1 for treated and some positive values for
> the controls). My question is can I use CEM_WEIGHTS and CEM_STRATA in
> using your tool?
>
> Please let me know whether this would be appropriate.
>
> Also for your information, I also have weights from my raw data
> (Primary sampling unit and strata for variance estimation (VARPSU and
> VARSTR)).
>
> How shall I set the svy (-svyset-)? Shall I do the following as shown
> in your example:
>
> egen upsu = group(strata psu)
> . svyset upsu [pw=finalwgt], strata(cstrata)
>
> In my case, shall I replace finalwt with cem_weights and cstrata with
> cem_strata and psu with varpsu.
>
> Please let me know.
>
> Thank you for your reply.
>
> Best ,
>
> Dan
>
> On Wed, Mar 21, 2012 at 8:41 PM, Danny Dan <danny2011dan@gmail.com> wrote:
>> Hello All,
>>
>> I have a question:
>>
>> I know that bootstrapping cannot be applied with weights, however, is
>> there anyway in STATA that after doing a weighted regression (like
>> regress `depvar' `indepvar' [weight=wt],  or, probit  `depvar'
>> `indepvar' [weight=wt]) I can use bootstrapping option separately to
>> generate the standard errors?
>>
>> I am asking this because I need to generate bootstrapped standard
>> errors because of its unknown structure in my model. To be more
>> precise, I am doing a 2-stage estimation and trying to use
>> bootstrapping to generate standard errors in the 2nd stage.
>>
>> Please let me know if I am not clear with my question then I will try
>> to clarify it further for the ease of comprehensibility.
>>
>> Please help.
>>
>> Thank you.
>>
>> Best,
>>
>> Dan
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/statalist/faq
>> *   http://www.ats.ucla.edu/stat/stata/
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/



-- 
Stas Kolenikov, also found at http://stas.kolenikov.name
Small print: I use this email account for mailing lists only.

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/