Re: st: xtlogit: panel data transformation's recast to double makes model incomputable

[Nick Cox <njcoxstata@gmail.com>]

So, it is not a problem with zero or negative values. I don't have an
explanation for your difficulties with -ladder- otherwise.

I don't think anyone needs a test to choose a transformation with 2
million values that are manifestly highly skew.

Your -summarize- results give min, max and nine quantiles in between.
Transformations of interest are here all monotonic, so preserve order,
so it is sufficient just to fire up Mata to look at what quantiles
become, e.g.

: y
             1
     +----------+
   1 |     .02  |
   2 |     .62  |
   3 |    1.66  |
   4 |    3.09  |
   5 |    8.66  |
   6 |   18.68  |
   7 |   33.48  |
   8 |   57.38  |
   9 |   74.08  |
  10 |  123.49  |
  11 |  785.37  |
     +----------+

: log10(y)
                   1
     +----------------+
   1 |  -1.698970004  |
   2 |  -.2076083105  |
   3 |    .220108088  |
   4 |   .4899584794  |
   5 |    .937517892  |
   6 |   1.271376872  |
   7 |   1.524785449  |
   8 |   1.758760544  |
   9 |   1.869700974  |
  10 |   2.091631791  |
  11 |   2.895074308  |
     +----------------+

: sqrt(y)
                  1
     +---------------+
   1 |  .1414213562  |
   2 |  .7874007874  |
   3 |  1.288409873  |
   4 |  1.757839583  |
   5 |  2.942787794  |
   6 |  4.322036557  |
   7 |  5.786190457  |
   8 |  7.574958746  |
   9 |  8.606973917  |
  10 |  11.11260546  |
  11 |  28.02445361  |
     +---------------+

A glance shows that roots are not strong enough, but logarithms do
much better, which matches a lot in economic thinking. However, it's
one thing to make a distribution more symmetric; the question of how
it behaves in a model remains.

Nick

On 3 April 2013 16:09, Tom <tommedema@gmail.com> wrote:
> "close" is a price, and therefore it is always positive. -sum close,
> detail- follows (after dropping some outliers in which I am not
> interested):
> http://pastebin.com/25gm94i0
>
> Because close is always positive I would like to return to the -ladder
> close- issue. It turns out that when I supply -noadjust-, i.e. execute
> -ladder close, noadjust- it does return some details:
> http://pastebin.com/m8Z4YmKd
>
> -noadjust- is simply the -noadjust-  option of -sktest-, used by
> -ladder-. The help file says that this is what it does:
>
>     " noadjust suppresses the empirical adjustment made by Royston
> (1991) to the overall chi-squared and its significance level and
> presents the unaltered test as described by D'Agostino, Balanger, and
> D'Agostino (1990)."
>
> But does anyone have a clue why this adjustment would prevent ladder
> from returning anything? Also see the results of -sktest close- versus
> -sktest close, noadjust-:
> http://pastebin.com/tPUf3xE6
>
> Tom
>
> On Wed, Apr 3, 2013 at 4:54 PM, Nick Cox <njcoxstata@gmail.com> wrote:
>> The transformations that -ladder- tries are typically only defined for
>> positive variables. As I recall, many of your variables are variously
>> positive and negative.
>>
>> The usual transformations for right-skewed variables, such as square
>> root and logarithmic, are thus no use in your case.
>>
>> This is precisely why cube root and inverse hyperbolic functions were
>> suggested earlier/
>>
>> On the face of it that is not an explanation of why all the
>> transformations fail, but I'd not be inclined to think that -ladder-
>> will help your problem.
>>
>> I'm afraid that you have keep reminding us of basics -- e.g. what is
>> -close- and what are its descriptive statistics -- as even those
>> following your threads can't be expected to keep track of all earlier
>> posts.
>>
>> Nick
>>
>> On 3 April 2013 15:33, Tom <tommedema@gmail.com> wrote:
>>
>>> Anyway, I think I may have a very important pointer that could help in
>>> finding the cause of this issue:
>>> when I run -ladder close- it returns absolutely nothing, i.e.:
>>>
>>>  ladder close
>>>
>>> Transformation         formula               chi2(2)       P(chi2)
>>> ------------------------------------------------------------------
>>> cubic                  close^3                    .            .
>>> square                 close^2                    .            .
>>> identity               close                      .            .
>>> square root            sqrt(close)                .            .
>>> log                    log(close)                 .            .
>>> 1/(square root)        1/sqrt(close)              .            .
>>> inverse                1/close                    .            .
>>> 1/square               1/(close^2)                .            .
>>> 1/cubic                1/(close^3)                .            .
>>>
>>> I believe this is very much related to the problems I am having with
>>> this variable, close. If I can determine why ladder refuses to give me
>>> any transformations, then I may be able to figure out what is the
>>> source of this issue.
>>>
>>> I am not just trying to get an answer, I am interested in why ladder
>>> is not returning any transformations because this may give me
>>> information on the root cause of the problem.
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/



--
Nick
njcoxstata@gmail.com
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/