RE: st: How to find extreme values

["Nakelse, Tebila (AfricaRice)" <T.Nakelse@cgiar.org>]

Thank you Nick, 
The  correct multiplier  I had  in mind is 1.5*iqr , as it is set in -extremes- as default, and not  1.25*iqr.
Anyway, -extremes- is very suitable to list  the extremes value. But I don't know if -extremes- can help to  create a variable to identify the extreme value in the dataset.
I guess it would be useful because of the fact that one can use to correct the extreme value (I mean  Replacement with a given estimate) or exclude the extreme value  from an estimation ( a regression for example). 


Tebila 



-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
Sent: Tuesday, March 27, 2012 9:38 AM
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: How to find extreme values

See also -extremes- from SSC. For example,

. sysuse auto
(1978 Automobile Data)

. extremes price

  +--------------+
  | obs:   price |
  |--------------|
  |  34.   3,291 |
  |  14.   3,299 |
  |  18.   3,667 |
  |  68.   3,748 |
  |  66.   3,798 |
  +--------------+

  +---------------+
  |  64.   12,990 |
  |  28.   13,466 |
  |  27.   13,594 |
  |  12.   14,500 |
  |  13.   15,906 |
  +---------------+

. extremes price, iqr sep(0)

  +-----------------------+
  | obs:    iqr:    price |
  |-----------------------|
  |  53.   1.559    9,690 |
  |  55.   1.580    9,735 |
  |  41.   1.877   10,371 |
  |   9.   1.877   10,372 |
  |  11.   2.349   11,385 |
  |  26.   2.401   11,497 |
  |  74.   2.633   11,995 |
  |  64.   3.096   12,990 |
  |  28.   3.318   13,466 |
  |  27.   3.378   13,594 |
  |  12.   3.800   14,500 |
  |  13.   4.455   15,906 |
  +-----------------------+

The table shows that on the usual boxplot rule there are 12 outliers in -price-, not 8. Tebila missed some overlapping symbols. -graph box price, marker(1, ms(Oh))- makes the fact of overlap easier to see. As I understand it -graph box- does not support jittering (at this moment I am using an ancient Stata).

Also, an outlier being at least 1.25 * iqr away from the nearer quartile is Tebila's convention, but it is not that used by -graph box-. -extremes- allows you to set your own multiplier.

A further note of caution: Suppose you reject outliers as being more than so many "deviations" away from some reference "level", then recalculate (the definitions of quoted terms being up to you). You might need to iterate more than once, as new outliers could be identified at each stage.

All that said, I would rather identify outliers than delete them, as also Maarten advocated.

Nick

On Tue, Mar 27, 2012 at 9:19 AM, Nakelse, Tebila (AfricaRice) <T.Nakelse@cgiar.org> wrote:

> Find below an example of identification and correction of extreme value  .
>
> *** correction of  the variable price
>
> sysuse auto, clear
>
> /* plot to visualize the extreme*/
> graph box price
>
> /* we can distinguish 8 extremes values*/
>
> ***  quartiles of  price
>
> egen Q1_price= pctile(price), p(25)
> egen Q3_price= pctile(price), p(75)
> egen  IC_price= iqr(price)
>
> ***Identification of extreme value
>
> gen touse=1 if (price< Q1_price-1.25*IC_price| price> 
> Q3_price+1.25*IC_price) & missing(price)==0 recode touse . =0
>
> tab touse
>
> ***Correction of the price
> gen pricec =price
> replace pricec  =Q1_price-1.25*IC_price if price < 
> Q1_price-1.25*IC_price & touse==1 replace pricec 
> =Q3_price+1.25*IC_price if price> Q3_price+1.25*IC_price &  touse==1
>
> /* the corrected price box plot to see if the extreme value remain*/ 
> *graph  box pricec

Maarten Buis
>
> On Tue, Mar 27, 2012 at 5:24 AM, Barth Riley <barthriley@comcast.net> wrote:
>> To remove outliers, you could:
>>
>> preserve
>> replace var = . if abs(var) >= 1000000 (or some other value) [perform 
>> analyses] restore
>>
>> preserve and restore are added if you want to make a temporary change 
>> to these values
>
> If I were to exclude such observations I would probably do something like:
>
> gen byte touse = abs(var) <= 1e6
> reg y var x if touse
>
> -reg- could be any command, the key is the -if touse- part. The variable touse will contain 0s and 1s such that those non-extreme values get 1 (true) and the extreme values get 0 (false), see:
> <http://www.stata.com/support/faqs/data/trueorfalse.html>. The reason why I prefer this is that it does not destroy any information in my dataset.
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/