Re: st: RE: Mean test in a Likert Scale

[Rob Ploutz-Snyder <robploutzsnyder@gmail.com>]

My 2 cents...when designing these sorts of instruments...

I was trained that a true likert scale doesn't label each of the
points in the 5-point (or other) scale, but instead has only TWO
labels at each extreme.  For example:

I like Statalist..............      Completely Disagree   1  2  3  4
5    Completely Agree

This is in CONTRAST to a scale that would label each and every point
(sometimes called "likert-type" or "modified-likert") for example:

1=completely disagree
2=disagree
3=neutral
4=agree
5=completely agree

With true likert scales, while still not continuous in scale, the
distance between each category in a true likert scale is not
subjective.  The delta between "1" and "2" is the same as the delta
between "2" and "3" etc.  and it is assumed that survey respondents
can appreciate this.  The same cannot be assumed about the difference
between "completely disagree" and "disagree" being equal to the delta
between "disagree" and "neutral."

So in that way, a  true-likert scale removes some of the subjectivity
on the deltas and seems to achieve a more proper ordinal scale as
opposed to purely categorical.

Still doesn't justify using parametric statistical techniques...
However, most well-vetted Sociology or Psychological instruments are
designed to use multiple questions that, together, are used to measure
a particular construct.  Social scientists don't usually intend to
compare responses on single questions, but instead ask many questions
that cluster together, often verified by exploratory or confirmatory
factor analysis, where "factor scores" are then created to capture the
overall construct of interest.  These factor scores can be derived by
different methods, the simplest being a mean of the items that cluster
together, but usually by more sophisticated regression-based methods
that weigh each item according to how well it correlates with the
overall factor structure.  These factor scores are continuously
scaled, unlike the individual items that were used to derive them, and
it is these factor scores that are often analyzed by various
parametric statistical techniques.

Whether or not the factor scores  are normally distributed in the
population (the real question) is dependent on the particulars of each
research study, but I don't categorically deny that the assumption is
invalid.


On Fri, Aug 31, 2012 at 10:38 AM, David Radwin <dradwin@mprinc.com> wrote:
> I agree with Nick that "don't use means for ordinal data" is a purist
> stance, even if it is more honored in the breach than the observance.
>
> I also agree that collapsing Likert scales as described below is throwing
> away information. But I don't think it is arbitrary to consider "agree"
> and "strongly agree" to be one category and "disagree" and "strongly
> disagree" to be the other category.
>
> There is a separate question of why one would measure agreement on a 5
> point (or 7 point or 101 point or whatever) scale only to later collapse
> it to yes/no, but analysts don't always have the ability to design the
> survey themselves.
>
> David
> --
> David Radwin
> Senior Research Associate
> MPR Associates, Inc.
> 2150 Shattuck Ave., Suite 800
> Berkeley, CA 94704
> Phone: 510-849-4942
> Fax: 510-849-0794
>
> www.mprinc.com
>
>
>> -----Original Message-----
>> From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-
>> statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
>> Sent: Friday, August 31, 2012 12:46 AM
>> To: statalist@hsphsun2.harvard.edu
>> Subject: Re: st: RE: Mean test in a Likert Scale
>>
>> Thanks for the extra detail, but I don't think it lets me add much to
>> my own advice, except that with a sample size of 20000 very small
>> differences are likely to seem significant by any particular
>> significance test.
>>
>> Nick
>>
>> On Fri, Aug 31, 2012 at 5:08 AM, Leonor Saravia <lmisaravia@gmail.com>
>> wrote:
>> > Dear Nick and David,
>> >
>> > I really appreciate your reply, thank you.
>> >
>> > I read carefully your answers to my questions and as Nick says, my
>> > first question pointed to the fact that there could be the sence in
>> > which computing the mean score of a Likert scale is allowed. I have
>> > seen practical studies were the mean of this kind of scales are
>> > calculated and interpreted. However, there is also literature that
>> > indicates that, as the Likert scales are an ordinal-level measure, you
>> > should not calculate the mean of it. So, I am confused because I do
>> > not understand whether calculating and interpreting the mean of a
>> > Likert scale is correct or not.
>> >
>> > The data I have is desagregated by individual (20000 observations) of
>> > a treatment and a control group, and has the answer for each of the 26
>> > questions, a number between 1 and 5, which are the values of a 5 point
>> > Likert scale from Disagree (1) to Agree (5).
>> >
>> > For instance, the first question (Q1) is: "Chilean people find
>> > entrepreneurial activities socialy valuable" and the possible answers
>> > are:
>> >
>> > 1 - Strongly disagree
>> > 2 - Disagree
>> > 3 - Nor agree nor disagree
>> > 4 - Agree
>> > 5 - Strongly agree
>> >
>> > So, the database has this structure:
>> >
>> > Observation     Group            Q1  Q2  .....  Q25  Q26
>> > 1                     Treatment      1     5   ......   3      1
>> > 2                     Control           3    1   .......  2      5
>> > .
>> > .
>> > 19999             Control          5     2  ........ 4       3
>> > 20000             Treatment      3     2  ......... 5      4
>> >
>> >
>> > From this, one could calculate the mean of Q1 for the treatment and
>> > control group, but I do not know if the number obtained can be
>> > interpreted and even more, if one can test mean differences between
>> > both groups.
>> >
>> > Thank you very much for your help and advice.
>> >
>> > Best regards,
>> >
>> > Leonor
>> >
>> >>I mostly disagree with David here. In particular, his proposal to
>> >>collapse the Likert scales just throws away information in an
>> >>arbitrary manner.
>> >
>> >>I don't think his advice is even consistent. If it's OK to treat means
>> >>of Bernoulli distributions as valid arguments for a t test, why is not
>> >>OK to treat means of Likert scales as if they were?
>> >
>> >>It's true that the reference case for a t test is two paired normal
>> >>distributions, and Likert scales can not be normal if only because
>> >>they are _not_ continuous, but there is always a judgment call on
>> >>whether summaries of the data will in practice work similarly.
>> >
>> >>A fair question is what exactly kind of advice is Leonor seeking? The
>> >>question presumably isn't really whether it is possible -- clearly it
>> >>is possible -- but perhaps somewhere between "Is it correct?" and "Is
>> >>it a good idea?"
>> >
>> >>Leonor's question appears to have the flavour of "I gather that this
>> >>is wrong. but is there a sense in which this is allowed?" The long
>> >>answer has to be that Leonor should tell us much more about the data
>> >>and the problem in hand if a good answer is to be given. If means make
>> >>sense as summary statistics, then comparing means with a t test is
>> >>likely to work well, but watch out.
>> >
>> >>David is clearly right in alluding to a purist literature in which you
>> >>are told as a matter of doctrine that ordinal data shouldn't be
>> >>summarised by means and so mean-based tests are also invalid. When
>> >>acting as academics, the same people work with grade-point averages
>> >>just like anybody else, at least in my experience.
>> >
>> >>There is also a pragmatist literature which points out that despite
>> >>all that, the sinful practice usually works well. Compare the t-test
>> >>with e.g. a Mann-Whitney-Wilcoxon test and it's very likely that the
>> >>P-values and z- or t-statistics will point to the same substantive
>> >>conclusion and indicate just about the same quantitative effect. It's
>> >>also likely that doing both tests will be needed because some reviewer
>> >>has been indoctrinated against t-tests here, and especially if anyone
>> >>is working with a rigid threshold (e.g. a 5% significance level).
>> >
>> >>Also, the behaviour of t-tests in cases like this can always be
>> >>examined by simulation, so no-one need be limited by textbook dogma
>> >>(or wickedness).
>> >
>> >>Nick
>> >
>> > On Fri, Aug 31, 2012 at 12:13 AM, David Radwin <dradwin@mprinc.com>
>> wrote:
>> >> Leonor,
>> >>
>> >>No, you can't correctly calculate the mean of an ordinal-level measure
>> >>like the Likert scale you describe, although plenty of people do it
>> >> anyway.
>> >>
>> >> But you can use -ttest- with these data if you first collapse each
>> >> variable to a dichotomous (dummy) variable, because the mean of a
>> >> dichotomous variable is identical to the proportion where the value
> is
>> 1.
>> >> As a guess, you might set the highest two values to 1, the lowest two
>> >> values to 0, and the middle value to missing to calculate the
>> proportion
>> >> agreeing or somewhat agreeing.
>> >>
>> >> David
>> >> --
>> >> David Radwin
>> >> Senior Research Associate
>> >> MPR Associates, Inc.
>> >> 2150 Shattuck Ave., Suite 800
>> >> Berkeley, CA 94704
>> >> Phone:             510-849-4942
>> >> Fax: 510-849-0794
>> >>
>> >> www.mprinc.com
>> >>
>> >>
>> >>> -----Original Message-----
>> >>> From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-
>> >>> statalist@hsphsun2.harvard.edu] On Behalf Of Leonor Saravia
>> >>> Sent: Thursday, August 30, 2012 3:23 PM
>> >>> To: statalist@hsphsun2.harvard.edu
>> >>> Subject: st: Mean test in a Likert Scale
>> >>>
>> >>> Hello,
>> >>>
>> >>> I'm working with a survey that presents 26 questions and each of
> them
>> >>> has as possible answer a 5 point Likert scale from Desagree (1) to
>> >>> Agree (5). This survey was applyed for a treatment and a control
>> >>> group.
>> >>>
>> >>> As far as I know, it is possible to analyze the information given
> only
>> >>> by the proportions of each answer; for instance, 25% agrees, 50%
>> >>> desagree, or so.
>> >>>
>> >>> I have two questions that maybe one of you have had before:
>> >>>
>> >>> a) Is it possible to calculate the mean score of a sample (treatment
>> >>> or control group) - adding the individual answers - when one is
>> >>> working with a Likert scale?
>> >>>
>> >>> b) If it is possible to calculate a mean score of a sample when
> using
>> >>> a Likert scale, to compare the answers of the treatment versus the
>> >>> control group, is it well done if I use the 'ttest' command?
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/