Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: st: Using random-effects coefficients to predict performance over time
From
Michael Housman <[email protected]>
To
"[email protected]" <[email protected]>
Subject
RE: st: Using random-effects coefficients to predict performance over time
Date
Wed, 1 Aug 2012 16:45:20 +0000
Thanks for the advice! I've spent some time digging in on this front and I'm able to produce graphs that look the way I'd like. But I've run into another snag - the omitted group only seems to produce a straight line instead of a curve. Let me add a bit more detail:
I have data in which each observation represents an employee-date and the dependent variable is a performance metric (e.g., average handle time, schedule adherence, hold time) for call center agents. In essence, I'm trying to model performance and plot the learning curve as a function of "day_of_service" for four different groups of employees.
I've generated a variable called "hire_score_order" that's numbered 1 to 4, representing the four different groups that I want to represent. I've interacted that term twice with day_of_service so I can visually represent the first- and second-order effects. Here's my "xtreg" command and the resulting output for a sample metric:
xtreg aht c.day_of_service##c.day_of_service##i.hire_score_order if hire_score_order ~= ., re
Random-effects GLS regression Number of obs = 242792
Group variable: emp_id Number of groups = 1984
R-sq: within = 0.0049 Obs per group: min = 1
between = 0.1248 avg = 122.4
overall = 0.0622 max = 500
Wald chi2(38) = 1544.57
corr(u_i, X) = 0 (assumed) Prob > chi2 = 0.0000
--------------------------------------------------------------------------------------------------------------------
aht | Coef. Std. Err. z P>|z| [95% Conf. Interval]
---------------------------------------------------+----------------------------------------------------------------
day_of_service | -.0035472 .0302398 -0.12 0.907 -.0628162 .0557218
|
c.day_of_service#c.day_of_service | -9.38e-07 4.63e-06 -0.20 0.839 -.00001 8.13e-06
|
hire_score_order |
2 | 168.1932 48.20808 3.49 0.000 73.70711 262.6793
3 | 20.51885 68.23659 0.30 0.764 -113.2224 154.2601
4 | 156.1946 109.0574 1.43 0.152 -57.55392 369.9431
|
hire_score_order#c.day_of_service |
2 | -2.088015 .5027992 -4.15 0.000 -3.073483 -1.102546
3 | -1.117207 .4928079 -2.27 0.023 -2.083092 -.1513208
4 | -2.408916 1.294864 -1.86 0.063 -4.946802 .1289699
|
hire_score_order#c.day_of_service#c.day_of_service |
2 | .0023866 .0016018 1.49 0.136 -.0007529 .0055262
3 | .0014925 .0014822 1.01 0.314 -.0014126 .0043976
4 | .0040321 .0037677 1.07 0.285 -.0033524 .0114167
|
_cons | 246.4581 81.31057 3.03 0.002 87.09236 405.8239
---------------------------------------------------+----------------------------------------------------------------
sigma_u | 521.47501
sigma_e | 930.20434
rho | .23912442 (fraction of variance due to u_i)
--------------------------------------------------------------------------------------------------------------------
Then, I have STATA calculate the marginal effects for all 4 groups at specific values for "day_of_service". Here's my "margins" command and the resulting output:
margins i.hire_score_order, at(day_of_service=(30(30)150)) post
Predictive margins Number of obs = 242792
Model VCE : Conventional
Expression : Linear prediction, predict()
1._at : day_of_ser~e = 30
2._at : day_of_ser~e = 60
3._at : day_of_ser~e = 90
4._at : day_of_ser~e = 120
5._at : day_of_ser~e = 150
--------------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
---------------------+----------------------------------------------------------------
_at#hire_score_order |
1 1 | 619.9763 20.74882 29.88 0.000 579.3093 660.6432
1 2 | 727.677 40.27612 18.07 0.000 648.7373 806.6168
1 3 | 608.3222 62.3349 9.76 0.000 486.148 730.4964
1 4 | 707.5323 97.22201 7.28 0.000 516.9806 898.0839
2 1 | 619.8673 20.13757 30.78 0.000 580.3984 659.3362
2 2 | 671.3716 41.54202 16.16 0.000 589.9507 752.7924
2 3 | 578.7269 61.98285 9.34 0.000 457.2427 700.211
2 4 | 646.0426 98.66036 6.55 0.000 452.6719 839.4134
3 1 | 619.7567 19.55176 31.70 0.000 581.4359 658.0774
3 2 | 619.3604 44.3594 13.96 0.000 532.4176 706.3033
3 3 | 551.8164 63.02595 8.76 0.000 428.2878 675.345
3 4 | 591.8092 105.5965 5.60 0.000 384.8439 798.7744
4 1 | 619.6444 18.99285 32.63 0.000 582.4191 656.8697
4 2 | 571.6435 47.19628 12.11 0.000 479.1405 664.1465
4 3 | 527.5908 64.51581 8.18 0.000 401.1421 654.0394
4 4 | 544.8319 113.5318 4.80 0.000 322.3136 767.3502
5 1 | 619.5303 18.46233 33.56 0.000 583.3448 655.7159
5 2 | 528.2209 49.33645 10.71 0.000 431.5233 624.9186
5 3 | 506.05 65.87943 7.68 0.000 376.9287 635.1713
5 4 | 505.1108 120.0564 4.21 0.000 269.8045 740.417
--------------------------------------------------------------------------------------
Variables that uniquely identify margins: day_of_service hire_score_order
Finally, I use the "marginsplot" command to plot those marginal effects and I use a number of different graphing options to make the graphs look presentable. Here's the location of some sample graphs that I've posted online for the 6 different performance metrics:
http://assets.wharton.upenn.edu/~housman/graphs/abs_xt_cllcmn_hr_scr_Overall.png
http://assets.wharton.upenn.edu/~housman/graphs/acw_xt_cllcmn_hr_scr_Overall.png
http://assets.wharton.upenn.edu/~housman/graphs/adh_xt_cllcmn_hr_scr_Overall.png
http://assets.wharton.upenn.edu/~housman/graphs/aht_xt_cllcmn_hr_scr_Overall.png
http://assets.wharton.upenn.edu/~housman/graphs/hold_xt_cllcmn_hr_scr_Overall.png
http://assets.wharton.upenn.edu/~housman/graphs/qa_xt_cllcmn_hr_scr_Overall.png
I'm happy with the overall look and feel of the graphs but the problem is that the blue line - representing employees where hire_score_order = 1 - always appears to be a straight line. Is there any sort of specification I can use to tell STATA to represent the blue line as a curve like it does with the green, yellow, and red lines?
Thanks again!
Best,
Michael
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of William Buchanan
Sent: Thursday, June 28, 2012 3:58 PM
To: [email protected]
Subject: Re: st: Using random-effects coefficients to predict performance over time
Hi Michael,
You don't need to create separate group variables. The factor operator i. tells Stata to expand the values into individual indicators. One problem that you've probably noticed is that you have several omitted terms in your model. Reducing the group variable to a single variable will solve that problem (since your syntax below would estimate the day of service variable and it's squared term twice).
That could potentially be one issue with the results that you're finding. If you wanted to know more about this topic Michael Mitchell's recent book on visualizing data analysis models is a phenomenal resource.
HTH,
Billy
Sent from my iPhone
On Jun 28, 2012, at 15:45, Michael Housman <[email protected]> wrote:
> Thanks so much! This is exactly what I was looking for. I've been playing with this functionality for a couple hours and have figured out how to plot the marginal effects.
>
> That said, I can't seem to figure out how to plot anything other than the linear prediction. Here's my code:
>
> xtreg adh controls c.day_of_service##c.day_of_service##i.Group1
> c.day_of_service##c.day_of_service##i.Group2 , re margins ,
> at(day_of_service = (30(10)90)) over(i.Group1 i.Group2)
>
> And here's what I see at the top of the marginal effects table:
>
> Expression : Linear prediction, predict()
>
> The lines in the marginal plot look straight. You'd mentioned "polynomial term interactions" in your response and I'm wondering if there's anything specific I need to specify in order to produce those.
>
> Thanks again!
>
> Best,
> Michael
>
>
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of William
> Buchanan
> Sent: Thursday, June 28, 2012 1:32 PM
> To: [email protected]
> Subject: Re: st: Using random-effects coefficients to predict
> performance over time
>
> Hi Michael,
>
> If you simplify your syntax:
>
> xtreg performance c.day_of_service##c.day_of_service##i.group
> other_covariates, re
>
> You should be able to use the -margins- command, and subsequently -marginsplot-, to plot the relationship for each of the groups for the time variable and the polynomial term interactions.
>
> With regards to storing the coefficients, that is already done for you. Type -ereturn list- after fitting your model to find out what matrices your results are stored in (coefficients generally are stored in e(b)). With regards to the scaling of -x- you can estimate that with -margins- if you use the -at()- option with appropriate values for the variables you are interested in.
>
> HTH,
> Billy
>
>
>
>
> On Jun 28, 2012, at 1:21 PM, Michael Housman wrote:
>
>> Hi,
>>
>> Apologies if this is a novice question but I'm struggling with something and was wondering if the group can help out.
>>
>> I have data on employee performance over time and I'm trying to depict visually the relationship between employee performance and days of tenure on the job. My hypothesis is that there are 3 groups of employees and that these three groups vary in terms of: (1) where their performance starts off, (2) how quickly they learn, (3) how quickly their learning flattens out. In other words, I believe (and the data seems to indicate) that the first derivative of the learning curve is positive and the second derivative is negative.
>>
>> How I'd planned on approaching this problem was running a random-effects regression (because I have time-invariant explanatory variables) and interacting the group dummy variables with the linear and squared term representing days of service. For example, here's some of the code that I've set up (simplified slightly):
>>
>> * Generate the linear and squared terms
>>
>> gen day_of_service = metric_date - hire_date gen day_of_service2 =
>> days_of_service ^ 2
>>
>> * Interact those terms with the group dummy variables
>> * I know this code can be simplified but I'm just using it here as an
>> example
>>
>> gen Group1_dos = Group1 * day_of_service gen Group1_dos2 = Group1 *
>> day_of_service2
>>
>> gen Group2_dos = Group2 * day_of_service gen Group2_dos2 = Group2 *
>> day_of_service2
>>
>> gen Group3_dos = Group3 * day_of_service gen Group3_dos2 = Group3 *
>> day_of_service2
>>
>> * Run random-effects regression (Group 3 is my omitted group)
>>
>> xtreg performance day_of_service day_of_service2 Group1 Group1_dos
>> Group1_dos2 Group2 Group2_dos Group2_dos2 other_covariates, re
>>
>> So this run a random-effects regression where I model the: (1) intercept, (2) slope, and (3) squared term for these groups (relatively to the one omitted group - Group3) and generate a coefficient representing each. That part I understand.
>>
>> What I don't understand is how I can depict this visually. In other words, I'd like to ask STATA to generate 3 separate curves from these coefficient estimates and then plot the lines on a graph. Here's how I imagine that equation would like for each of the 3 groups:
>>
>> Group1: f(t) = day_of_service*t + day_of_service2*t2 + Group1 +
>> Group1_dos*t + Group1_dos*t2
>> Group2: f(t) = day_of_service*t + day_of_service2*t2 + Group2 +
>> Group2_dos*t + Group2_dos*t2
>> Group3: f(t) = day_of_service*t + day_of_service2*t2
>>
>> So the performance on any given day would be indicated by f(x) where x represents the agent's days of tenure on the job. I understand this much. Here's what I don't understand:
>>
>> 1) How can I get STATA to save the coefficients in a data matrix that it keeps in memory?
>> 2) How can I get STATA to then generate a dataset where x runs from, say, 0 to 180 days and then calculates f(t) for these three groups?
>> 3) How can I get STATA to depict this visually (most likely as a line graph)?
>>
>> Truth be told, piece (3) is something I understand but pieces (1) and (2) are the ones I don't get. Thanks in advance for any help!
>>
>> Best,
>> Mike Housman
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *
>> * For searches and help try:
>> * http://www.stata.com/help.cgi?search
>> * http://www.stata.com/support/statalist/faq
>> * http://www.ats.ucla.edu/stat/stata/
>
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
>
>
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
