Hello --

I am trying to fit a mixed effect linear model to some data. For context, I have 4 raters (raterid) who each rate 3 notes (noteid) from 10 different learners (residentID). Each learner is rated by the same 4 raters, providing a final score (tscore). Hence, noteid is nested in raters, who are nested in learners.

Data structure:
noteID residentID Tscore reviewer_n
1041901 8 11 1
1041901 8 9 2
1041901 8 12 3
1041901 8 11 4
1041902 8 12 1
1041902 8 12 2
1041902 8 12 3
1041902 8 9 4
1041903 8 11 1
1041903 8 11 2
1041903 8 12 3
1041903 8 10 4
1111901 10 10 1
1111901 10 10 2
1111901 10 11 3
1111901 10 6 4
1111902 10 9 1
1111902 10 8 2
1111902 10 10 3
1111902 10 8 4
1111903 10 11 1
1111903 10 10 2
1111903 10 11 3
1111903 10 6 4

For this project, I would like to estimate inter-rater reliability and examine rater differences.
  1. To estimate Inter-rater reliability and obtain the ICC I used the following syntax:

mixed tscore || residentID: || noteID:
estat icc
Relevant output for Intraclass correlation:

------------------------------------------------------------------------------
Level | ICC Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
residentID | .1368313 .0841699 .0377377 .3905274
noteID|residentID | .1368313 .0841699 .0377377 .3905274
------------------------------------------------------------------------------

Is it correct to assume that for residentID, the proportion variance in tscore between residents is .14? Also, why am I getting identical values for the other nested level of the model (noteID|residentID)? Is my nesting incorrect?
  1. If I want to estimate rater differences, would I use the following syntax?
mixed score i.raterid || residentID: ||noteID:

Because rater is a categorical variable, it provides a reference group. Is it possible to estimate an overall omnibus effect of rater?


Thanks!