Hi,

I am currently trying to measure inter rater reliability for a set of data as below. The raters were asked to rate physical suffering on a scale (1 = unknown, 2 =nil, 3 = mild, 4 = moderate, 5 = severe)

Code:
* Example generated by -dataex-. To install: ssc install dataex
clear
input byte(vignette rater14 rater18 rater29 rater39 rater42 rater47 rater59 rater60 rater64)
 1 3 3 1 3 1 3 3 2 3
 2 5 5 4 5 4 1 5 4 4
 3 3 3 3 3 2 3 3 3 3
 4 5 5 4 5 5 4 5 4 5
 5 3 3 3 4 . 3 3 3 3
 6 3 4 3 5 . 3 4 3 3
 7 5 5 4 5 . 4 4 . 5
 8 3 3 3 4 . 3 3 . 3
 9 3 4 3 3 . 3 4 . 4
10 4 4 2 4 . 2 3 . 4
11 3 4 5 4 . 3 4 . 3
12 4 4 4 4 . 3 3 . 3
13 5 5 5 5 . 5 4 . 5
14 5 5 5 5 . 5 4 . 5
end

I am looking to use a weighed kappa to determine the inter rater reliability, however I am unsure of which weighting to use. I think I understand the difference between linear and quadratic (with linear punishing the difference by being off by any number of categories the same, while quadratic weighting means the penalties increase).

I am unsure of the ordinal weighting suggested in the 'help kappaetc' file and the mathematical explanation is a little over my head.

Could someone please explain how the ordinal weighting option differs from linear and quadratic? And which might be most appropriate for my data? I do not think I want to use linear weighting, as I do want harsher penalities if there is disagreement ranging from 2 (nil) to 5 (severe), however I am unsure what the difference between ordinal and quadratic weighting is.

On the above data:
If using ordinal weights, the Fleiss kappa = 0.4916.
If using quadratic weights, the Fleiss kappa = 0.5272
If using linear weights, the Fleiss kappa = 0.4275

Thanks a lot,
Olivia