Hi all,
I am running into an issue dealing with generateng inter-rater agreement for a nested survey data. The data contains responses to a 25-question culture survey from over 100,000 respondents from 75 organizations. Those respondents come from 12 different job positions and 10 different units (positions are nested in units, units are nested in organizations). What I am trying to test is how inter-rater agreement across different job positions within same unit of an organization might drive performance outcomes. I attempted to try kappa2 and other types of kappa commands however the current data structure is in long format and I could not convert it to a wide format, which requires over 100,000 variables.

I am new to this type of tests and am wondering whether you have any suggestions?

Data set looks like this:

Organization ... unit ... position ... respondentID .... Q1...Q2...Q3...Q4....

Thanks!
David