The problem
A model lab had preference data in eight languages from a previous vendor and no way to tell which language slices were reliable. Reward model behaviour was inconsistent and nobody could explain why.
What we did
We required a written reason for every comparison, in English and in the source language, and published rater agreement per language and per prompt category.
Two languages came back with agreement low enough that we recommended excluding them from the training mix until the rubric was revised. The lab did, and the inconsistency resolved.
Outcome
The lab could weight the mix by slice reliability instead of treating all eight languages as equivalent.
