Tools

Whether a vote settles a disagreement

Two out of three is not a verdict when the two share a scale or a backbone; majority only means something if the voters are independent.

By 2 min readTools

Guides on Tools: Every presentation choice on a result page, and what it does, A taxonomy of rating tools by what they output, A method for judging any rating tool before trusting it

When three rating tools score the same photo and two agree, the majority is only worth trusting if the two agreeing tools got there separately. Two tools built on the same hosted model can agree for reasons unrelated to your submission, and then two-to-one is really one opinion counted twice.

Majority needs independent voters

A vote of two against one is informative when each vote was cast on its own evidence. It is not informative when two of the three votes came from the same source wearing different labels. Two tools sharing a hosted model underneath can converge for reasons that have nothing to do with your submission being genuinely closer to one score than another - the majority is really a majority of one.

Before reading "two agree" as a finding, it is worth asking what would make the two agreeing tools different enough to count as separate observers: different rubric, different calibration, different training lineage. If the answer is "not much", the count is two-to-one in name and one-to-one in substance. Measurement science has a name for what a shared backbone contributes: systematic measurement error (JCGM VIM, entry 2.17), the component that "in replicate measurements remains constant or varies in a predictable manner", which more readings from the same source cannot average out.

The lone dissenter is not automatically wrong

The instinct to override the outlier is usually backwards. A tool that disagrees with a shared backbone might be measuring something the other two are not - a different weighting, a different subjective read, or simply more noise on this particular file. Discarding it because it lost a vote throws away the one data point that was actually independent of the other two.

What would make three tools a real panel

Three tools whose scores you can trust as three separate opinions need three separate foundations: different models, different rubrics, ideally different companies with no shared codebase. What a given hosted model is actually doing under a tool's rubric is usually invisible from the results page, which is exactly why "different tool" and "different foundation" are not the same claim - two of your three might share a foundation you have no way to see. That is closer to what happens when you add a human judge to a set of model-based tools - a person is not drawing on the same hosted vision model at all, so their agreement or disagreement with the pair carries different weight than a third automated tool would. It is a different kind of independence again from a physical measurement taken with a tape, which does not vote on a scale at all and is not part of this comparison.

None of this tells you which of the three numbers is "right" - rating tools do not have a right answer to converge on, only more or less consistent and disclosed ones. It tells you what two-out-of-three is actually worth: sometimes a real majority, often one opinion that got counted twice. Rate Cock is one instance among many that could sit in either role in a three-tool set, depending on what it shares with whichever tools it is compared against.

Read next

Full archive