Scores

What a 1 and a 10 are pinned to

Every scale needs something at each end; rating tools pin theirs to a percentile, a rubric maximum, or nothing at all.

By 4 min readScores

Guides on Scores: How much weight a rating tool result deserves, Every part of a rating result, and what each one is for, What can and cannot be compared, and how

A scale anchor is whatever a rating tool has decided a 1 and a 10 mean - a rubric maximum, a percentile of its users, or, most often, nothing at all. Without one, "8 out of 10" is a position on a line with nothing nailed down at either end.

A ruler has two fixed points and everything between them is defined relative to those two. A rating scale needs the same thing, and a floating scale is one that can drift without anyone being able to prove it moved.

The three kinds of anchor

A rubric maximum. The 10 is defined as the theoretical top of a described rubric - every listed criterion satisfied to its fullest stated degree. This is the most defensible anchor because it does not depend on who else has used the tool. It is also the rarest, because writing a rubric specific enough to have a genuine ceiling is more work than most tools put in.

A percentile. The 10 is defined relative to the tool's own population - the top slice of everyone who has submitted. This anchor is honest about what it is (a ranking device) but dishonest if presented as an absolute judgement, because the same submission would sit at a different point on the scale if the population changed size or composition. Whether a score is reporting a percentile or an absolute value is worth knowing before you treat either end as fixed.

Nothing. No stated rubric ceiling, no disclosed population, just a number that has always been generated somewhere between 1 and 10. This is the majority case. The ends exist because the interface needs a minimum and a maximum, not because anyone defined what they represent.

What an unanchored scale does

It floats. A tool with no fixed end can let its outputs drift upward for months - more 8s, more 9s, fewer 5s - without the scale itself registering the change, because there is nothing at either end to check the drift against. Score inflation is what this looks like in practice: a slow shift where nobody adjusted the anchor, because there was never one to adjust. Describing each level is one fix that has been tested: in a randomised trial on an online labour market, Garg and Johari (2018) found that positive-skewed verbal rating scales "significantly reduce rating inflation and are much more informative" about quality.

It also makes a 10 an empty threat. If a rubric maximum is not defined, nobody inside the company can say with confidence what submission would actually earn a 10, which usually means nobody does, and the top of the scale sits permanently and conveniently out of reach.

The question worth asking

Before trusting a result, ask: what would a 10 on this tool actually have to be? A tool that can answer with a specific, checkable description - every rubric line maxed, or top one percent of a stated population over a stated window - has an anchor. A tool that answers with something closer to "as good as it gets" has told you the top is decorative.

The same question works pointed at the bottom. A rubric-anchored 1 means every listed criterion failed outright, which is a genuinely rare submission. A percentile-anchored 1 means bottom of a population that mostly does not submit its worst material in the first place, which is a different and much softer claim. An unanchored 1 means whatever the model happened to output on its lowest day, and that is not a definition at all. Without an official anchor, readers supply their own: Tversky and Kahneman (1974, Science) described "adjustment from an anchor" as a heuristic that leads to "systematic and predictable errors", and a previous score makes an easy private anchor.

Where this cashes out

Anchoring is one of the more checkable things about a tool, and it separates tools that will tell you what a number means from tools that will not. Rate Cock states its axis maximums against a named rubric rather than a moving population, which is the kind of detail that lets a reader independently check whether a given 9 is close to that stated ceiling or a long way off it. Most tools give you neither the rubric nor the population and expect the number alone to carry the claim. What a vision model is actually doing before any of this mapping happens is a separate layer entirely - the anchor question sits above the model, in the choice of how its raw output gets translated into a number on a fixed scale.

Anchoring is a rating-tool problem specifically, because a physical measurement does not need one - a length in centimetres is anchored by the tape, not by a rubric someone wrote. A subjective judgement from a person carries the same anchoring question in a different form: what a specific reviewer rates a 10 is still whatever that person's personal ceiling happens to be, disclosed or not, and asking them what a 10 would take is exactly the same move as asking a tool.

None of this tells you whether a given result is good. It tells you whether the scale it is printed on has an end worth trusting, and that turns out to be most of the work.

Read next

Full archive