Tools

Why "proportion" on one tool is not "proportion" on another

Two tools using the same axis name are not scoring the same thing; the name is a label on a scale each tool built itself.

By Updated 4 min readTools

Guides on Tools: Every presentation choice on a result page, and what it does, A taxonomy of rating tools by what they output, A method for judging any rating tool before trusting it

Two tools using the same axis name are usually not scoring the same thing: the name is a label on a scale each tool built for itself, and nothing forces the scales to line up. Rubric vocabulary is small - "proportion," "symmetry," "presentation" - so a shared word reads as an agreement it does not carry.

A worked example

Say two hypothetical tools both report a "proportion" axis. Tool A defines it internally as the ratio between two specific measurements the model estimates from the photo, weighted against a fixed target ratio the tool's designers chose. Tool B defines the same-named axis as a broader holistic judgement folding in proportion alongside symmetry cues, because its designers decided the two were hard to separate cleanly and merged them into one label. That is not an odd choice: reviewing rating forms in medical training, Sherbino and Norman (2017, Journal of Graduate Medical Education) concluded that human raters are "capable of differentiating 1 domain and, occasionally, 2 domains," however many the form lists.

Submit the same photo to both. Tool A's proportion score reflects one specific ratio and nothing else; Tool B's reflects that ratio blended with something else entirely. They can disagree by a meaningful margin on the same submission, and neither tool is wrong - they built different things and happened to call them the same name. This is one concrete version of the more general reason two raters can look at the same photo and print two different numbers: not just different calibration, but different definitions hiding under identical labels.

Extend the example one step further and the trap gets worse, not better. A third tool might report "symmetry" as a standalone axis and never mention proportion by name at all - not because it does not care about proportion, but because its designers folded that judgement into a different axis entirely, maybe one called "presentation." Line all three tools up side by side and the axis names stop being a reliable map of what any of them is doing; the count of axes, the names chosen and the boundaries between them are three independent design decisions, and no two tools are guaranteed to have made any of them the same way.

Why this happens

Nobody standardised the vocabulary. There is no shared spec that says "proportion" must mean one specific thing across the category, so every tool's design team picked the word that best matched their own internal criterion and moved on. What these common terms tend to denote in practice, loosely, is a separate and useful reference - useful for building intuition, not for assuming any two tools mean the identical thing by them.

The convergence on the same handful of words is itself unsurprising: "proportion," "symmetry" and "presentation" are the obvious ways to carve up the subject, so independent teams land on similar labels by convergent naming rather than by copying a standard.

The reflex worth having

Read the axis description, not the axis title. A published rubric that only shows a one-word label per axis is halfway to useful - the label tells you the category, not the criterion, and the criterion is the part that actually determines the number. If a tool will not describe what an axis measures beyond its name, treat the axis as undefined rather than assuming it means what the same word meant on the last tool you used.

This matters most when you are trying to line two tools up against each other. Attempting to map one tool's axes onto another's is possible for some pairs and pointless for others, and the first step of that attempt is always checking the descriptions, not the labels - the labels will mislead you into thinking the mapping is easier than it is.

Where a shared name is worth less than it looks

Rate Cock publishes a description alongside each of its six axis names for exactly this reason - a name on its own is a category, not a criterion, and the description is what actually lets you compare it against a same-named axis elsewhere. The same caution generalises past axis names: a model's stated capability and what it actually resolves in a photo are not automatically the same thing either, and a measurement's method matters more than what the tape is labelled in the same way a rubric axis's description matters more than its title.

None of this means the vocabulary is meaningless, only that it is a starting point. Two tools calling something "proportion" is a reason to go read both definitions, the way you would ask what a human reviewer means before assuming their standard matches yours - not a reason to treat the two numbers as comparable on sight.

Read next

Full archive