Scores

What "objective" can and cannot mean here

A rating tool can be consistent, and that is the most it can be; objectivity would need a fact of the matter, and there is none.

By 4 min readScores

Guides on Scores: How much weight a rating tool result deserves, Every part of a rating result, and what each one is for, What can and cannot be compared, and how

There is no objective rating score, because how a submission looks has no fact of the matter to check a number against, the way a length has a tape. When a rating tool calls itself "objective", it is claiming something narrower: not that the number is true, but that no person formed the opinion.

What the word is standing in for

Read "objective" literally in a rating tool's context and it collapses to "automated." The model did not get tired, did not have a bad day, was not swayed by anything in the room, did not know the person on the other end of the upload. Those are real properties, and they are worth having - a model genuinely does not carry the specific, situational biases a person can. But absence of one kind of bias is not the same as arrival at a fact. Research on human judgements of looks suggests there is no single fact to arrive at: across three experiments on facial attractiveness, Hönekopp (2006) found that "private taste is about as powerful as shared taste," contrary to the prevailing view that standards are largely shared. The model still ran a rubric someone built, mapped its output through a calibration someone chose, and returned a number shaped by decisions made well before your photo was ever uploaded. None of those decisions were arbitrary in a bad-faith sense, but none of them were discovered either - a rubric is a list of what a tool decided to weight and by how much, not a list of facts the tool found lying around.

Swap "objective" for "consistent" and the sentence keeps almost all of its useful content and loses the false promise. A consistent tool returns close to the same output for close to the same input, every time, regardless of who is asking. That is checkable - you can retake, you can resubmit the same file twice, and see whether the number holds. Objectivity in the strong sense is not checkable at all, because there is nothing external to check it against.

Why the distinction actually matters

Once "objective" is understood as "consistent," a whole set of questions become answerable that were unanswerable while the word implied truth. "Is this tool objective" has no honest answer, because the premise is broken - there was never a fact for it to be objective about. "Is this tool consistent" has a real answer, and you can go find it yourself: submit the identical file twice and see how far apart the two numbers land, or retake under matching conditions and check the spread. A tool can score you a flat, disappointing 5 every single time and be highly consistent. A tool can score you generously and inconsistently, swinging a point and a half between two nearly identical photos. Consistency and generosity are unrelated, and neither one is truth - which is exactly why naming the achievable property separately from the word "objective" is worth doing.

This is a narrower claim than the difference between consistency and accuracy in general, which has its own fuller treatment - the point here is specifically about what the marketing word "objective" is smuggling in, not about the full relationship between the two properties.

What replaces "trust the objective score"

Trust the score to the extent it is consistent, and treat the number itself as one tool's particular, undisclosed rubric applied the same way each time - useful for comparing your own results to each other, much less useful as a claim about anything beyond the tool that produced it. A tool that leans hard on the word "objective" in its own copy, without publishing a rubric or any information about its own calibration, is making a claim it cannot support and is not offering the one thing that would let you check it. That pattern, among several others, is worth watching for when deciding how much weight any tool's marketing deserves.

The same substitution - consistent, not objective - holds outside this category too. A model's stated accuracy is a claim about its own evaluation process, not an external fact it discovered, and treating it as one is the same mistake in a different outfit. The one place a genuine fact of the matter does exist nearby is measurement: a length has an actual value a tape can approach, which is precisely why that property lives on a different site than this one - it is the rare case in this space where "objective" is not doing any smuggling at all. A human reviewer, by contrast, never claimed objectivity to begin with; the honesty there is in the label, not in the method. Among automated tools, the ones worth taking seriously are the ones that make consistency checkable rather than asserting objectivity and hoping the word goes unquestioned - Rate Cock publishing a breakdown per axis is one way a tool can be checked rather than merely believed.

Read next

Full archive