Scores

When a small movement in your score is worth noticing

A shift of a few tenths is usually inside the noise; the question is how to tell when it is not.

By 4 min readScores

Guides on Scores: How much weight a rating tool result deserves, Every part of a rating result, and what each one is for, What can and cannot be compared, and how

Usually not. A half-point change is real only if it is clearly bigger than the spread you get from resubmitting the same photo under the same conditions, and if it repeats. Most tools never show that spread, so a move from 7.1 to 7.6 looks like progress whether or not it is.

Compare the change to the spread, not to zero

The instinct is to treat any nonzero delta as meaningful - it went up, so something improved. That treats zero as the baseline for comparison, and zero is the wrong baseline, because a rating tool does not return the identical number on repeated identical inputs. The right comparison is between the observed delta and the spread you would see across a few repeats of the same submission under the same conditions. Metrology has a name for that spread: the international vocabulary of metrology (JCGM VIM) calls it measurement repeatability, meaning precision under a fixed set of repeatability conditions.

If two or three repeats of one photo already produce a range of, say, four tenths of a point in either direction, a half-point change between two different sessions is barely outside that range and should be read with real scepticism. If repeats of the same photo come back nearly identical, a half-point shift between sessions is doing more work and is more likely to reflect something real. The spread you would see on repeats is the error bar a tool does not print, and estimating it is the entire prerequisite for reading any delta honestly.

Why the display makes small changes look meaningful

A score is presented as a single clean figure to one decimal place, with nothing beside it indicating a range. That presentation invites the reader to treat every digit as load-bearing, when in most tools the first decimal place is closer to a rounded internal value than to a precise, stable measurement. Nothing in the interface warns you that 7.1 and 7.6 might both be reasonable outputs for the same underlying submission on different days - the format looks equally confident whether the change is real or noise.

This is a separate question from whether a gap between two different submissions is significant, which depends on comparing across subjects rather than across time. How big a gap between two scores needs to be before it counts as real covers that comparison; a half-point change over time on your own results is asking whether the same submission moved, which is a narrower and in some ways easier question to answer.

When a delta does clear the noise

A change is worth taking seriously when it is noticeably larger than your estimated repeat spread, when it persists across more than one fresh submission rather than showing up once, and when nothing about your submission conditions changed in a way that could explain it on its own. A half-point shift that shows up once and does not repeat on a second attempt under the same conditions is much more likely to be an ordinary draw from the same distribution than a real change in whatever the tool is scoring.

One thing a real-looking delta does not automatically mean is that the delta continues. An unusually favourable result is partly luck, and a fresh submission after one often drifts back toward the centre - which is a different phenomenon from noise but produces a similar-looking small movement and is worth ruling out before reading too much into a single jump. Barnett, van der Pols and Dobson (2005) describe regression to the mean as something that "can make natural variation in repeated data look like real change", and note it gets more noticeable as measurement error grows.

Reading it in practice

Establish your own noise floor first with two or three repeats, then judge any later delta against that floor rather than against zero. Rate Cock keeps prior results visible per account, which makes this comparison easier to run than on a tool that only surfaces the latest number and nothing before it.

The same discipline applies to a physical delta as much as a rated one - two measurements taken a month apart need to be checked against normal measurement variance before either one is read as a real change, for the same underlying reason. And the caution runs the other way too when the number comes from a person rather than a model: a human reviewer's mood or attention on a given day is one more source of the kind of noise a half-point delta might just be reflecting, whichever side of the interaction produced the number. Some of that noise, on the automated side, traces back to how sensitive the underlying model is to small differences between two otherwise similar photos - which is worth knowing before assuming a delta says anything about the subject at all.

A half-point is real when it survives being checked against your own spread. Most of the time, it has not been checked at all.

Read next

Full archive