Score reliability, from one tool and across several

Reliability means getting about the same score again when nothing relevant has changed. You can check it three ways: submit the same file twice, submit fresh retakes under fixed conditions, and compare against a second tool. Each check isolates a different source of wobble, and none of them says whether the score is right.

What reliability is, and what it is not

Reliability is a property of repetition. A tool is reliable if the same situation produces nearly the same output, and unreliable if the output wanders. That is all.

It is not accuracy. A tool that always returns 8.2 for everything is perfectly reliable and tells you nothing, and consistent is not accurate is worth reading before anything below. Reliability is a necessary condition for a score to mean something, not a sufficient one. If the number moves by a point when nothing changed, no reading of it survives.

It is also not the same as how many digits the display shows. How a displayed value gets rounded is a separate question, covered elsewhere on this site, and nothing here depends on it.

The useful framing comes from measurement science. NIST's handbook, in its gauge study guidance, makes the point that the number of displayed digits does not indicate resolution, and that resolution has to be checked under actual measurement conditions. A rating tool is a gauge with an unusually chatty display, and the same discipline applies.

Three sources of wobble

When two results differ, the gap has three possible origins, and they stack.

• The tool. Same file in, different number out. This comes from sampling inside a model, from load balancing across model versions, or from a pipeline that is not deterministic.
• Your submission. A new photo of the same subject is a new file. Distance, tilt, light and state all moved a little even if you tried to hold them still.
• The tool's scale. Two different tools put the same photo at different numbers because their rubrics and centres differ.

Each check below peels one of these off. Run them in order and you can say how much of any gap belongs to which.

Check one: the same file twice

Submit one identical file several times, on the same day, with nothing else changed. Whatever spread you see is the tool's own noise, because the input did not vary.

A worked example. Five submissions of one saved file return 7.1, 7.4, 6.9, 7.2 and 7.4. The mean is 7.2 and the range is 0.5. So on this tool, on this day, a difference of a few tenths between two results is ordinary, and a half-point gap tells you nothing yet.

Two outcomes are worth recognising at once. If all five come back identical, the tool is either deterministic or caching, and retakes that agree suspiciously well deserve their own suspicion. If the five span two points, the tool is too noisy to track anything small, and you can stop there.

The dedicated treatment of the idea, with its borrowed name, is in test-retest reliability for rating tools, and the cheapest way to run it is the reproducibility test. Do it on a day other than the one you care about as well. A file that scores differently a week later has told you the tool moved, not you.

Check two: fresh retakes under fixed conditions

Now take five new photos of the same subject under the conditions you can control: same device, distance, light, framing, state. Submit each once.

Suppose they return 6.8, 7.5, 7.0, 6.4 and 7.3. The mean is 7.0 and the range is 1.1.

Compare that with check one. The same file spread over 0.5; fresh photos spread over 1.1. Roughly half of the retake spread is the tool, and the other half comes from the fact that no two photos are identical however carefully you hold still. That second half is submission noise. It is the part you can reduce by tightening your protocol and the part you can never reduce to zero.

This is the range that matters when you compare any two real results. The error bar nobody prints on a rating turns it into a usable interval, and how big a gap is a real gap says when a difference clears it. For this example, two future results 0.6 apart sit comfortably inside what retakes already produce, so the gap does not mean the subject changed.

A caution on sample size. Five is enough to see the shape of the spread. It is not enough to pin down its edges, so treat the range as a lower bound on how far results can stray.

Check three: a second tool

Once a single tool's own noise is known, bring in a second one. This check asks a different question. It does not ask whether either tool repeats itself; it asks whether they tell the same story.

Take six photos and submit each to both tools. Tool A returns 6.2, 7.0, 7.4, 5.8, 6.6 and 7.8, and tool B returns 7.4, 7.9, 8.1, 7.0, 7.2 and 8.9. Call the photos 1 to 6 in that order.

Subtract A from B photo by photo and the gaps are 1.2, 0.9, 0.7, 1.2, 0.6 and 1.1. The mean gap is 0.95 and it barely moves. That is a constant offset, which is a scale difference: B is simply more generous by about a point.

Then compare order instead of values. Tool A ranks the photos 6, 3, 2, 5, 1, 4 from best to worst. Tool B ranks them 6, 3, 2, 1, 5, 4. Only photos 1 and 5 trade places, and on A they sit 0.4 apart, which is inside the retake range from check two.

The reading is clean. These two tools disagree in level and agree in order, which is the benign kind of disagreement. Measuring agreement between two tools without statistics software shows the same hand method with more care, and when two tools agree explains why even good agreement proves less than it feels like it does.

Why agreement is weaker evidence than repetition

Two tools agreeing is reassuring, and the reassurance is thin. Many tools sit on similar hosted models, so two matching results may be one judgement seen twice. Agreement also says nothing about the rubric both of them might share. Repetition of one tool on one file is stronger evidence of reliability than agreement between two tools, because it controls everything except the tool.

The statistical literature draws the same line. Koo and Li (2016), in their guideline for selecting and reporting intraclass correlation coefficients, put values below 0.5 as poor, 0.5 to 0.75 as moderate, 0.75 to 0.9 as good and above 0.9 as excellent, applied to the confidence interval rather than the single estimate. They also warn that no universal standard exists, and that the form you choose depends on whether the final measure is one reading or the mean of several. You do not need the coefficient to use the idea. You need to know that how much repeated results agree is a quantity with bands, and that "close enough" is a judgement you are making, not a threshold the tool has met.

Where the object being compared is a physical measurement rather than a rated one, the questions of repeat-measurement practice belong to measuremycock.com's method hub, which handles that side.

Putting the three together

Lay the three numbers side by side and you have a reliability profile. Same file: 0.5. Fresh retakes: 1.1. Second tool: offset of about 0.95, order mostly preserved.

From that profile you can say specific things. A gap under about half a point on this tool is not a finding. A gap over a point and a half between two of your own results, under fixed conditions, probably is. A second tool will read about a point higher and will mostly order your photos the same way, so cross-tool comparison should be done on rank, not on the number.

Reliability also changes over time, which is where it quietly fails. A tool that was steady in March may shift in June, and the way you find out is repeating check one on a saved file and comparing it with what you got before. How tools differ in what they say about uncertainty is a clue to whether they expect you to run these checks at all. Tools that print a confidence figure are rarely printing a repeat-to-repeat spread, as what a confidence value is explains.

Some tools are easier to test than others. Rate Cock reports six axes rather than a total, so you can run each check axis by axis and see which axis is noisy rather than only whether the total is. A tool that returns one number gives you one column to examine, and auditing a total-only tool covers what you can still do.

Where reliability runs out

Everything above assumes the tool is a stable instrument you are probing. Sometimes it is not. Models get updated without notice, scales get retuned, and a tool that was reliable last quarter can be reliable around a different centre now. Researchers have documented the pattern for large language models directly: Chen, Zaharia and Zou (2023) found that the behaviour of GPT-3.5 and GPT-4 varied greatly across a few months, with one task's accuracy falling from 84% in March 2023 to 51% in June. A rating tool built on such a model inherits the possibility, and a reliable-looking week is no promise about the next one.

For the machine-side questions, such as how hosted models produce variation and what that does to accuracy claims, aipenis.com's accuracy hub is the owner. For how people who review such tools describe confidence in their own verdicts, ratepenis.com's reviews hub covers it.

The practical habit is small. Keep one fixed file. Resubmit it every month or so and write down the number and the date. If the number holds, your older results stay comparable. If it steps, everything before the step lives on a different scale, and you now know the date it happened.