A tool is not a fixed instrument

A ruler does not change length between uses. A rating tool is not a ruler, whatever its display suggests, and treating a score from a year ago as directly comparable to a score from today assumes a stability the tool never promised.

Where drift comes from

Three sources, and a given tool can have any or all of them running at once.

Rubric edits. A tool's criteria are product decisions, and product decisions change. An axis gets added, dropped, or reweighted, and the internal judgement that produces a number is now built from a different formula than it was last year, even if nobody outside the company noticed the change happen.

Model or mapping changes. The model doing the underlying assessment can be swapped or retrained, and separately, the mapping from that internal judgement onto the ten-point display can be retuned - shifted upward for retention, tightened to reduce a ceiling problem, or adjusted for any internal reason a user is never told. How far a silently updated model can move is on record: Chen, Zaharia and Zou (2023) found GPT-4's accuracy at identifying prime numbers fell from 84% in its March 2023 version to 51% in June, concluding that the "same" LLM service "can change substantially in a relatively short amount of time." What calibration actually means for a tool covers what has to hold fixed for a tool to stay calibrated at all; drift is what happens whenever one of those things stops holding.

Population shift, for percentile-mapped tools. If a tool's number is a statement about rank within its own history of results, then a shift in who is using the tool moves the number even when nothing about the rubric or the model changed. Percentile versus absolute scoring is the distinction that decides whether this source of drift applies to a given tool at all.

What it means for comparison

A score from January and a score from December are, in the strict sense, readings from two different instruments that happen to share a brand name and a ten-point display. The size and direction of the difference between them is not something a reader can know from outside the company - it could be larger than the actual change in the subject, smaller, or in the opposite direction, and there is no way to tell which without more information than a results page usually gives.

For a rating tool itself this stays a directional point, not a magnitude one: no honest number can be put on how much a given tool has drifted without access to its version history, and a plausible-sounding figure here would be worse than none. What can be said is that any comparison spanning more than a few months carries this risk, and the risk grows with the gap.

What to look for

A tool that dates its changes - a changelog, a version note, any visible marker that the rubric or scale moved on a specific date - gives a reader a way to check whether a comparison spans a drift event. A tool that says nothing leaves the reader guessing, which is most of them. Rate Cock publishing full breakdowns on entries at least gives a reader something to spot-check a suspected shift against, by comparing a public result from before and after a suspected change, rather than trusting the number alone.

The same instability by a different name shows up wherever a scoring or reviewing process depends on people or models that change over time: a judged review is shaped by whoever is reviewing that week, and the accuracy of an automated estimate depends on a model version that a provider can update without notice. A fixed physical unit does not have this problem at all - a centimetre a year ago is a centimetre today, which is one reason that kind of measurement is a fundamentally different and more stable kind of number than any rating will be.

If a comparison over time matters to you, the honest fix is not trusting the tool to have held still. It is treating a large gap in time as its own variable, worth flagging the same way a changed camera or a changed protocol would be.

A practical workaround

None of the three drift sources can be ruled out from outside a company, but one cheap check narrows the uncertainty: keep a fixed reference photo and resubmit it alongside every new result. If the reference photo's score moves between two dates, some form of drift happened on the tool's side in that window, whatever the cause. If it holds steady, the tool has not moved, at least not in a way that touched this particular submission, and a later change in your own result is more likely to be about the subject than about the instrument. It is not proof of which of the three sources moved, only that something did, but that is already more than the results page alone will tell you.