Tools

The tools that return a word or a type

Some tools return a category rather than a score; a category is a coarser and sometimes more honest claim, and it cannot be compared numerically.

By 3 min readTools

Guides on Tools: Every presentation choice on a result page, and what it does, A taxonomy of rating tools by what they output, A method for judging any rating tool before trusting it

A label-not-number tool returns a category - a tier, a type, a bucket - instead of a score, and that is a coarser but often more honest claim. It still counts as a rating in the sense that matters, but a bucket cannot be averaged or ranked like a decimal.

What a category actually claims

A number claims a position on a continuous scale, with a distance between any two values that at least looks meaningful. A category claims membership in a bucket, full stop. "Type B" or "Tier 2" tells you which group the submission landed in and nothing about how far it sits from the boundary of the next one - a submission just inside a bucket and one just outside it can be nearly identical, and the label will not show you that.

That is a narrower claim than a score, and in one respect a more honest one: it does not imply a precision the underlying judgement does not have. A tool with real uncertainty around its internal value is arguably better off saying "type B" than manufacturing a 6.4 that looks steadier than it is.

What you can do with it

You can track whether the bucket changes over time, which is a real, checkable signal - moving from one category to another across several submissions means something moved, even without a magnitude attached. You can also use it the way a rank-only result works - as a coarse placement rather than a precise one, useful for a rough read and not much finer than that.

What you cannot do with it

You cannot average it, rank it precisely against another submission in the same bucket, or treat the gap between two categories as equal to the gap between two others - the same ordinal-versus-interval problem that numeric scores already have, just more visible here because there is no decimal pretending otherwise. You also cannot compare it across tools; one tool's "Tier 2" and another's have no defined relationship, the same problem that makes a numeric 7 on two different tools not the same claim either. Words do not fix meaning even with a guide attached: Budescu, Broomell and Por (2009) found readers' interpretations of the IPCC's verbal uncertainty labels "deviated significantly" from the official definitions, even when the definitions were in front of them. Nor does a category convert into a physical unit - a bucket is a judgement, not a measurement in centimetres, and treating it as a rough stand-in for one is a category error the label itself does nothing to prevent.

A different, related design puts a word next to a number rather than instead of it - the "Good," "Great," "Elite" labels some tools stack on top of a decimal are a presentation choice on a scalar tool, not the categorical design covered here, worth keeping apart even though they use similar words.

Reading a categorical result honestly

Treat the bucket as the whole claim, not a rounded stand-in for a number the tool declined to show you - most categorical tools do not have a hidden decimal underneath, the bucket is the actual output. Rate Cock sits at the other end of that choice, publishing per-axis numbers rather than buckets, which is worth naming as a contrast: the underlying model is not what decides whether an output is a number or a category, the tool's own design is, and a bucket you can act on beats a decimal you cannot every time.

Read next

Full archive