Photos

Making a submission that can go to several tools

If you intend to compare tools, the submission has to be identical to each; here is what identical means and what breaks it.

By 4 min readPhotos

Guides on Photos: A complete protocol for comparable submissions, Everything about submitting more than one image, How to find out how much of your score is you

To compare several rating tools, send every one the same unmodified file - not the same photo re-shot or re-exported per platform. "The same photo" and "the identical file" are different claims, and the gap between them is where most cross-tool comparisons quietly go wrong.

One file, not one photo

Take a single standardised submission under a fixed set of conditions - device, distance, angle, light, crop, background, state - and send that exact file to every tool in the comparison. Do not re-shoot for each one, and do not re-export a version per tool "to make sure it looks right" on that platform. Each re-export is a chance for a resize, a recompression or a colour-space conversion to change what the tool actually receives, and any of those differences becomes indistinguishable from a genuine disagreement once you are looking at four different numbers. Image-model researchers treat these as real perturbations: Hendrycks and Dietterich (2019) count JPEG compression and pixelation among the 15 common corruptions in their robustness benchmark.

What silently makes inputs differ

A few habits break identical without feeling like they should. Uploading through a messaging app first, then saving and re-uploading, runs the file through a second compression pass before the second tool ever sees it. Letting one site's upload widget auto-crop or auto-orient the image changes the frame the tool reads, even though the file on your device looks unchanged. Orientation is not a cosmetic detail to a model; Engstrom and colleagues (2019) found state-of-the-art classifiers vulnerable to "translations and rotations." Submitting from a phone to one tool and from a downloaded copy on a laptop to another can silently swap colour profiles. None of these show up by looking at the photo; they show up in the file's data, which is exactly the part a visual check will not catch.

What "one input format" does not mean

Different tools have different upload constraints - a file size limit, an accepted format, a minimum resolution - and meeting each tool's constraint is not the same failure as reprocessing the image for it. Check each tool's stated limits before you shoot, so the single file you produce already satisfies the strictest one, rather than needing to be altered afterward to fit a narrower requirement on a second platform. A file that already meets every constraint can go out unmodified; a file that has to be squeezed down for one tool after the fact is no longer the file the others saw. A physical measurement has its own documented method for a reason - the same insistence on one procedure, applied once, rather than a different approximation for every occasion.

Order and pacing across tools

Submit to all the tools in the same sitting where the schedule allows it, rather than spreading the comparison across days. A subject's state can shift between sessions in ways that have nothing to do with any tool, and a comparison spread across a week has that shift mixed into it whether or not you notice. Where a same-day submission genuinely is not possible, note the gap in your log the same way you would note any other condition - a comparison run across a delay is not invalid, but it is a different, noisier comparison than one run in a single sitting, and it should be read that way.

Keeping one master file

Treat the single standardised submission as a master copy, kept unedited, and generate anything a specific tool needs - a resized duplicate that meets a stated limit, say - from that master rather than from a chain of previous exports. Each additional generation away from the original file is another opportunity for a lossy step to compound, and a duplicate made directly from the master carries only one such step instead of an accumulating series of them. This is a small habit that costs nothing beyond keeping the original file somewhere findable, and it is the difference between "four tools saw the same input" and "four tools saw four increasingly divergent copies of the same input."

Why this matters more as the comparison gets finer

A loose comparison - is this tool obviously more generous than that one - survives a bit of sloppiness in the inputs. A comparison built to catch a smaller gap needs a correspondingly stricter process, and file identity is one of the cheapest places to tighten first, because it costs nothing beyond discipline about how the file moves between devices and uploads.

What this is not

This is about preparing the input, not running the comparison itself. The actual procedure for running a two-tool comparison - same day, several submissions, comparing rank rather than raw number - assumes the input problem covered here is already solved. Compression is one specific way a file degrades between capture and upload, worth its own read if you want the mechanism rather than the checklist.

The tools themselves differ on plenty that has nothing to do with your file. Rate Cock, for instance, reports six axes rather than a single total, which changes what a comparison against it can even show you. A hosted vision model's read of an image is a separate subject from the file you feed it, and so is whatever a human reviewer does with the same material - but none of those differences are worth investigating until the file going in is genuinely one file, sent unmodified, to everyone on the list.

Read next

Full archive