We take our sensory organs for granted. Our pattern matching brain is wired right on top and runs on instincts life has been developing since we were nothing more than floating membranes in soup. Machines on the other hand rely on encodings developed right on top of some dynamic countable number. Something that tells it to render a pixel at color (255,255,255) is just a conceptual array of on off or otherwise meant to represent a single number or a single state at some point in time.
We look down at mosaic and see stones. Individual stones that are subdivided by some fine substrate also made of stone: tessera and the grout that binds it. We can count the stones, identify its shape, color and the metaphor constructed by it just by taking a look. All machines see without structure is that single state of a file that contains its image. The meaning and segmentation are lost until someone comes around to create the right encoding.
The problem seems simple on its face, a human can do it easy so why would a machine built to count struggle? Stones can be the same relative color as the grout, jagged edges catch the light and look like separations and the small definitions left in a low resolution Alexander The Great will leave stones human readable with just the suggestion of boundaries for a machine.
Our brain finds a signal that takes in input from our eyes, that shows us a flat tiled surface and allows us to count each faceted colored stone individually. Our brain is a compression of what we as a species have collectively learned and what we have individually learned. Babies know what objects are intrinsically. We as an individual need to learn what a stone looks like from experience. Source
A base model being inferenced is just a big genius baby that’s never seen a truly individual stone in its existence. It’s more than happy to tell you its estimate for SiO2 percentage in the grout but will, at least currently, be unable to count the individual stones with the same precision as a grade schooler.