Image-Based Fashion Search

How it works

At the core is an embedding model that converts an image into a vector — a list of numbers capturing its visual features. Catalogue images are embedded the same way, and search becomes a matter of finding the catalogue vectors nearest to the query vector. Nothing is matched by text; the comparison is entirely in visual space.

Modern systems refine this with detection and segmentation, isolating the garment or shoe from its background and other objects before embedding. That lets a shopper photograph a full outfit and search for just the boots, which is a meaningful improvement over whole-image matching. The pipeline is well understood; the difficulty is in the edges.

Where it is strong

Visual search is at its best exactly where text search is weakest. It bypasses fashion's naming chaos — no need to know whether to call something a "trainer", "sneaker" or "kick" — and it captures details that resist description, such as a particular sole shape or panelling. For inspiration-driven shopping, where a shopper knows the look but not the label, it is close to ideal.

It also lowers friction. Point, shoot, browse results. For casual shoppers who do not speak the vocabulary of the category, that removes the biggest barrier to finding what they saw.

Failure mode: colour

Colour is deceptively hard. The same shoe photographs differently under warm indoor light, cool daylight or a phone's automatic white balance, and a model can read a genuine navy as black or a cream as white. Since colourway is often the difference between two distinct products, a colour error is not cosmetic — it returns the wrong shoe.

This connects directly to how catalogues handle colourways. Visual systems that lean only on pixels struggle here; those that combine visual similarity with structured colour metadata do better, because the metadata anchors what the pixels merely suggest.

Failure mode: texture and material

Suede and smooth leather can look identical in a small or low-light photo, as can knit and mesh. Material is often invisible to the camera at the resolution shoppers actually supply, yet it changes the product, the price and the use case. A visual match on shape that ignores material can be confidently wrong.

Texture failures are harder to fix than colour because there is frequently no signal to recover in the image at all. The realistic mitigation is to treat visual similarity as a candidate generator and let structured attributes filter it, rather than trusting the pixels alone.

Failure mode: near-duplicates

Fashion is full of products that look almost the same: a general-release shoe and its limited colourway, an original and a close copy, this season's model and last season's. Visual search excels at finding look-alikes, which is precisely the problem — it will surface all of them and struggle to distinguish the one the shopper actually wants.

Resolving near-duplicates requires more than vision. It needs the disambiguation that canonical records and style codes provide, so that "visually similar" can be narrowed to "actually this product". Vision finds the neighbourhood; structured data finds the address.

What the catalogue has to provide

Every failure mode points back to catalogue quality. Useful visual search needs several things the average product feed does not supply by default:

• Consistent, well-lit product imagery from predictable angles, so the model compares products rather than lighting conditions.
• Images reliably tied to correctly attributed records, so a visual match resolves to the right product rather than a look-alike.
• Structured metadata — colour, material, size — to filter what the pixels alone cannot decide.
• Enough detail shots to distinguish the near-duplicates that plain hero images conflate.

Marketing photography optimised for a single hero shot is often the wrong input for a search index, which is a recurring source of quiet failure. Think with Google has documented rising shopper use of image-led and camera-based search, particularly on mobile. The opportunity is real, but capturing it is as much a data-operations task as a machine-learning one. For the search context this sits within, see our overview of fashion search engines.