Cleaning a 20,000-SKU Magento Catalogue With AI — Without Hiring

Ask a Magento merchant what their biggest technical problem is and they will say speed, or checkout, or the upgrade they keep postponing. Ask their merchandising team and you get a different answer: the catalogue is a mess. Half the products have a description copied from the supplier PDF, the material attribute is populated on 40% of SKUs, and nobody can remember which of the three colour taxonomies is the real one.

This is not a cosmetic problem. Thin attributes break layered navigation, starve on-site search of the terms shoppers actually type, and produce category pages with nothing for Google to rank. And as agent-mediated shopping grows, structured product data stops being an SEO nicety and becomes the difference between being quoted by an assistant and being invisible to it.

The traditional fix is a data-entry team and eighteen months. The realistic fix in 2026 is a pipeline. Here is the one we run.

Decide What You Are Actually Fixing

Do not start with descriptions. Start with attributes, because attributes are what everything else depends on: filters, search, feeds, and machine readability. Descriptions are the visible problem; attributes are the expensive one.

Run a completeness audit first. For every attribute in your Magento attribute set, get the fill rate and the distinct-value count. That single report tells you more than a week of meetings — it will show you the attribute that is 94% populated and useless because it has 3,000 distinct free-text values, and the one that is 30% populated and would transform your category pages if it were not.

• Fill rate below 60% on a filterable attribute — this is why your layered navigation is thin
• High distinct-value counts on what should be a select attribute — free-text where a dropdown belongs, so nothing groups
• Multiple attributes meaning the same thing — colour, color, shade, inherited from three different import scripts
• Missing GTIN, MPN or brand — the fields that decide whether your products can be matched at all outside your own site

The Pipeline

Four stages, and the third is the one that matters.

1. Export With Context

Export the catalogue with everything the model will need to ground its answers: existing description, supplier text, category path, attribute-set name, brand, and any specification blob you have lying in a custom attribute. Include the SKU and the entity ID so results can be matched back without ambiguity.

Batch by attribute set rather than sending the whole catalogue as one job. A prompt tuned for furniture dimensions is a bad prompt for cosmetics ingredients, and mixing them produces the mediocre middle of both.

2. Enrich With a Constrained Prompt

The prompt should do one job: read the supplied source text and return a fixed schema. Not write marketing copy. Not improve anything. Extract and normalise.

• Give it the allowed values. If material is a select attribute with 24 options, put all 24 in the prompt and require the answer to be one of them or null.
• Require a null, explicitly, for anything not present in the source. This is the single most important instruction in the whole pipeline.
• Ask for a confidence signal per field, so the review stage can be targeted rather than uniform.
• Return structured output, validated against a schema, so a malformed response fails loudly instead of importing quietly.

3. Review by Sampling, Not by Reading Everything

Nobody is going to review 20,000 rows, and pretending otherwise is how these projects fail. Sample instead, and let the confidence signal do the routing:

• Low-confidence rows: review all of them. These are the ones where the source data was genuinely ambiguous.
• High-confidence rows: sample 5%, stratified across categories and suppliers so a single bad supplier feed cannot hide in the average.
• Set an acceptance threshold before you start — we use 98% correct on the sample — and if a batch fails it, fix the prompt and re-run the batch rather than correcting rows by hand.
• Never sample only the easy categories. The failure mode you are looking for lives in the messiest supplier data you have.

A merchandiser reviewing a well-designed sample can clear a batch in an afternoon. The same person reviewing every row would still be going at Christmas, which is why the uncontrolled version of this project never finishes.

4. Import in Reversible Batches

Import per attribute set, per batch, with the pre-import values stored somewhere you can restore from. Magento's import is perfectly capable of overwriting 20,000 rows of good data with 20,000 rows of bad data very quickly, and the moment you discover it is usually a week later when someone notices the filters look strange.

Reindex, then check the layered navigation and search results on the affected categories before you release the next batch. The visible surface is your regression test.

Descriptions: A Different, Riskier Job

Once attributes are clean, generating description copy from them becomes far safer — the model is assembling sentences from verified facts rather than inferring facts from prose. Do it in that order, never the reverse.

Even then, treat it carefully. Google's guidance is about usefulness rather than provenance: automatically generated text that genuinely helps a shopper is fine, and text generated to fill a page is not, regardless of who or what wrote it. In practice that means:

• Ground every claim in an attribute. No invented benefits, no invented compatibility, no invented care instructions.
• Give the prompt a real brand-voice brief with three approved examples — the difference between generic and on-brand output is almost entirely in the examples.
• Roll out to a holdout: enrich half a category, leave the other half, and compare organic impressions and conversion after a full crawl cycle. This is the only way you will know whether it worked.
• Keep a human sign-off on regulated categories. Anything with a health, safety, ingredient or compliance claim does not go through an automated pipeline. Ever.

Where This Goes Wrong

Four failure modes, all of which we have seen.

1. Hallucinated specifications. Dimensions, weights and compatibility invented from context. This is a returns problem and, in some categories, a legal one — which is why the null instruction matters more than the schema.
2. Normalising away real distinctions. Forcing 3,000 free-text colours into 24 options is the point, right up until "Racing Green" and "British Racing Green" collapse into "Green" and a customer receives the wrong car part.
3. Silent taxonomy drift. Each batch nudges values slightly differently, and six months later the attribute is inconsistent again in a new way. Fix this with a locked allowed-value list, not with prompt discipline.
4. Optimising for volume, not visibility. A 100% fill rate on an attribute nobody filters by has achieved nothing. Prioritise by what your search and navigation actually use.

What It Costs

Model inference on a 20,000-SKU catalogue is the cheapest part of this project by a wide margin — the cost sits in the engineering to build the pipeline and the merchandiser time to review the samples. Budget for a few weeks of integration work and a defined block of specialist review time, and treat the token spend as a rounding error against either.

That ratio is what makes the approach worth it. The thing that used to be expensive — human hours per SKU — becomes a sampled quality function instead of a linear one, and the thing that is now cheap scales with the catalogue rather than with the headcount.

The Second Payoff: Machine Readability

A clean, normalised catalogue is the prerequisite for everything downstream that people are currently excited about. Semantic search needs consistent attributes to be worth anything. Recommendation quality is bounded by product-data quality. Merchant feeds reject on missing GTINs. And AI shopping assistants can only quote products whose attributes they can read and trust.

We wrote separately about getting your products recommended by ChatGPT and Perplexity, and the honest summary is that most of that work is this work. The unglamorous data project is the one that unlocks the interesting ones.

Our AI and machine learning team builds these pipelines against live Magento catalogues, and our wider AI integration practice exists precisely to make sure the plumbing is right before the clever part is switched on. You can see the same principle in our Portmeirion inventory work, where the model was only ever as good as the data feeding it.

Start With the Completeness Audit

Before any of this, get the fill-rate and distinct-value report for your attribute sets — it usually reframes the problem. We will run it against your catalogue and tell you which attributes are worth the pipeline and which are noise. Talk to our team; we reply within a business day.