The problem
The same box of cereal appears in one dataset as "Cheerios Honey Nut 15.4oz" and in another as "General Mills Honey Nut Cheerios Cereal, 15.4 ounce box." Neither string matching nor embedding similarity is sufficient on its own: exact matching misses obvious pairs, and semantic similarity happily matches products that differ only in the size or flavor that actually distinguishes them. Comparing every product against every other product is also quadratic, which doesn't survive contact with a real catalog.
The approach
The pipeline runs in five stages, each one narrowing the problem so the expensive step at the end sees as few pairs as possible.
- 1 · Normalization. Product names, units, and package sizes are canonicalized so downstream comparisons operate on consistent text rather than on formatting noise.
- 2 · Category blocking. Candidate pairs are only considered within compatible categories, which collapses the all-pairs comparison down to a tractable search space.
- 3 · Hybrid candidate generation. FAISS vector similarity over BAAI/bge-small-en-v1.5 embeddings runs alongside TF-IDF lexical scoring, and the two ranked lists are merged with Reciprocal Rank Fusion. Semantic search catches paraphrases; lexical search catches brand names, SKUs, and sizes that embeddings blur together. Fusing them yields high recall without giving up precision on identifiers.
- 4 · Deterministic scoring. A weighted rule-based score triages candidates into auto-accept, auto-reject, and a gray band, so only genuinely ambiguous pairs need a closer look.
- 5 · LLM adjudication. An LLM decides the gray-band cases only. Because the earlier stages do the filtering, the model is invoked on a small fraction of the candidate set — which is why the entire run cost roughly $0.41.
Why it's built this way
The design principle is to spend compute in proportion to difficulty. Cheap deterministic stages handle the easy majority of pairs, and the expensive, non-deterministic stage handles only what the cheap stages can't resolve. That ordering is what makes the pipeline both accurate and nearly free to run — and it keeps the vast majority of decisions reproducible and auditable rather than hidden inside a model.
Highlights
- Five-stage pipeline: normalize → block → hybrid retrieval → score → adjudicate.
- FAISS + TF-IDF fused with Reciprocal Rank Fusion over bge-small-en-v1.5 embeddings.
- Auto-accept / auto-reject / gray-band triage to minimize manual review.
- 12,721 matches produced for approximately $0.41 in total compute.
Stack
- Python
- FAISS
- TF-IDF
- Reciprocal Rank Fusion
- bge-small-en-v1.5
- LLM adjudication