Menu
Entity Resolution / Information Retrieval
Case 062026Machine Learning / Entity Resolution & Information Retrieval

BuER

Multilingual entity resolution pipeline linking 1.7M noisy business records

BuER is an entity resolution pipeline engineered for the Amazon ML Challenge 2026 to link noisy, multilingual business records across three disparate sources. It tackles combinatorial entity disambiguation through country-partitioned candidate retrieval, dictionary-backed Indic-to-Latin token transliteration, RapidFuzz string similarity, structural street-number conflict pruning, and an exclusive global assignment optimizer that assigns each candidate to its single highest-scoring entity.

Scale in entity resolution is not won by throwing monolithic transformers at pairwise comparisons; it is won by intelligent candidate blocking and deterministic conflict elimination before scoring begins.

PythonEntity ResolutionNLPInverted IndexRapidFuzzAmazon ML

Macro F0.5

0.987

author-reported competition score on unseen test split

Competition Rank

250

among 90,000+ participating engineers in Amazon ML Challenge

Submitted Pool

1.73M

source entities processed and disambiguated

BuER pipeline architecture and candidate resolution network
Proof strip

The page keeps the strongest metrics and the strongest caveats in the same frame. Numbers are included because they clarify the system, not because they decorate it.

Match Links

5,803,442

exclusive assignments generated for final submission

Candidate Pairs

12.9M

7.46 candidates per source entity under posting frequency limits

Indic Lexicon

3,103 tokens

curated Indic transliteration dictionary with Unidecode fallback

Case-study frame

The project is explained across seven recurring dimensions so the reader can understand not only what was built, but why the implementation matters, where the proof is strongest, and where the limits still are.

Case-study dimension

Why This Exists

Cross-source business entity resolution at national and global scale faces an exponential pairwise comparison problem: naive matching of 1.73 million entities against millions of candidate pool records requires trillions of comparisons.

BuER was engineered to solve this combinatorial explosion under competitive time and memory constraints, handling severely corrupted multilingual tokens, varying script encodings, and missing address components.

Case-study dimension

Why It Matters Now

Modern enterprise supply chains, tax fraud detection, and credit underwriting all depend on reliable company deduplication across noisy data streams without manual human curation.

As business data increasingly spans multilingual markets across Asia and the West, entity resolution systems must handle script divergence like Indic-to-Latin transliteration directly rather than assuming standardized English records.

Case-study dimension

What It Improves

The pipeline introduces country-partitioned blocking combined with token posting-frequency ceilings, pruning over 99.8% of the comparison space while retaining an average of 7.46 high-probability candidates per entity.

A custom 3,103-token Indic dictionary maps regional phonetic variations directly to normalized Latin roots, eliminating false negatives caused by script transliteration inconsistencies.

Case-study dimension

What Makes It Different

Rather than relying on greedy local thresholds, BuER enforces global exclusivity: each pool candidate record is assigned strictly to its single highest-scoring source entity.

Structural street-number compatibility checks run before fuzzy address distance calculations, guaranteeing that two companies sharing a street name but occupying different building numbers are never erroneously conflated.

Case-study dimension

Why It Is Hard To Replicate

The engineering difficulty lies in tuning blocking frequency thresholds to balance recall against memory footprint on 12.9 million candidate pairs.

Balancing weight between token-sort ratios, partial string matching, and transliteration confidence requires deep empirical optimization rather than generic heuristics.

Case-study dimension

How It Can Grow

The architecture is modular: the postprocessing optimizer already supports prefix selection, and could be extended with active-learning feedback loops or GPU-accelerated embedding retrieval for ambiguous edge cases.

Adding geographic coordinate geospatial bounding boxes would provide an additional physical validation layer for address matching.

Case-study dimension

Why It Lasts

Data noise in business directories is permanent: formatting changes, transcription typos, and translation divergences will always exist in real-world ingestion pipelines.

A deterministic, fast blocking and scoring architecture provides a durable, cost-effective baseline that runs in minutes on commodity hardware without cloud API dependencies.

Technical proof

Candidate Blocking

Country partitioning and posting limits reduce trillions of comparisons to 12.9M pairs

Pairwise entity comparison across 1.73M records is impossible without aggressive blocking. The indexer partitions records by ISO country code and indexes tokens in an inverted index with strict frequency thresholds.

Ubiquitous corporate suffixes like 'Pvt Ltd' or 'Enterprises' are pruned from the posting list, ensuring the candidate pool is populated only by discriminative business identifiers.

country-partitioned inverted indexposting-frequency limits prune generic corporate tokens7.46 average candidates per source entity
BuER candidate blocking and inverted index architecture

Multilingual Resolution

Deterministic transliteration bridges Indic script variations before fuzzy scoring

Indic company names frequently appear in varying scripts (Devanagari, Bengali, Telugu, Tamil) alongside English phonetic transcriptions. BuER uses a curated 3,103-token lexicon with Unidecode fallback to normalize phonetics.

RapidFuzz token-sort and partial ratios then compare normalized names and addresses, while structural regex guards flag building-number contradictions before score aggregation.

3,103-token Indic-to-Latin transliteration lexiconRapidFuzz token sort and partial address matchingexclusive assignment prevents duplicate record allocation
BuER transliteration engine and RapidFuzz feature weighting
Supplementary visuals
BuER system architecture diagram
End-to-end pipeline: from raw TSV partitions to exclusive high-scoring entity links.
BuER candidate blocking mechanism
Country-partitioned blocking index reducing comparison surface by over 99.8%.