Company Dataset Diff Tool

Objective

Part of my job is scoping TAM by pulling company lists from data providers, cross-mathcing them and pushing clean records into the CRM. A normal request looks like this: every EMEA Enterprise account in the Software industry. I pull it from one vendor, then from another, and now I have two files that are supposed to describe the same market.

The problem with programmatic matching is that it only matches text written identically on both sides. One file has the domain as https://www.acme.ie/ and the other has acme.ie. One keeps Acme Ltd and the other writes Acme Limited. The lookup finds nothing, so the company reads as unique to one vendor when it is sitting in both files. Checking a sample by hand does not solve it either. A sample small enough to review in an afternoon is too small to tell me anything reliable about the rest of the file.

So the tool has two jobs. The first is to tell me how much of the two lists is actually the same companies. The second is to cut the time I spend reviewing by hand, by giving me only the records where the two vendors genuinely disagree instead of the whole file. If I pull EMEA Enterprise Software accounts from two providers, I want to review and merge them with confidence as fast as possible.

None of the individual steps are new to me. I already do versions of them with small scripts and manual checks. What this project does is put them in one place, rewrite them in JavaScript, and run the whole thing inside a browser tab with a proper interface, keeping the data on my machine and ensuring compliance with strict security and privacy standards.

About the build

1. Add the files.
   • Two or more CSV exports, one company per row. First row headers.
   • Everything runs in the browser. No upload, no server, no account.
2. Confirm what each column is.
   • The tool reads every column and proposes which one holds the company domain, company name, country, employee count, revenue, industry and LinkedIn URL.
   • It looks at header names it recognises (domain, website, url, account name, headcount, company size, turnover, sector and about sixty more) and at the values inside the column.
   • Values outrank headers. A column called employee_count full of email addresses is an email column whatever the header says.
   • Nothing moves forward until user confirms the mapping.
3. Match the records. Three tiers, each with its own confidence.
   • Tier 1, identical domain after normalisation, so https://www.acme.ie/ and acme.ie both reduce to acme.ie. Accepted automatically.
   • Tier 2, identical company name plus country after normalisation, so Acme Ltd and ACME LIMITED both reduce to acme. Accepted automatically.
   • Tier 3, similar name in the same country, scored by Levenshtein distance and only offered at 0.88 similarity or above. Shown to the user as a suggestion and never counted unless they tick it, because one wrong match quietly contaminates every number after it and nobody can see it happen.
   • Working out where a domain actually ends. Before two domains can be compared, something like careers.acme.co.uk/jobs has to be reduced to the part the company really registered, which is acme.co.uk. Counting dots does not work, because acme.ie is a two part registration and acme.co.uk is a three part one, and there are thousands of these shapes worldwide. Cut one label too many and every company on a .co.uk address reduces to co.uk, so they all match each other and tier 1 merges unrelated companies automatically. Cut one too few and careers.acme.ie never matches acme.ie, so one company reads as two. The Public Suffix List is the published answer to where that line sits. It is maintained by Mozilla and used by browsers to decide which sites are allowed to set cookies for which domain, and the tool uses all of it, wildcard and exception rules included.
   • That list is built into the page, not fetched while it runs. A script turns the published list into a JavaScript file ahead of time, so the tool makes no network call, keeps working with no connection, and gives the same answer on every run. The report names the version of the list it used, because a different list would produce different matches.
4. Check the join before anything is compared. This step exists because a bad join is silent. What failed to match gets counted as unique to that vendor and the overlap number is then wrong. It surfaces:
   • Duplicate domains inside one file, which create a one to many join.
   • Rows with no usable domain and no usable name, which cannot be matched at all.
   • Values that could not be read.
5. Compare, field by field. Each field is compared on its own terms, because a strict text comparison manufactures conflicts that are not there.
   • Employee counts are compared by band, so 480 against 520 is not a conflict.
   • Revenue inside 20% counts as agreement, inside 2x as a minor variant, beyond that as a disagreement. Different currencies convert at one fixed rate, and the report always states the rate and the date it was set.
   • Industries are only compared once both sides map to a shared bucket. Until then they are reported as unmapped.

The comparison logic sits in its own file with no reference to the page, so it can be run and tested outside the browser.

What it produces

• How much the two files overlap. How many companies are in both, and how many only one of the two vendors has.
• How much each file actually fills in, field by field. This says nothing about whether the values are right, only how often a usable value is there at all.
• Agreement and disagreement rates per field, with the rows that could not be compared counted separately and labelled with the reason.
• The short list of conflicting records, on screen and as a CSV download.
• The same disagreement rates sliced by country and by company size.

Lessons learned while using it

• Revenue came back with an 82% disagreement rate, and all of it was one file quoting dollars and the other euro. Revenue now either converts at a fixed rate with the date stated, or reports as not comparable.
• Industry taxonomies were being counted as disagreements. "Computer Software" against "Information Technology and Services" is two providers using different vocabularies, not either one getting the company wrong. Those now report as unmapped until a crosswalk puts both sides into a shared bucket.

Things to add in the next version

1. More columns, starting with parent company. Account hierarchy is where provider data disagrees often or have incomplete data.
2. A check against the outside world for the mismatches. A scraping step, through something like Apify, that looks up the disputed records on LinkedIn and Google and reports which side matches what is published. That begins to answer accuracy, but losses on data privacy. It would only run on the handful of records already flagged as conflicts.
3. In a local version of this I would add a local AI agent on the "check the join" step. It would pre-screen the fuzzy suggestions so I only look at the genuinely ambiguous ones, which is the slowest manual part of the process today. Locally run, so the data still never leaves the machine.
4. Again in a local version of this project, agent to produce three or four bullet summary at the top of the report. The headline findings in plain language, so the report is readable without going through every table underneath it.