Canonical IDs and Entity Resolution for Structured Web Data
Structured web data becomes difficult to maintain when the same entity appears under different URLs, titles, sellers, or platforms.
A reliable pipeline needs more than field extraction. It also needs a clear identity model.
URLs are source references, not always identities
A URL can change while the underlying entity remains the same.
A product may move to another category, receive a new URL, or appear on a regional version of the same marketplace.
For this reason, the source URL should be stored separately from the canonical entity ID.
A basic record might look like this:
{
"canonical_product_id": "product_001",
"source_platform": "marketplace_a",
"source_url": "https://example.com/item/123",
"title": "Example Product",
"brand": "Example Brand",
"collected_at": "2026-11-17T09:00:00Z"
}
The canonical ID supports joins and historical tracking. The source URL supports verification and traceability.
Separate products, listings, offers, and variants
One product can have:
- Multiple marketplace listings
- Multiple sellers
- Multiple regional offers
- Multiple sizes or colors
- Different product URLs
These should not automatically be merged into one flat record.
A clearer model separates the entities:
Product
└── Variant
└── Marketplace Listing
└── Seller Offer
This prevents a seller change from being interpreted as a product change.
It also prevents a price for one variant from being applied to every version of the product.
Use multiple signals for entity resolution
Matching records across sources usually requires several signals:
- Brand
- SKU
- Manufacturer part number
- GTIN or UPC
- Product attributes
- Variant information
- Image similarity
- Category
- Seller identity
- Source platform
These signals should have different weights.
A matching manufacturer part number may provide stronger evidence than a similar product title. A shared image may be useful, but it does not always prove that two listings are identical.
The pipeline should also retain a confidence or resolution status:
{
"canonical_product_id": "product_001",
"source_record_id": "marketplace_a_123",
"match_status": "confirmed",
"match_confidence": 0.96,
"matched_by": [
"brand",
"manufacturer_part_number",
"variant_attributes"
]
}
The exact score is less important than making the matching decision visible.
Resolve identity before detecting changes
Change detection should happen only after identity resolution.
Otherwise, a URL change may appear as a new product, while a duplicate listing may appear as a price change.
A useful change record can include:
{
"canonical_product_id": "product_001",
"field": "price",
"previous_value": "699.99",
"new_value": "679.99",
"source_url": "https://example.com/item/123",
"collected_at": "2026-11-17T09:00:00Z",
"validation_status": "valid"
}
This structure makes price monitoring, catalog updates, and market research easier to audit.
Preserve source-specific context
Canonicalization should not remove the original source information.
Keep the source platform, source record ID, URL, seller, region, and collection timestamp. These fields are necessary when a team needs to verify a match or investigate an unexpected change.
A canonical data model should make cross-source analysis possible without hiding the differences between sources.
Final perspective
Entity resolution is not only a matching problem.
It is also a data modeling problem.
Stable canonical IDs, separate listing and offer records, weighted matching signals, and explicit change history make structured web data more reliable over time.
Comments
Join the discussionNo comments yet. Be the first to comment.
