<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Data Notes]]></title><description><![CDATA[Data Notes]]></description><link>https://thordata-notes.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6a96777a3ea007e108ad3dc1/5e901830-3793-4010-bc16-88325b149929.png</url><title>Data Notes</title><link>https://thordata-notes.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Wed, 18 Nov 2026 13:16:28 GMT</lastBuildDate><atom:link href="https://thordata-notes.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Canonical IDs and Entity Resolution for Structured Web Data]]></title><description><![CDATA[Structured web data becomes difficult to maintain when the same entity appears under different URLs, titles, sellers, or platforms. A reliable pipeline…]]></description><link>https://thordata-notes.hashnode.dev/canonical-ids-and-entity-resolution-for-structured-web-data</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/canonical-ids-and-entity-resolution-for-structured-web-data</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Tue, 17 Nov 2026 06:35:30 GMT</pubDate><content:encoded><![CDATA[<h1>Canonical IDs and Entity Resolution for Structured Web Data</h1>
<p>Structured web data becomes difficult to maintain when the same entity appears under different URLs, titles, sellers, or platforms.</p>
<p>A reliable pipeline needs more than field extraction. It also needs a clear identity model.</p>
<h2>URLs are source references, not always identities</h2>
<p>A URL can change while the underlying entity remains the same.</p>
<p>A product may move to another category, receive a new URL, or appear on a regional version of the same marketplace.</p>
<p>For this reason, the source URL should be stored separately from the canonical entity ID.</p>
<p>A basic record might look like this:</p>
<pre><code class="language-json">{
  "canonical_product_id": "product_001",
  "source_platform": "marketplace_a",
  "source_url": "https://example.com/item/123",
  "title": "Example Product",
  "brand": "Example Brand",
  "collected_at": "2026-11-17T09:00:00Z"
}
</code></pre>
<p>The canonical ID supports joins and historical tracking. The source URL supports verification and traceability.</p>
<h2>Separate products, listings, offers, and variants</h2>
<p>One product can have:</p>
<ul>
<li>Multiple marketplace listings</li>
<li>Multiple sellers</li>
<li>Multiple regional offers</li>
<li>Multiple sizes or colors</li>
<li>Different product URLs</li>
</ul>
<p>These should not automatically be merged into one flat record.</p>
<p>A clearer model separates the entities:</p>
<pre><code class="language-text">Product
  └── Variant
        └── Marketplace Listing
              └── Seller Offer
</code></pre>
<p>This prevents a seller change from being interpreted as a product change.</p>
<p>It also prevents a price for one variant from being applied to every version of the product.</p>
<h2>Use multiple signals for entity resolution</h2>
<p>Matching records across sources usually requires several signals:</p>
<ul>
<li>Brand</li>
<li>SKU</li>
<li>Manufacturer part number</li>
<li>GTIN or UPC</li>
<li>Product attributes</li>
<li>Variant information</li>
<li>Image similarity</li>
<li>Category</li>
<li>Seller identity</li>
<li>Source platform</li>
</ul>
<p>These signals should have different weights.</p>
<p>A matching manufacturer part number may provide stronger evidence than a similar product title. A shared image may be useful, but it does not always prove that two listings are identical.</p>
<p>The pipeline should also retain a confidence or resolution status:</p>
<pre><code class="language-json">{
  "canonical_product_id": "product_001",
  "source_record_id": "marketplace_a_123",
  "match_status": "confirmed",
  "match_confidence": 0.96,
  "matched_by": [
    "brand",
    "manufacturer_part_number",
    "variant_attributes"
  ]
}
</code></pre>
<p>The exact score is less important than making the matching decision visible.</p>
<h2>Resolve identity before detecting changes</h2>
<p>Change detection should happen only after identity resolution.</p>
<p>Otherwise, a URL change may appear as a new product, while a duplicate listing may appear as a price change.</p>
<p>A useful change record can include:</p>
<pre><code class="language-json">{
  "canonical_product_id": "product_001",
  "field": "price",
  "previous_value": "699.99",
  "new_value": "679.99",
  "source_url": "https://example.com/item/123",
  "collected_at": "2026-11-17T09:00:00Z",
  "validation_status": "valid"
}
</code></pre>
<p>This structure makes price monitoring, catalog updates, and market research easier to audit.</p>
<h2>Preserve source-specific context</h2>
<p>Canonicalization should not remove the original source information.</p>
<p>Keep the source platform, source record ID, URL, seller, region, and collection timestamp. These fields are necessary when a team needs to verify a match or investigate an unexpected change.</p>
<p>A canonical data model should make cross-source analysis possible without hiding the differences between sources.</p>
<h2>Final perspective</h2>
<p>Entity resolution is not only a matching problem.</p>
<p>It is also a data modeling problem.</p>
<p>Stable canonical IDs, separate listing and offer records, weighted matching signals, and explicit change history make structured web data more reliable over time.</p>
]]></content:encoded></item><item><title><![CDATA[Designing a Variant-Aware E-commerce Data Pipeline]]></title><description><![CDATA[E-commerce product pages rarely represent only one sellable item. A single parent product may contain multiple sizes, colors, capacities, sellers, or…]]></description><link>https://thordata-notes.hashnode.dev/designing-a-variant-aware-e-commerce-data-pipeline</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/designing-a-variant-aware-e-commerce-data-pipeline</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Mon, 16 Nov 2026 08:32:09 GMT</pubDate><content:encoded><![CDATA[<h1>Designing a Variant-Aware E-commerce Data Pipeline</h1>
<p>E-commerce product pages rarely represent only one sellable item.</p>
<p>A single parent product may contain multiple sizes, colors, capacities, sellers, or regional offers. If a pipeline stores only the page title and one price, it can hide important changes.</p>
<p>A better approach separates the parent product from its variants.</p>
<h2>Model the parent product and variants separately</h2>
<p>The parent record can contain:</p>
<ul>
<li>Product title</li>
<li>Brand</li>
<li>Category</li>
<li>Platform</li>
<li>Product URL</li>
<li>Source timestamp</li>
</ul>
<p>The variant record can contain:</p>
<ul>
<li>Variant ID</li>
<li>Variant name</li>
<li>SKU or platform ID</li>
<li>Price</li>
<li>Currency</li>
<li>Availability</li>
<li>Seller</li>
<li>Variant URL</li>
</ul>
<p>This prevents unrelated variants from being merged into one price history.</p>
<p>A price change for one size or color should not appear as a change to every version of the product.</p>
<h2>Store timestamped snapshots</h2>
<p>Price monitoring should append validated observations instead of overwriting the previous value.</p>
<p>A snapshot may look like this:</p>
<pre><code class="language-json">{
  "product_id": "product_001",
  "variant_id": "variant_blue_128gb",
  "price": 699.99,
  "currency": "USD",
  "availability": "in_stock",
  "seller": "example_store",
  "source_url": "https://example.com/product",
  "collected_at": "2026-11-16T09:30:00Z"
}
</code></pre>
<p>With timestamped snapshots, a pipeline can identify:</p>
<ul>
<li>Price increases</li>
<li>Price decreases</li>
<li>Availability changes</li>
<li>Seller changes</li>
<li>Currency changes</li>
<li>Temporary extraction failures</li>
</ul>
<p>Overwriting the previous value removes the evidence needed to explain the change.</p>
<h2>Add validation before comparison</h2>
<p>Not every difference is a real market change. A page layout change or failed extraction can create an incorrect price movement.</p>
<p>Useful validation rules include:</p>
<ul>
<li>Product or variant ID is present</li>
<li>Price can be parsed as a number</li>
<li>Currency is recognized</li>
<li>URL is valid</li>
<li>Availability uses a controlled value</li>
<li>Collection timestamp is stored</li>
<li>Duplicate records are removed</li>
<li>Extraction status is retained</li>
</ul>
<p>Invalid results should be flagged instead of silently replacing a valid snapshot.</p>
<h2>Match the data to the business workflow</h2>
<p>Different use cases need different fields.</p>
<p>Price monitoring may prioritize price, seller, availability, and timestamp.</p>
<p>Catalog enrichment may need descriptions, images, categories, and attributes.</p>
<p>Market research may focus on brands, ratings, reviews, and regional coverage.</p>
<p>The output should use clear field definitions so downstream systems can filter, compare, and aggregate the records without repeated manual cleanup.</p>
<h2>Final perspective</h2>
<p>Reliable e-commerce data is not created by collecting the largest number of pages.</p>
<p>It comes from representing product structure correctly, preserving changes over time, and validating records before they enter analysis.</p>
<p>A variant-aware and snapshot-based pipeline produces data that is easier to compare, audit, and use in recurring business workflows.</p>
]]></content:encoded></item><item><title><![CDATA[A Practical Architecture for Model-Ready Dataset Delivery]]></title><description><![CDATA[A dataset is only useful when downstream systems can consume it reliably. That means delivery should be designed together with the schema. The right…]]></description><link>https://thordata-notes.hashnode.dev/a-practical-architecture-for-model-ready-dataset-delivery</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/a-practical-architecture-for-model-ready-dataset-delivery</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Mon, 09 Nov 2026 08:23:50 GMT</pubDate><content:encoded><![CDATA[<p>A dataset is only useful when downstream systems can consume it reliably.</p>
<p>That means delivery should be designed together with the schema. The right approach for offline training is different from the right approach for retrieval, evaluation, or recurring business monitoring.</p>
<h2>Start with the downstream workflow</h2>
<p>A dataset may be used for:</p>
<ul>
<li>Offline model training</li>
<li>Fine-tuning</li>
<li>Retrieval and enrichment</li>
<li>Evaluation</li>
<li>Analytics</li>
<li>Recurring monitoring</li>
<li>Production API requests</li>
</ul>
<p>Each workflow has different requirements.</p>
<p>Offline training usually benefits from versioned files. Retrieval systems often need on-demand API access. Recurring monitoring workflows need scheduled delivery, update boundaries, and a way to identify changed records.</p>
<p>Treating all of these as one access pattern creates unnecessary complexity.</p>
<h2>Define a stable data contract</h2>
<p>A useful delivery contract should describe:</p>
<ul>
<li>Schema version</li>
<li>Stable record ID</li>
<li>Field types</li>
<li>Timestamp semantics</li>
<li>Source reference</li>
<li>Update frequency</li>
<li>Deduplication rules</li>
<li>Delivery format</li>
<li>Snapshot or revision ID</li>
<li>Retry and failure behavior</li>
</ul>
<p>The stable record ID is especially important. A record may keep the same identity while its title, price, transcript, description, or engagement fields change.</p>
<p>This makes it possible to distinguish identity from mutable attributes.</p>
<h2>Choosing a file format</h2>
<h3>CSV</h3>
<p>CSV is useful when the schema is flat and human inspection is important. It works well for small exports and simple analysis.</p>
<p>Its main weakness is that nested structures, arrays, null values, and data types are not represented consistently.</p>
<h3>JSON and NDJSON</h3>
<p>JSON is a practical choice for API responses and records containing nested metadata.</p>
<p>NDJSON is often better for large record-based workflows because each line can be processed independently:</p>
<pre><code class="language-json">{"record_id":"a001","title":"Example record","source":"platform_a"}
{"record_id":"a002","title":"Another record","source":"platform_b"}
</code></pre>
<h3>Parquet</h3>
<p>Parquet is useful for larger analytical workloads. Its columnar structure supports efficient filtering and better type preservation.</p>
<p>It is often a good fit for data lakes, feature pipelines, and recurring batch delivery.</p>
<p>The important point is not to choose one format for every situation. The same logical dataset can have multiple representations if record IDs and schema definitions remain consistent.</p>
<h2>Design incremental updates explicitly</h2>
<p>Recurring delivery should distinguish between:</p>
<ul>
<li>New records</li>
<li>Updated records</li>
<li>Unchanged records</li>
<li>Removed records</li>
</ul>
<p>A simple update model might look like this:</p>
<pre><code class="language-json">{
  "record_id": "a001",
  "operation": "update",
  "changed_fields": ["title", "updated_at"],
  "source_timestamp": "2026-11-09T08:00:00Z",
  "dataset_revision": "2026-11-09T09:00:00Z",
  "schema_version": "2.1",
  "content_hash": "sha256:example"
}
</code></pre>
<p>A content hash helps prevent downstream systems from reprocessing identical data.</p>
<p>Deletion semantics also need to be explicit. A missing record does not always mean that the record was deleted. It may simply be outside the current filter or update window.</p>
<p>For that reason, some pipelines use tombstones:</p>
<pre><code class="language-json">{
  "record_id": "a001",
  "operation": "delete",
  "dataset_revision": "2026-11-09T09:00:00Z"
}
</code></pre>
<h2>Match delivery to the system</h2>
<p>Different delivery methods solve different problems:</p>
<ul>
<li>File delivery works well for batch training and offline analysis</li>
<li>Direct API access works well for on-demand requests</li>
<li>Cloud storage works well for data lake workflows</li>
<li>Webhooks work well for notifying downstream systems about new batches</li>
<li>Scheduled delivery works well for recurring monitoring and refresh cycles</li>
</ul>
<p>A delivery method should also define what happens when a batch fails. Consumers need to know whether they should retry the same revision, wait for a replacement, or continue with the previous valid version.</p>
<h2>Keep published revisions immutable</h2>
<p>Once a dataset revision has been delivered, changing it in place makes debugging and reproducibility harder.</p>
<p>A safer process is:</p>
<ol>
<li>Create a new revision</li>
<li>Validate the schema and record counts</li>
<li>Publish the revision</li>
<li>Notify consumers</li>
<li>Retain the previous revision for recovery</li>
</ol>
<p>This approach allows a training run or evaluation report to point to an exact data revision.</p>
<h2>Final perspective</h2>
<p>Model-ready data is not defined only by field quality.</p>
<p>It also depends on whether the data can be delivered in a predictable format, updated without ambiguity, and traced back to a specific revision.</p>
<p>Schema design, file format, access method, and update semantics should be treated as one architecture decision.</p>
]]></content:encoded></item><item><title><![CDATA[What Robotics Teams Need from Video Datasets Beyond More Clips]]></title><description><![CDATA[Video data for robotics and embodied AI has different requirements from ordinary video classification. A single frame may show an object clearly, but it…]]></description><link>https://thordata-notes.hashnode.dev/what-robotics-teams-need-from-video-datasets-beyond-more-clips</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/what-robotics-teams-need-from-video-datasets-beyond-more-clips</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Sat, 07 Nov 2026 06:41:19 GMT</pubDate><content:encoded><![CDATA[<p>Video data for robotics and embodied AI has different requirements from ordinary video classification.</p>
<p>A single frame may show an object clearly, but it does not explain how the object was approached, manipulated, moved, or used in a real environment.</p>
<p>For robotics workflows, the sequence is often the important part.</p>
<h2>Start with the task</h2>
<p>Before collecting video, define the behavior the model needs to learn or evaluate.</p>
<p>Possible tasks include:</p>
<ul>
<li>Object interaction</li>
<li>Tool use</li>
<li>Navigation</li>
<li>Manipulation</li>
<li>Human demonstrations</li>
<li>First-person task execution</li>
<li>Multi-step household activities</li>
<li>Scene changes during an action</li>
</ul>
<p>The task definition determines which fields, viewpoints, and temporal signals are required.</p>
<h2>Preserve action sequences</h2>
<p>A useful robotics video record may include:</p>
<ul>
<li>Video ID</li>
<li>Source URL</li>
<li>Publication time</li>
<li>Platform</li>
<li>Channel or creator</li>
<li>Scene information</li>
<li>Action labels</li>
<li>Start and end times</li>
<li>Object references</li>
<li>Camera perspective</li>
<li>Audio or narration</li>
<li>Transcript or captions</li>
</ul>
<p>A video without temporal structure may be useful for visual recognition but insufficient for learning a multi-step action.</p>
<h2>Capture context around the action</h2>
<p>The action itself is only part of the signal.</p>
<p>The model may also need to see:</p>
<ul>
<li>The initial environment</li>
<li>The objects involved</li>
<li>The agent’s approach</li>
<li>The action sequence</li>
<li>The result of the action</li>
<li>Changes in the surrounding scene</li>
</ul>
<p>Cropping too aggressively can remove the context needed to understand why the action happened or whether it succeeded.</p>
<h2>Treat viewpoint as a dataset field</h2>
<p>First-person and third-person video support different learning objectives.</p>
<p>First-person footage can show what an agent sees during a task. Third-person footage can provide a clearer view of body movement, object relationships, and overall scene state.</p>
<p>Store the viewpoint explicitly rather than treating all videos as interchangeable.</p>
<p>Other useful fields include camera movement, scene type, task category, and environment.</p>
<h2>Keep temporal continuity</h2>
<p>Randomly sampling isolated frames can destroy the relationship between actions.</p>
<p>For sequence-based workflows, preserve:</p>
<ul>
<li>Frame order</li>
<li>Video timestamps</li>
<li>Action boundaries</li>
<li>Scene transitions</li>
<li>Before-and-after context</li>
<li>Clip relationships</li>
</ul>
<p>If a video is split into shorter clips, keep the parent video ID and original time range with every clip.</p>
<h2>Build task-specific subsets</h2>
<p>A robotics dataset may contain many unrelated videos.</p>
<p>Use structured fields to create subsets by:</p>
<ul>
<li>Task</li>
<li>Object</li>
<li>Environment</li>
<li>Action</li>
<li>Camera perspective</li>
<li>Language</li>
<li>Region</li>
<li>Duration</li>
<li>Scene type</li>
</ul>
<p>This is more useful than filtering only by title or keyword.</p>
<h2>Evaluate sequence quality</h2>
<p>Useful checks include:</p>
<ul>
<li>Are action segments complete?</li>
<li>Are the start and end times valid?</li>
<li>Is the object visible during the action?</li>
<li>Is the result of the action observable?</li>
<li>Are repeated clips or reposts present?</li>
<li>Is the viewpoint correctly labeled?</li>
<li>Does the transcript describe the correct sequence?</li>
<li>Are there enough examples across environments?</li>
</ul>
<p>A dataset can have high video coverage and still lack the repeated task patterns needed for generalization.</p>
<p>Thordata’s current Video Datasets page describes use cases for robotics and embodied AI, including first-person video, task demonstrations, action sequences, scene context, motion patterns, and temporal information.</p>
<p>The page also lists structured metadata, captions, transcripts, audio and scene information, source references, JSON, CSV, and Parquet delivery.</p>
<p>For robotics data, the key question is not how many clips the dataset contains.</p>
<p>It is whether the clips preserve the actions, context, and temporal structure needed by the robot or embodied model.</p>
<p>Use only public or properly authorized data and follow applicable privacy, rights, and platform requirements.</p>
<p>Video dataset overview:</p>
<p><a href="https://www.thordata.com/products/multi-platform-video-datasets">https://www.thordata.com/products/multi-platform-video-datasets</a></p>
<p>Multimodal data solutions:</p>
<p><a href="https://www.thordata.com/solutions/multimodal-data">https://www.thordata.com/solutions/multimodal-data</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[One-Time Dataset or Recurring Delivery? A Practical Decision Guide]]></title><description><![CDATA[A dataset purchase is not always a one-time decision. Some projects need a fixed snapshot for analysis or model evaluation. Others need new and updated…]]></description><link>https://thordata-notes.hashnode.dev/one-time-dataset-or-recurring-delivery-a-practical-decision-guide</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/one-time-dataset-or-recurring-delivery-a-practical-decision-guide</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Fri, 06 Nov 2026 08:16:42 GMT</pubDate><content:encoded><![CDATA[<p>A dataset purchase is not always a one-time decision.</p>
<p>Some projects need a fixed snapshot for analysis or model evaluation. Others need new and updated records delivered on a schedule.</p>
<p>The correct delivery model depends on what the data pipeline needs after the first download.</p>
<h2>When a one-time dataset is enough</h2>
<p>A one-time dataset may be suitable for:</p>
<ul>
<li>Exploratory analysis</li>
<li>Historical research</li>
<li>A model baseline</li>
<li>Prototype development</li>
<li>One-off market research</li>
<li>Initial schema evaluation</li>
<li>A fixed evaluation set</li>
</ul>
<p>In these cases, the main requirements are usually coverage, field quality, provenance, and a stable export format.</p>
<p>The dataset should still include enough metadata to explain when the records were collected and how they were processed.</p>
<h2>When recurring delivery is better</h2>
<p>Recurring delivery is more useful when the workflow needs:</p>
<ul>
<li>New records</li>
<li>Updated records</li>
<li>Trend analysis</li>
<li>Continuous monitoring</li>
<li>Periodic model refreshes</li>
<li>Regular business reports</li>
<li>Long-term market intelligence</li>
</ul>
<p>A recurring workflow should define the update frequency before implementation begins.</p>
<p>Possible schedules include daily, monthly, quarterly, or semi-annual delivery, depending on how quickly the source data changes.</p>
<h2>Prefer incremental updates where possible</h2>
<p>A recurring dataset does not always need to resend the entire corpus.</p>
<p>Incremental delivery can provide:</p>
<ul>
<li>New records since the previous release</li>
<li>Records whose fields changed</li>
<li>Deleted or unavailable records</li>
<li>A release timestamp</li>
<li>A dataset version</li>
<li>A stable record identifier</li>
</ul>
<p>This reduces transfer volume and makes downstream synchronization easier.</p>
<p>The receiving system should be able to distinguish between a new record, an updated record, and a record that should be removed or marked unavailable.</p>
<h2>Treat the schema as a contract</h2>
<p>Recurring data delivery makes schema compatibility more important.</p>
<p>Before accepting a new release, check:</p>
<ul>
<li>Required fields</li>
<li>Data types</li>
<li>Identifier rules</li>
<li>Timestamp formats</li>
<li>Null handling</li>
<li>New or removed fields</li>
<li>Country and language codes</li>
<li>Nested object structure</li>
</ul>
<p>A source may add a field without breaking the data technically, but a renamed field or changed data type can break downstream jobs.</p>
<p>Keep a schema version with every delivery.</p>
<h2>Keep snapshots and change records</h2>
<p>Even when receiving incremental updates, retain release metadata.</p>
<p>A useful delivery record may include:</p>
<ul>
<li>Dataset name</li>
<li>Dataset version</li>
<li>Release timestamp</li>
<li>Previous version</li>
<li>Number of new records</li>
<li>Number of updated records</li>
<li>Number of removed records</li>
<li>Schema version</li>
<li>File checksum</li>
<li>Delivery status</li>
</ul>
<p>This creates a simple audit trail and makes it easier to reproduce an analysis later.</p>
<h2>Match the format to the downstream system</h2>
<p>Different workflows need different formats.</p>
<p>NDJSON can work well for record-oriented processing. JSON is useful for application integration. CSV is convenient for basic analysis and spreadsheet workflows.</p>
<p>The delivery method matters as well. Depending on the project, data may need to arrive through cloud storage, SFTP, webhook, or direct API access.</p>
<p>The format and delivery method should be agreed before the first production release.</p>
<h2>Use sample-first evaluation</h2>
<p>Before committing to a recurring dataset, review a sample release.</p>
<p>Check:</p>
<ul>
<li>Field completeness</li>
<li>Record identifiers</li>
<li>Duplicate behavior</li>
<li>Update semantics</li>
<li>Historical timestamps</li>
<li>Language and region coverage</li>
<li>File format</li>
<li>Delivery path</li>
<li>Schema stability</li>
</ul>
<p>A sample is not only for checking whether the data looks useful. It is also for testing whether the data can enter the existing pipeline without expensive transformation.</p>
<p>Thordata's current Dataset Pricing page describes one-time and subscription access, cleaned and structured data, daily updates, new or updated record delivery, customizable fields, and NDJSON, JSON, and CSV formats.</p>
<p>The Video Datasets page also describes ready-to-use datasets, custom collection, API access, recurring delivery, source tracking, and JSON, CSV, or Parquet output.</p>
<p>Choose a one-time dataset when the project needs a stable snapshot. Choose recurring delivery when freshness, change tracking, or continuous analysis is part of the business requirement.</p>
<p>Dataset information:</p>
<p><a href="https://www.thordata.com/pricing/dataset">https://www.thordata.com/pricing/dataset</a></p>
<p>Video dataset information:</p>
<p><a href="https://www.thordata.com/products/multi-platform-video-datasets">https://www.thordata.com/products/multi-platform-video-datasets</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[Designing a Video Retrieval Dataset: From URLs to Searchable Moments]]></title><description><![CDATA[A video retrieval system needs more than a folder of media files. To search video effectively, the system needs structured references that connect the…]]></description><link>https://thordata-notes.hashnode.dev/designing-a-video-retrieval-dataset-from-urls-to-searchable-moments</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/designing-a-video-retrieval-dataset-from-urls-to-searchable-moments</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Tue, 03 Nov 2026 05:45:11 GMT</pubDate><content:encoded><![CDATA[<p>A video retrieval system needs more than a folder of media files.</p>
<p>To search video effectively, the system needs structured references that connect the source, the content, the language, and, when possible, the relevant moment in the video.</p>
<p>The dataset schema determines what the retrieval system can find.</p>
<h2>Start with the retrieval task</h2>
<p>Before creating embeddings or search indexes, define what the user will search for.</p>
<p>Possible tasks include:</p>
<ul>
<li>Finding videos about a topic</li>
<li>Finding a specific spoken phrase</li>
<li>Finding a visual action</li>
<li>Finding a scene type</li>
<li>Finding videos from a creator or channel</li>
<li>Finding content from a specific time period</li>
<li>Finding similar video segments</li>
<li>Retrieving evidence for a generated answer</li>
</ul>
<p>Each task requires different fields.</p>
<p>Topic search may need titles, descriptions, and categories. Spoken-phrase search needs transcripts and timestamps. Action search may need scene information, temporal labels, or visual embeddings.</p>
<h2>Build a searchable record</h2>
<p>A video retrieval record may include:</p>
<ul>
<li>Video ID</li>
<li>Platform</li>
<li>Original URL</li>
<li>Channel or creator ID</li>
<li>Title</li>
<li>Description</li>
<li>Publication time</li>
<li>Language</li>
<li>Captions</li>
<li>Transcript</li>
<li>Scene information</li>
<li>Engagement fields</li>
<li>Segment start and end times</li>
<li>Dataset version</li>
</ul>
<p>Keep the source fields separate from generated fields.</p>
<p>For example, preserve the original title and transcript while storing translated text, summaries, embeddings, and extracted topics in separate fields.</p>
<p>This makes the index easier to rebuild when the processing model changes.</p>
<h2>Index at more than one level</h2>
<p>A single embedding for an entire video may be useful for broad discovery, but it can be too coarse for specific questions.</p>
<p>Consider indexing:</p>
<ul>
<li>Whole-video metadata</li>
<li>Chapters</li>
<li>Transcript segments</li>
<li>Scene segments</li>
<li>Audio events</li>
<li>Action intervals</li>
</ul>
<p>A user searching for “the moment when the speaker explains the setup” may need a segment-level result instead of an entire ten-minute video.</p>
<p>The dataset should retain the relationship between the segment and its parent video.</p>
<h2>Combine lexical and semantic retrieval</h2>
<p>Keyword search is useful when the query contains an exact name, phrase, product, or identifier.</p>
<p>Semantic search is useful when the query describes an idea using different words from the source content.</p>
<p>A practical system can combine both:</p>
<ol>
<li>Filter by metadata and source fields.</li>
<li>Search captions and transcripts lexically.</li>
<li>Retrieve semantically similar segments.</li>
<li>Rank results using relevance, time, source, and confidence.</li>
<li>Return the original video and the relevant segment.</li>
</ol>
<p>This hybrid approach also makes debugging easier because the system can show which field contributed to a result.</p>
<h2>Preserve temporal context</h2>
<p>A result should show where the match occurs.</p>
<p>Useful temporal fields include:</p>
<ul>
<li>Segment start time</li>
<li>Segment end time</li>
<li>Chapter</li>
<li>Scene label</li>
<li>Transcript sentence</li>
<li>Audio event interval</li>
</ul>
<p>Without temporal context, the user may find the right video but still need to search manually for the relevant moment.</p>
<h2>Track language and source context</h2>
<p>Multilingual retrieval requires explicit language fields.</p>
<p>Store:</p>
<ul>
<li>Original language</li>
<li>Caption language</li>
<li>Transcript language</li>
<li>Translation status</li>
<li>Region or market</li>
<li>Platform</li>
<li>Channel or creator</li>
</ul>
<p>Do not treat translated text as the original source. Keep both values when translation is used.</p>
<h2>Evaluate retrieval quality</h2>
<p>Useful evaluation questions include:</p>
<ul>
<li>Did the system return the correct video?</li>
<li>Did it return the correct segment?</li>
<li>Was the result relevant to the query?</li>
<li>Was the language correct?</li>
<li>Was the source URL preserved?</li>
<li>How many irrelevant results appeared before the relevant result?</li>
<li>Was the temporal range precise enough?</li>
</ul>
<p>Possible metrics include video-level recall, segment recall, precision at K, mean reciprocal rank, and time-to-relevant-segment.</p>
<p>The evaluation set should contain both easy queries and long-tail queries.</p>
<p>Thordata’s current Video Datasets page lists video metadata, captions, transcripts, audio, scene information, source references, semantic search, content retrieval, and temporal context for multimodal workflows.</p>
<p>It also lists more than 17 billion indexed video URLs, more than 700 million independent channels, coverage across more than 100 languages, and delivery in JSON, CSV, or Parquet.</p>
<p>The most useful retrieval dataset is not simply the largest one. It is the one that preserves enough structure to connect a user’s query to the right source and the right moment.</p>
<p>Use only data that is publicly accessible or properly authorized for the intended use.</p>
<p>Video dataset overview:</p>
<p><a href="https://www.thordata.com/products/multi-platform-video-datasets">https://www.thordata.com/products/multi-platform-video-datasets</a></p>
<p>Multimodal data solutions:</p>
<p><a href="https://www.thordata.com/solutions/multimodal-data">https://www.thordata.com/solutions/multimodal-data</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[Designing a Useful Taxonomy for Large Video Datasets]]></title><description><![CDATA[A large video dataset can contain millions of topics, styles, languages, and content formats. Without a clear taxonomy, that scale becomes difficult to…]]></description><link>https://thordata-notes.hashnode.dev/designing-a-useful-taxonomy-for-large-video-datasets</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/designing-a-useful-taxonomy-for-large-video-datasets</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Mon, 02 Nov 2026 07:39:36 GMT</pubDate><content:encoded><![CDATA[<p>A large video dataset can contain millions of topics, styles, languages, and content formats.</p>
<p>Without a clear taxonomy, that scale becomes difficult to search, filter, sample, and evaluate.</p>
<p>A useful taxonomy is not just a list of labels. It is a structure that connects source content with the task the dataset is intended to support.</p>
<h2>Start with the model or business task</h2>
<p>Do not begin by copying categories from a platform.</p>
<p>Start with the intended use case.</p>
<p>A video retrieval system may need labels for:</p>
<ul>
<li>Topic</li>
<li>Scene</li>
<li>Action</li>
<li>Object</li>
<li>Language</li>
<li>Speaker</li>
<li>Time period</li>
</ul>
<p>A recommendation system may need:</p>
<ul>
<li>Content category</li>
<li>Audience signal</li>
<li>Creator type</li>
<li>Engagement range</li>
<li>Publication period</li>
</ul>
<p>A video-generation evaluation set may need:</p>
<ul>
<li>Visual style</li>
<li>Camera movement</li>
<li>Scene transition</li>
<li>Motion pattern</li>
<li>Human action</li>
<li>Temporal consistency</li>
</ul>
<p>The taxonomy should support the decisions the dataset must enable.</p>
<h2>Use multiple levels of labels</h2>
<p>A flat category list often becomes too broad or too specific.</p>
<p>A hierarchical structure can separate:</p>
<ul>
<li>High-level domain</li>
<li>Content category</li>
<li>Subcategory</li>
<li>Specific activity</li>
<li>Scene or visual attribute</li>
</ul>
<p>For example:</p>
<pre><code class="language-text">Sports
  Football
    Training
    Match Highlights
    Interviews
</code></pre>
<p>Not every record needs every level. The important part is that the hierarchy is defined consistently.</p>
<h2>Account for long-tail content</h2>
<p>Popular categories are usually easy to collect.</p>
<p>Long-tail content is more difficult but can be important for model generalization.</p>
<p>Measure the distribution of records across labels and identify:</p>
<ul>
<li>Underrepresented categories</li>
<li>Overrepresented sources</li>
<li>Rare languages</li>
<li>Narrow creator groups</li>
<li>Missing regions</li>
<li>Categories with inconsistent naming</li>
</ul>
<p>A large dataset can still be biased if most records come from a small number of popular topics.</p>
<h2>Separate source labels from task labels</h2>
<p>Platform categories and model categories are not always the same.</p>
<p>Keep source metadata such as:</p>
<ul>
<li>Original platform category</li>
<li>Platform ID</li>
<li>Creator or channel</li>
<li>Original title</li>
<li>Original language</li>
</ul>
<p>Then create task-specific labels separately.</p>
<p>This preserves the original context while allowing different teams to build different views of the same dataset.</p>
<h2>Handle multilingual content deliberately</h2>
<p>A topic may be expressed differently across languages and regions.</p>
<p>Record:</p>
<ul>
<li>Detected language</li>
<li>Original title</li>
<li>Caption language</li>
<li>Transcript language</li>
<li>Translation status</li>
<li>Region or market</li>
</ul>
<p>Avoid treating translated text as if it were original content. Keep both the original and translated values when translations are used.</p>
<h2>Review overlapping labels</h2>
<p>Some videos belong to more than one category.</p>
<p>A cooking video may also be educational. A sports interview may also be news content. A travel video may contain food, culture, and landscape scenes.</p>
<p>Use multi-label fields where the task requires them, and document how overlapping labels are assigned.</p>
<h2>Measure taxonomy quality</h2>
<p>Useful checks include:</p>
<ul>
<li>Label coverage</li>
<li>Label balance</li>
<li>Duplicate labels</li>
<li>Ambiguous definitions</li>
<li>Multi-label frequency</li>
<li>Long-tail coverage</li>
<li>Inter-annotator agreement</li>
<li>Records without task-relevant labels</li>
</ul>
<p>The taxonomy should be reviewed with a sample batch before it is applied to the full dataset.</p>
<p>Thordata’s current Video Datasets page describes customizable sources, categories, regions, fields, and delivery formats. It lists structured metadata, captions, transcripts, audio, scene information, source references, and coverage across more than 100 languages.</p>
<p>The page also lists 17B+ indexed video URLs and 700M+ independent channels. Scale is useful only when the content can be organized into a taxonomy that supports the intended model or analysis task.</p>
<p>Video dataset overview:</p>
<p><a href="https://www.thordata.com/products/multi-platform-video-datasets">https://www.thordata.com/products/multi-platform-video-datasets</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[Why Decodeability Should Come Before Deduplication in Video Dataset Pipelines]]></title><description><![CDATA[A video dataset can look complete while containing files that cannot be decoded reliably. Some files may have incomplete downloads, unsupported codecs…]]></description><link>https://thordata-notes.hashnode.dev/why-decodeability-should-come-before-deduplication-in-video-dataset-pipelines</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/why-decodeability-should-come-before-deduplication-in-video-dataset-pipelines</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Fri, 30 Oct 2026 05:44:19 GMT</pubDate><content:encoded><![CDATA[<p>A video dataset can look complete while containing files that cannot be decoded reliably.</p>
<p>Some files may have incomplete downloads, unsupported codecs, missing audio streams, invalid metadata, truncated frames, or incorrect file extensions.</p>
<p>If these files enter training directly, the problem may appear later as a worker crash, a stalled batch, or a misleading evaluation result.</p>
<p>Decodeability should therefore be an early validation gate.</p>
<h2>Start with a media validation pass</h2>
<p>Before deduplication, caption alignment, and train/test splitting, inspect each media file.</p>
<p>Useful checks include:</p>
<ul>
<li>File can be opened by the selected decoder</li>
<li>Duration is greater than zero</li>
<li>At least one valid video stream exists</li>
<li>Frame dimensions are within the project limits</li>
<li>Frame samples can be decoded</li>
<li>Audio stream exists when required</li>
<li>Codec is supported by the training environment</li>
<li>File size is not suspiciously small</li>
<li>Metadata is internally consistent</li>
</ul>
<p>A file extension is not enough. A file named <code>.mp4</code> may still contain an unsupported or incomplete stream.</p>
<h2>Do not silently discard failures</h2>
<p>When a file fails validation, preserve a quarantine record.</p>
<p>A quarantine manifest may include:</p>
<ul>
<li>Source platform</li>
<li>Source video ID</li>
<li>Original URL</li>
<li>Local or object-storage path</li>
<li>File size</li>
<li>Codec</li>
<li>Duration</li>
<li>Resolution</li>
<li>Failure category</li>
<li>Decoder error</li>
<li>Validation timestamp</li>
<li>Pipeline version</li>
</ul>
<p>This gives the team a way to measure failures and decide whether they should be repaired, replaced, or excluded.</p>
<p>Silently dropping files makes the final dataset smaller without explaining why.</p>
<h2>Apply deduplication after basic validation</h2>
<p>Deduplication is more meaningful after the media can be read.</p>
<p>Use several levels of identity:</p>
<ul>
<li>Canonical source URL</li>
<li>Platform video ID</li>
<li>File hash</li>
<li>Metadata signature</li>
<li>Perceptual hash from sampled frames</li>
</ul>
<p>Exact hashes can detect identical files. Perceptual hashes can help identify re-encoded or resized copies of similar content.</p>
<p>Keep the original source identifiers even after duplicates are grouped. The information may be needed for provenance and later review.</p>
<h2>Validate the audio and text layers separately</h2>
<p>A video may decode correctly while its audio or transcript is unusable.</p>
<p>Check:</p>
<ul>
<li>Audio stream availability</li>
<li>Audio duration</li>
<li>Sample rate</li>
<li>Transcript language</li>
<li>Caption timestamps</li>
<li>Empty transcript segments</li>
<li>Alignment between text and media</li>
</ul>
<p>Do not mark a video as fully valid when only the video stream works but the training task requires audio or captions.</p>
<h2>Quarantine instead of failing everything</h2>
<p>A robust ingestion pipeline should separate fatal infrastructure errors from record-level data failures.</p>
<p>A single corrupt file should usually become a quarantined record rather than stopping the entire batch.</p>
<p>The pipeline can continue processing valid media while reporting:</p>
<ul>
<li>Total input files</li>
<li>Valid files</li>
<li>Quarantined files</li>
<li>Duplicate files</li>
<li>Missing audio</li>
<li>Missing captions</li>
<li>Decode failures</li>
<li>Retryable infrastructure errors</li>
</ul>
<p>This makes the process more resilient and easier to operate.</p>
<h2>Run a second validation after transformation</h2>
<p>Media may change during transcoding, resizing, clipping, or format conversion.</p>
<p>Run validation again after transformation.</p>
<p>Check that:</p>
<ul>
<li>The output file opens</li>
<li>Duration is still correct</li>
<li>Resolution matches the target</li>
<li>Audio remains available</li>
<li>Frames can be sampled</li>
<li>Captions still reference the correct time range</li>
<li>The output path matches the manifest</li>
</ul>
<p>The training pipeline should consume the validated output, not only the original source record.</p>
<h2>Tie validation to the training run</h2>
<p>Every training run should record:</p>
<ul>
<li>Dataset version</li>
<li>Manifest version</li>
<li>Validation rules</li>
<li>Decoder version</li>
<li>Transformation version</li>
<li>Quarantine count</li>
<li>Train, validation, and test split</li>
<li>Source coverage</li>
</ul>
<p>This makes it possible to explain why two runs used different data.</p>
<p>Thordata’s current Video Datasets page lists structured metadata, captions, transcripts, audio and scene information, source references, and delivery in JSON, CSV, or Parquet. It also lists 17B+ indexed video URLs, 700M+ independent channels, and coverage across more than 100 languages.</p>
<p>The main lesson is simple:</p>
<p>Validate whether the media can be used before asking whether it is unique, aligned, or suitable for training.</p>
<p>Use data only within the applicable rights, consent, privacy, and platform requirements.</p>
<p>Video dataset overview:</p>
<p><a href="https://www.thordata.com/products/multi-platform-video-datasets">https://www.thordata.com/products/multi-platform-video-datasets</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[Why Temporal Alignment Matters in Video-Language Datasets]]></title><description><![CDATA[A video and a caption are not automatically a useful training pair. For many multimodal tasks, the model needs to understand not only what appears in a…]]></description><link>https://thordata-notes.hashnode.dev/why-temporal-alignment-matters-in-video-language-datasets</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/why-temporal-alignment-matters-in-video-language-datasets</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Thu, 29 Oct 2026 06:07:09 GMT</pubDate><content:encoded><![CDATA[<p>A video and a caption are not automatically a useful training pair.</p>
<p>For many multimodal tasks, the model needs to understand not only what appears in a video, but when a specific event, action, or scene occurs.</p>
<p>This makes temporal alignment an important part of dataset quality.</p>
<h2>What is temporal alignment?</h2>
<p>Temporal alignment connects a text segment to the relevant part of a video.</p>
<p>For example:</p>
<ul>
<li>A transcript sentence maps to a time interval.</li>
<li>A caption describes a specific scene.</li>
<li>An action label identifies when an action starts and ends.</li>
<li>An audio segment corresponds to a spoken phrase.</li>
<li>A scene summary covers a defined sequence of frames.</li>
</ul>
<p>Without timing information, a long video may be paired with text that is generally related but not useful for moment-level learning.</p>
<h2>Check the text-video relationship</h2>
<p>A video-language dataset should be evaluated at more than the record level.</p>
<p>Useful checks include:</p>
<ul>
<li>Does the caption describe the correct video?</li>
<li>Does the transcript match the spoken audio?</li>
<li>Does the text correspond to the correct time range?</li>
<li>Are captions duplicated or empty?</li>
<li>Is the language label correct?</li>
<li>Are timestamps valid and ordered?</li>
<li>Do scene boundaries match visible changes?</li>
</ul>
<p>A caption field being present does not prove that it is aligned.</p>
<h2>Store temporal fields explicitly</h2>
<p>Depending on the task, a record may include:</p>
<ul>
<li>Video ID</li>
<li>Source URL</li>
<li>Start time</li>
<li>End time</li>
<li>Caption text</li>
<li>Transcript text</li>
<li>Language</li>
<li>Scene label</li>
<li>Speaker information</li>
<li>Processing version</li>
</ul>
<p>For example:</p>
<pre><code class="language-json">{
  "video_id": "example-001",
  "start_time": 42.5,
  "end_time": 48.2,
  "text": "The cyclist turns onto the road.",
  "language": "en",
  "alignment_status": "reviewed"
}
</code></pre>
<p>The exact schema will vary, but the important point is that text should be connected to a defined part of the video.</p>
<h2>Use alignment metrics</h2>
<p>Possible quality metrics include:</p>
<ul>
<li>Timestamp validity rate</li>
<li>Caption coverage</li>
<li>Transcript availability</li>
<li>Language identification accuracy</li>
<li>Segment overlap</li>
<li>Empty-text rate</li>
<li>Duplicate segment rate</li>
<li>Human review agreement</li>
</ul>
<p>A dataset can have broad platform coverage and still perform poorly if these fields are unreliable.</p>
<h2>Align the dataset with the model task</h2>
<p>Different tasks need different levels of annotation.</p>
<p>Video retrieval may need searchable metadata and approximate temporal segments.</p>
<p>Video question answering may require precise event timing.</p>
<p>Speech understanding may depend on transcript accuracy and speaker separation.</p>
<p>Action recognition may require start and end boundaries for the action.</p>
<p>Video generation evaluation may need scene transitions, motion patterns, and temporal consistency.</p>
<p>The dataset should be evaluated against the task rather than against a generic completeness checklist.</p>
<h2>Keep source and processing information</h2>
<p>Track the source platform, original video URL, platform ID, channel information, collection time, and processing version.</p>
<p>This helps teams reproduce a sample, investigate an alignment error, and compare later dataset releases.</p>
<p>Thordata’s current Video Datasets page lists metadata, captions, transcripts, audio, scene information, source tracking, and temporal context for multimodal workflows. It also lists delivery in JSON, CSV, and Parquet formats, with API and cloud delivery options.</p>
<p>The page states coverage of more than 100 languages and more than 17 billion indexed video URLs. These scale figures should be evaluated together with the quality and alignment of the fields required by the model.</p>
<p>Use video data only within the applicable rights, consent, privacy, and platform requirements.</p>
<p>Video dataset overview:</p>
<p><a href="https://www.thordata.com/products/multi-platform-video-datasets">https://www.thordata.com/products/multi-platform-video-datasets</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[A Practical Quality Checklist for Multimodal Video Datasets]]></title><description><![CDATA[A video dataset can be large, diverse, and expensive while still producing unreliable training results. The problem is usually not the file count. It is…]]></description><link>https://thordata-notes.hashnode.dev/a-practical-quality-checklist-for-multimodal-video-datasets</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/a-practical-quality-checklist-for-multimodal-video-datasets</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Mon, 26 Oct 2026 07:13:46 GMT</pubDate><content:encoded><![CDATA[<p>A video dataset can be large, diverse, and expensive while still producing unreliable training results.</p>
<p>The problem is usually not the file count. It is the lack of a clear quality contract.</p>
<p>Before a dataset enters a training pipeline, define what a valid record means and how the dataset will be checked.</p>
<h2>1. Define the record schema</h2>
<p>A video record may contain:</p>
<ul>
<li>Video ID</li>
<li>Source platform</li>
<li>Original URL</li>
<li>Channel or creator ID</li>
<li>Title</li>
<li>Description</li>
<li>Publication time</li>
<li>Language</li>
<li>Category</li>
<li>Engagement metrics</li>
<li>Captions</li>
<li>Transcript</li>
<li>Audio reference</li>
<li>Scene information</li>
<li>Collection timestamp</li>
</ul>
<p>The exact fields depend on the model task, but the schema should be defined before model training begins.</p>
<h2>2. Check source provenance</h2>
<p>Every record should be traceable to its original source.</p>
<p>Useful provenance fields include:</p>
<ul>
<li>Platform name</li>
<li>Platform-specific ID</li>
<li>Original URL</li>
<li>Channel ID</li>
<li>Collection timestamp</li>
<li>Dataset version</li>
<li>Processing version</li>
</ul>
<p>This makes it possible to investigate duplicates, remove affected records, reproduce a sample, and understand how the dataset changed over time.</p>
<h2>3. Measure metadata completeness</h2>
<p>Calculate the completion rate for each important field.</p>
<p>For example:</p>
<ul>
<li>Title completion</li>
<li>Description completion</li>
<li>Caption availability</li>
<li>Transcript availability</li>
<li>Language identification</li>
<li>Publication time</li>
<li>Source URL validity</li>
</ul>
<p>A dataset may have a high total record count but a low completion rate for the fields needed by the model.</p>
<h2>4. Detect duplicates and near duplicates</h2>
<p>Video duplication can happen at several levels:</p>
<ul>
<li>Identical source URLs</li>
<li>Reposted videos</li>
<li>Multiple platform copies</li>
<li>Repeated clips</li>
<li>Different resolutions of the same source</li>
<li>Near-duplicate frames</li>
</ul>
<p>Use stable source IDs where possible, then add content-level checks for repeated or highly similar media.</p>
<p>Duplicates can distort training distribution and make evaluation results look better than they really are.</p>
<h2>5. Check video-text alignment</h2>
<p>For multimodal training, the relationship between video, audio, captions, and transcripts matters.</p>
<p>Review whether:</p>
<ul>
<li>Captions belong to the correct video.</li>
<li>Transcript language matches the source.</li>
<li>Timestamps align with the video.</li>
<li>Audio is available when required.</li>
<li>Scene descriptions refer to the correct time range.</li>
<li>Empty or placeholder text has been removed.</li>
</ul>
<p>A caption field being present does not prove that it is useful.</p>
<h2>6. Split by source and time</h2>
<p>Random row-level splitting can create leakage.</p>
<p>The same creator, series, repost, or near-duplicate video may appear in both training and evaluation sets.</p>
<p>Where appropriate, consider:</p>
<ul>
<li>Channel-level splits</li>
<li>Creator-level splits</li>
<li>Time-based splits</li>
<li>Topic-balanced splits</li>
<li>Platform-balanced splits</li>
</ul>
<p>The split strategy should reflect how the model will be used in production.</p>
<h2>7. Track dataset versions</h2>
<p>A training run should be linked to a specific dataset version.</p>
<p>Record:</p>
<ul>
<li>Dataset version</li>
<li>Schema version</li>
<li>Processing code version</li>
<li>Source coverage</li>
<li>Filtering rules</li>
<li>Removed records</li>
<li>Train/validation/test split</li>
<li>Delivery timestamp</li>
</ul>
<p>Without version information, it becomes difficult to reproduce a model result or explain why a later run changed.</p>
<h2>8. Use a sample before full delivery</h2>
<p>Review a representative sample before committing to a full dataset.</p>
<p>Inspect:</p>
<ul>
<li>Different platforms</li>
<li>Different languages</li>
<li>Different content categories</li>
<li>Short and long videos</li>
<li>Videos with and without transcripts</li>
<li>Different publication periods</li>
<li>Different creator types</li>
</ul>
<p>Thordata’s current Video Datasets page lists structured metadata, captions, transcripts, audio and scene information, source tracking, JSON/CSV/Parquet delivery, and coverage across more than 100 languages.</p>
<p>The page also lists 17B+ indexed video URLs and 700M+ independent channels. These scale figures should be evaluated together with schema quality, provenance, and sample-level validation.</p>
<p>A multimodal dataset is ready for training only when the team can explain what each record contains, where it came from, how it was processed, and how it will be evaluated.</p>
<p>Use data only within the applicable rights, consent, privacy, and platform requirements.</p>
<p>Video dataset overview:</p>
<p><a href="https://www.thordata.com/products/multi-platform-video-datasets">https://www.thordata.com/products/multi-platform-video-datasets</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[A Practical API Key Security Workflow for n8n Data Automations]]></title><description><![CDATA[An API key is not configuration text. It is a credential that grants access to a service, consumes usage, and may connect to sensitive business workflows…]]></description><link>https://thordata-notes.hashnode.dev/a-practical-api-key-security-workflow-for-n8n-data-automations</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/a-practical-api-key-security-workflow-for-n8n-data-automations</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Sat, 24 Oct 2026 05:39:35 GMT</pubDate><content:encoded><![CDATA[<p>An API key is not configuration text.</p>
<p>It is a credential that grants access to a service, consumes usage, and may connect to sensitive business workflows.</p>
<p>A secure automation design should protect the key throughout its lifecycle: creation, storage, use, logging, rotation, and revocation.</p>
<h2>1. Store credentials outside workflow logic</h2>
<p>Do not place an API key directly in:</p>
<ul>
<li>Code nodes</li>
<li>Query parameters</li>
<li>Shared workflow JSON</li>
<li>Public repositories</li>
<li>Screenshots</li>
<li>Documentation examples</li>
<li>Email messages</li>
<li>Chat transcripts</li>
</ul>
<p>Use the automation platform's credential system instead.</p>
<p>Thordata's n8n integration documentation describes creating a dedicated Thordata SERP API credential and using it from the Thordata SERP node.</p>
<p>The workflow should reference the credential without exposing the secret to individual nodes.</p>
<h2>2. Separate environments</h2>
<p>Use different credentials for:</p>
<ul>
<li>Local testing</li>
<li>Staging workflows</li>
<li>Production workflows</li>
<li>Temporary experiments</li>
</ul>
<p>This limits the impact of a leaked test key and makes it easier to revoke one workflow without interrupting every automation.</p>
<p>Environment separation also makes usage easier to investigate. If a production key is used from an unexpected workflow, the team can identify the source more quickly.</p>
<h2>3. Apply least privilege</h2>
<p>A workflow should have access only to the services and data it needs.</p>
<p>For example, a daily keyword-monitoring workflow may need SERP access, but it does not need access to unrelated storage systems or administrative credentials.</p>
<p>Keep separate credentials for separate business functions when practical.</p>
<p>Least privilege reduces the impact of accidental exposure and limits the damage caused by a compromised workflow.</p>
<h2>4. Protect logs and execution history</h2>
<p>Request logs can expose secrets even when the workflow itself is private.</p>
<p>Review:</p>
<ul>
<li>Authorization headers</li>
<li>Query strings</li>
<li>Error messages</li>
<li>Code-node output</li>
<li>Failed execution data</li>
<li>Exported workflow files</li>
<li>Debug screenshots</li>
</ul>
<p>Use redaction where possible. Never print the full API key in a Code node or debugging message.</p>
<p>The same rule applies to support tickets and shared internal documents.</p>
<h2>5. Rotate keys deliberately</h2>
<p>Key rotation should be a scheduled operation rather than an emergency response.</p>
<p>A basic rotation process is:</p>
<ol>
<li>Create a replacement key.</li>
<li>Store it in the credential manager.</li>
<li>Test the workflow with the replacement.</li>
<li>Monitor the next production run.</li>
<li>Revoke the old key.</li>
<li>Record the rotation date and owner.</li>
</ol>
<p>Do not revoke the old key before confirming that the replacement works.</p>
<h2>6. Create an incident path</h2>
<p>If a key may have been exposed, the team should know what to do.</p>
<p>The response should include:</p>
<ul>
<li>Disable or revoke the exposed key.</li>
<li>Check recent usage and execution logs.</li>
<li>Identify affected workflows.</li>
<li>Create a replacement key.</li>
<li>Update the credential store.</li>
<li>Review how the exposure happened.</li>
<li>Remove the secret from shared locations.</li>
</ul>
<p>Fast revocation is more useful than assuming that a private workflow can never leak.</p>
<h2>7. Validate the workflow after credential changes</h2>
<p>A successful credential test does not prove that the complete workflow is correct.</p>
<p>After updating a key, test:</p>
<ul>
<li>The selected search operation</li>
<li>Required query fields</li>
<li>Country and language parameters</li>
<li>Structured JSON output</li>
<li>Downstream database or notification nodes</li>
<li>Error handling</li>
<li>Usage reporting</li>
</ul>
<p>Thordata's n8n documentation describes passing structured SERP JSON to AI agents, Code nodes, databases, spreadsheets, notification services, and reporting tools.</p>
<h2>8. Keep compliance in scope</h2>
<p>Credentials protect access to the API. They do not make every workflow permissible.</p>
<p>Use the service only for lawful, authorized public-data workflows. Do not use it for nonpublic information, fake engagement, ad fraud, SEO manipulation, spam, or other activities prohibited by the Acceptable Use Policy.</p>
<p>The current SERP API pricing page lists a 7-day free trial with 500 responses. Paid plans start at $1.20 per 1,000 responses, with high-volume pricing down to $0.70 per 1,000 responses.</p>
<p>n8n integration documentation:</p>
<p><a href="https://doc.thordata.com/doc/scraping/serp-api/integration/sdk-integration/how-to-integrate-thordata-with-n8n.md">https://doc.thordata.com/doc/scraping/serp-api/integration/sdk-integration/how-to-integrate-thordata-with-n8n.md</a></p>
<p>n8n integration page:</p>
<p><a href="https://www.thordata.com/products/serp-api/n8n-integration">https://www.thordata.com/products/serp-api/n8n-integration</a></p>
<p>Acceptable Use Policy:</p>
<p><a href="https://www.thordata.com/acceptable-use-policy">https://www.thordata.com/acceptable-use-policy</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[Building Observability into a Web Data Pipeline]]></title><description><![CDATA[A scraping pipeline can show healthy request logs while producing incomplete or incorrect data. Responses may contain an access page, a CAPTCHA challenge…]]></description><link>https://thordata-notes.hashnode.dev/building-observability-into-a-web-data-pipeline</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/building-observability-into-a-web-data-pipeline</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Fri, 23 Oct 2026 02:39:35 GMT</pubDate><content:encoded><![CDATA[<p>A scraping pipeline can show healthy request logs while producing incomplete or incorrect data.</p>
<p>Responses may contain an access page, a CAPTCHA challenge, an empty JavaScript shell, a changed page structure, or a partial record.</p>
<p>This is why web data observability must continue after the HTTP response is received.</p>
<h2>Separate transport success from data success</h2>
<p>A transport-success metric answers:</p>
<blockquote>
<p>Did the request receive a response?</p>
</blockquote>
<p>A data-success metric answers:</p>
<blockquote>
<p>Did the response produce a valid record?</p>
</blockquote>
<p>These are different measurements.</p>
<p>A useful pipeline should track both.</p>
<p>For each task, record:</p>
<ul>
<li>Request status</li>
<li>Response type</li>
<li>Source URL</li>
<li>Parser status</li>
<li>Required-field completion</li>
<li>Validation result</li>
<li>Collection timestamp</li>
<li>Task ID</li>
<li>Scraper or parser version</li>
</ul>
<h2>Monitor the right metrics</h2>
<p>Important metrics include:</p>
<h3>Valid-record rate</h3>
<p>How many requests produced records that passed validation?</p>
<h3>Field completeness</h3>
<p>How often are required fields present and usable?</p>
<h3>Challenge-page rate</h3>
<p>How often did the response contain an access page, CAPTCHA, or unexpected page type?</p>
<h3>Duplicate rate</h3>
<p>Are repeated records entering the dataset?</p>
<h3>Schema-drift rate</h3>
<p>Are fields missing, renamed, or changing type compared with previous runs?</p>
<h3>Cost per usable record</h3>
<p>How much did it cost to produce one validated record?</p>
<p>This is usually more useful than cost per request.</p>
<h2>Add response validation</h2>
<p>A response should not be marked successful only because it returned HTTP 200.</p>
<p>Validate:</p>
<ol>
<li>The source URL matches the requested URL.</li>
<li>The page belongs to the expected source and page type.</li>
<li>Required fields are present.</li>
<li>Values have the expected format.</li>
<li>The record is not duplicated.</li>
<li>The response is not an access, login, consent, or challenge page.</li>
<li>The schema matches the current version.</li>
</ol>
<p>Invalid records should be stored with a clear failure state instead of being silently discarded.</p>
<h2>Detect schema drift</h2>
<p>Websites change.</p>
<p>A field may disappear, change from a number to a string, move to a nested object, or become unavailable for only some page types.</p>
<p>Keep a schema contract for each scraper. Compare incoming records with the expected contract and alert when:</p>
<ul>
<li>A required field disappears.</li>
<li>A field changes type.</li>
<li>Null values increase sharply.</li>
<li>A new field appears unexpectedly.</li>
<li>The number of extracted records drops.</li>
<li>A parser returns empty values for a previously stable field.</li>
</ul>
<p>Schema alerts should be separate from ordinary network alerts because they usually require parser or source review.</p>
<h2>Classify failures</h2>
<p>Use separate states for different problems:</p>
<ul>
<li><code>transport_failed</code></li>
<li><code>access_challenge</code></li>
<li><code>render_failed</code></li>
<li><code>parse_failed</code></li>
<li><code>validation_failed</code></li>
<li><code>schema_changed</code></li>
<li><code>duplicate_record</code></li>
<li><code>valid_record</code></li>
</ul>
<p>This makes recovery more precise.</p>
<p>A timeout may justify a limited retry. A parser failure needs investigation. A schema change may require a code update. Retrying all of them in the same way increases cost without improving data quality.</p>
<h2>Monitor scheduled tasks</h2>
<p>Thordata Web Scraper API supports scheduled tasks using visual settings or five-field Cron expressions.</p>
<p>A scheduled workflow should record:</p>
<ul>
<li>Expected run time</li>
<li>Actual start time</li>
<li>Completion time</li>
<li>Number of inputs</li>
<li>Number of valid records</li>
<li>Number of failed records</li>
<li>Delivery status</li>
</ul>
<p>A task that runs on schedule but produces zero valid records should be treated as a pipeline failure.</p>
<h2>Build an operational dashboard</h2>
<p>A practical dashboard can show:</p>
<ul>
<li>Valid-record rate over time</li>
<li>Required-field completeness</li>
<li>Errors by source</li>
<li>Schema changes</li>
<li>Duplicate rate</li>
<li>Average latency</li>
<li>Cost per valid record</li>
<li>Last successful task</li>
<li>Last known parser version</li>
</ul>
<p>These metrics help teams identify whether a problem belongs to access, parsing, validation, source changes, or downstream delivery.</p>
<p>The current Web Scraper API pricing page lists a 7-day free trial with 5,000 credits. Paid plans start at $1.00 per 1,000 results, with high-volume pricing down to $0.50 per 1,000 results.</p>
<p>Use Web Scraper API only for lawful, authorized public-data workflows and respect applicable website terms and restrictions.</p>
<p>Web Scraper API documentation:</p>
<p><a href="https://doc.thordata.com/doc/scraping/web-scraper-api.md">https://doc.thordata.com/doc/scraping/web-scraper-api.md</a></p>
<p>Scheduler documentation:</p>
<p><a href="https://doc.thordata.com/doc/scraping/web-scraper-api/scheduler.md">https://doc.thordata.com/doc/scraping/web-scraper-api/scheduler.md</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[Building an Automated News Monitoring Workflow with n8n]]></title><description><![CDATA[A daily news digest does not need to start with a complex AI agent. A predictable workflow can be built from four basic stages: This structure is easier to…]]></description><link>https://thordata-notes.hashnode.dev/building-an-automated-news-monitoring-workflow-with-n8n</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/building-an-automated-news-monitoring-workflow-with-n8n</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Thu, 22 Oct 2026 02:44:56 GMT</pubDate><content:encoded><![CDATA[<p>A daily news digest does not need to start with a complex AI agent.</p>
<p>A predictable workflow can be built from four basic stages:</p>
<ol>
<li>Search for current articles.</li>
<li>Filter and normalize the results.</li>
<li>Format a readable digest.</li>
<li>Deliver it by email or another notification channel.</li>
</ol>
<p>This structure is easier to debug and makes the original sources visible to the reader.</p>
<h2>1. Define the monitoring scope</h2>
<p>Start with a list of topics instead of one broad query.</p>
<p>Examples include:</p>
<ul>
<li>Competitor names</li>
<li>Product categories</li>
<li>Industry regulations</li>
<li>Technology releases</li>
<li>Market trends</li>
<li>Brand mentions</li>
<li>Regional developments</li>
</ul>
<p>Each topic should have a purpose and a delivery frequency. A daily digest may be suitable for fast-moving topics, while a weekly report may be enough for slower categories.</p>
<h2>2. Connect the search layer</h2>
<p>Thordata SERP API can be connected to n8n through the Thordata integration.</p>
<p>The n8n workflow can use structured results from Google, Bing, DuckDuckGo, Yandex, and other supported search operations.</p>
<p>A typical configuration includes:</p>
<ul>
<li>Search query</li>
<li>Country or region</li>
<li>Language</li>
<li>Result count</li>
<li>Search type</li>
<li>Time schedule</li>
</ul>
<p>The n8n integration documentation describes both fixed workflows through a community node and AI-agent workflows through MCP.</p>
<h2>3. Normalize the results</h2>
<p>Search results from different queries may contain duplicate URLs or repeated coverage of the same story.</p>
<p>Before creating the digest, normalize:</p>
<ul>
<li>Article URL</li>
<li>Title</li>
<li>Source name</li>
<li>Publication date</li>
<li>Search query</li>
<li>Result snippet</li>
<li>Collection timestamp</li>
</ul>
<p>Use the canonical URL where possible. Remove tracking parameters and deduplicate identical articles.</p>
<h2>4. Apply simple filters</h2>
<p>A useful filter can remove:</p>
<ul>
<li>Duplicate URLs</li>
<li>Empty titles</li>
<li>Missing links</li>
<li>Irrelevant domains</li>
<li>Articles outside the requested time range</li>
<li>Results that match a keyword but not the intended topic</li>
</ul>
<p>Filtering before summarization reduces unnecessary processing and keeps the final digest focused.</p>
<h2>5. Preserve source links</h2>
<p>A digest should make it easy to inspect the original article.</p>
<p>Every item should retain:</p>
<ul>
<li>Title</li>
<li>Source</li>
<li>Publication time when available</li>
<li>Original URL</li>
<li>Query or topic that produced the result</li>
</ul>
<p>Summaries can improve scanning, but they should not replace source access.</p>
<h2>6. Choose deterministic automation first</h2>
<p>A fixed workflow is often better than an AI agent for a simple daily digest.</p>
<p>A deterministic workflow is easier to test because the same nodes run in the same order. It can still pass results to an AI node later for summarization or categorization.</p>
<p>Use an AI agent when the workflow needs dynamic research decisions. Use fixed nodes when the task is predictable and repeatable.</p>
<h2>7. Deliver the digest</h2>
<p>The final n8n workflow can send the formatted results to:</p>
<ul>
<li>Email</li>
<li>Slack</li>
<li>Discord</li>
<li>Notion</li>
<li>Google Sheets</li>
<li>A database</li>
<li>An internal API</li>
</ul>
<p>The delivery format should match the reader. Email may work well for a morning digest, while a database or spreadsheet may be better for long-term monitoring.</p>
<p>The current SERP API pricing page lists a 7-day free trial with 500 responses. Paid plans start at \(1.20 per 1,000 responses, with high-volume pricing down to \)0.70 per 1,000 responses.</p>
<p>Use only lawful, authorized public sources. Keep API keys private, retain original links, and respect applicable platform terms.</p>
<p>n8n integration documentation:</p>
<p><a href="https://doc.thordata.com/doc/scraping/serp-api/integration/sdk-integration/how-to-integrate-thordata-with-n8n.md">https://doc.thordata.com/doc/scraping/serp-api/integration/sdk-integration/how-to-integrate-thordata-with-n8n.md</a></p>
<p>n8n product page:</p>
<p><a href="https://www.thordata.com/products/serp-api/n8n-integration">https://www.thordata.com/products/serp-api/n8n-integration</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[Building a Customer Review Intelligence Workflow]]></title><description><![CDATA[Customer reviews contain more than a rating. They describe product expectations, recurring problems, competing products, delivery experiences, missing…]]></description><link>https://thordata-notes.hashnode.dev/building-a-customer-review-intelligence-workflow</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/building-a-customer-review-intelligence-workflow</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Wed, 21 Oct 2026 03:11:32 GMT</pubDate><content:encoded><![CDATA[<p>Customer reviews contain more than a rating.</p>
<p>They describe product expectations, recurring problems, competing products, delivery experiences, missing features, and the reasons customers decide to recommend or reject an item.</p>
<p>The challenge is turning thousands of individual reviews into structured product intelligence.</p>
<h2>1. Define the business question</h2>
<p>Start with a question that can guide the analysis.</p>
<p>Examples include:</p>
<ul>
<li>Why are customers returning this product?</li>
<li>Which product features drive satisfaction?</li>
<li>What problems appear repeatedly after purchase?</li>
<li>How does customer feedback differ from competitors?</li>
<li>Which complaints are increasing over time?</li>
<li>What do customers compare before making a purchase?</li>
</ul>
<p>A clear question determines which fields and review segments are needed.</p>
<h2>2. Collect the right fields</h2>
<p>A review intelligence record may include:</p>
<ul>
<li>Product ID</li>
<li>Product title</li>
<li>Brand</li>
<li>Rating</li>
<li>Review text</li>
<li>Review date</li>
<li>Verified-purchase status</li>
<li>Variant information</li>
<li>Helpful-vote count</li>
<li>Source URL</li>
<li>Collection timestamp</li>
</ul>
<p>The product identifier is especially important. It allows review records to be joined with product information, categories, sellers, and model versions.</p>
<h2>3. Separate reviews by purpose</h2>
<p>Do not analyze all reviews as one group.</p>
<p>Useful segments may include:</p>
<ul>
<li>One- and two-star reviews</li>
<li>Three-star reviews</li>
<li>Four- and five-star reviews</li>
<li>Verified purchases</li>
<li>Recent reviews</li>
<li>Reviews mentioning returns</li>
<li>Reviews mentioning competitors</li>
<li>Reviews for different product variants</li>
</ul>
<p>Comparing segments often reveals more than calculating a single average sentiment score.</p>
<h2>4. Extract recurring topics</h2>
<p>Sentiment answers whether a review sounds positive or negative.</p>
<p>Topic analysis helps explain why.</p>
<p>Possible topics include:</p>
<ul>
<li>Product quality</li>
<li>Durability</li>
<li>Performance</li>
<li>Setup</li>
<li>Noise</li>
<li>Size</li>
<li>Packaging</li>
<li>Delivery</li>
<li>Customer support</li>
<li>Compatibility</li>
<li>Value for money</li>
</ul>
<p>A useful workflow can rank topics by frequency and compare their share across ratings or time periods.</p>
<p>For example, a topic that appears disproportionately in low-rated reviews may deserve more attention than a general negative-sentiment score.</p>
<h2>5. Compare customer language over time</h2>
<p>Review dates make it possible to monitor changes after:</p>
<ul>
<li>A product redesign</li>
<li>A supplier change</li>
<li>A packaging update</li>
<li>A price change</li>
<li>A new product version</li>
<li>A customer-support policy change</li>
</ul>
<p>The workflow should preserve the original review date and collection timestamp so historical analysis remains possible.</p>
<h2>6. Connect reviews to product decisions</h2>
<p>The final output should not stop at a topic list.</p>
<p>A product team may need:</p>
<ul>
<li>Top complaint themes</li>
<li>Positive features to preserve</li>
<li>New feature requests</li>
<li>Competitor comparisons</li>
<li>Common return reasons</li>
<li>Topic changes by product version</li>
<li>Topic changes by market</li>
</ul>
<p>This turns review mining into a product intelligence workflow.</p>
<h2>7. Validate the data</h2>
<p>Before analysis, check:</p>
<ul>
<li>Duplicate reviews</li>
<li>Missing product identifiers</li>
<li>Incomplete review text</li>
<li>Invalid ratings</li>
<li>Inconsistent dates</li>
<li>Different language versions</li>
<li>Variant mismatches</li>
</ul>
<p>A sentiment model cannot repair a dataset that joins reviews to the wrong product.</p>
<p>Thordata’s Review Monitoring solution describes collecting public review data across platforms and regions for customer feedback analysis and brand monitoring.</p>
<p>The official Amazon scraping documentation includes a review-specific scraper using <code>amazon_comment_by-url</code>, with product URLs as the input.</p>
<p>The current Web Scraper API pricing page lists a 7-day free trial with 5,000 credits. Paid plans start at \(1.00 per 1,000 results, with high-volume pricing down to \)0.50 per 1,000 results.</p>
<p>Use review data only from permitted public sources and follow applicable platform terms, privacy requirements, and data-use restrictions.</p>
<p>Review monitoring:</p>
<p><a href="https://www.thordata.com/solutions/review-monitoring">https://www.thordata.com/solutions/review-monitoring</a></p>
<p>Amazon review scraping documentation:</p>
<p><a href="https://doc.thordata.com/doc/scraping/web-scraper-api/parameter-description/amazon-scraping-parameters.md">https://doc.thordata.com/doc/scraping/web-scraper-api/parameter-description/amazon-scraping-parameters.md</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[How to Evaluate a Web Data Provider Before Moving to Production]]></title><description><![CDATA[A web data provider should not be evaluated only by its request price or the number of supported websites. The real question is whether it can consistently…]]></description><link>https://thordata-notes.hashnode.dev/how-to-evaluate-a-web-data-provider-before-moving-to-production</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/how-to-evaluate-a-web-data-provider-before-moving-to-production</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Tue, 20 Oct 2026 02:51:04 GMT</pubDate><content:encoded><![CDATA[<p>A web data provider should not be evaluated only by its request price or the number of supported websites.</p>
<p>The real question is whether it can consistently deliver the fields, freshness, coverage, and output format that a business workflow requires.</p>
<p>A short, controlled pilot is usually more useful than a long feature comparison.</p>
<h2>1. Define the acceptance criteria</h2>
<p>Before testing a provider, define what a successful result means.</p>
<p>Useful criteria include:</p>
<ul>
<li>Required-field completion</li>
<li>Valid-record rate</li>
<li>Source and URL coverage</li>
<li>Data freshness</li>
<li>Duplicate rate</li>
<li>Error rate</li>
<li>Response time</li>
<li>Output format</li>
<li>Integration effort</li>
<li>Cost per valid record</li>
</ul>
<p>Without acceptance criteria, a pilot can become a collection of impressions instead of a measurable evaluation.</p>
<h2>2. Build a representative test set</h2>
<p>Do not test only easy URLs.</p>
<p>A representative test set should include:</p>
<ul>
<li>Common page types</li>
<li>Dynamic pages</li>
<li>Pages with missing fields</li>
<li>Different categories or regions</li>
<li>Recently updated pages</li>
<li>URLs that previously caused failures</li>
<li>A small number of edge cases</li>
</ul>
<p>The goal is to model the production workload, not to produce a perfect demo.</p>
<h2>3. Compare the schema</h2>
<p>Two providers may both return JSON while producing very different data quality.</p>
<p>Check:</p>
<ul>
<li>Field names</li>
<li>Data types</li>
<li>Null handling</li>
<li>Nested structures</li>
<li>Currency and unit formats</li>
<li>Product or content identifiers</li>
<li>Timestamp formats</li>
<li>Source URL preservation</li>
</ul>
<p>Schema consistency determines how much post-processing your team will need.</p>
<p>Thordata Web Scraper API provides structured output in JSON, CSV, and XLSX formats and offers preconfigured scrapers for more than 120 public domains.</p>
<h2>4. Measure usable records</h2>
<p>Request volume is not the same as usable output.</p>
<p>A useful metric is:</p>
<pre><code class="language-text">cost per valid record = total collection cost / number of validated records
</code></pre>
<p>A record should count as valid only when the required fields are present, the source is correct, and the values pass basic validation.</p>
<p>This metric is more meaningful than the nominal cost per request.</p>
<h2>5. Test failure handling</h2>
<p>Ask what happens when:</p>
<ul>
<li>A page is unavailable</li>
<li>A required field is missing</li>
<li>A URL is malformed</li>
<li>A source changes its layout</li>
<li>A request times out</li>
<li>The returned page is incomplete</li>
<li>The data does not match the expected schema</li>
</ul>
<p>A production workflow needs clear status values and a way to identify failed URLs.</p>
<p>Do not treat every incomplete record as a successful response.</p>
<h2>6. Evaluate freshness and repeatability</h2>
<p>Run the same sample at different times.</p>
<p>Compare:</p>
<ul>
<li>Field stability</li>
<li>Update delay</li>
<li>Duplicate behavior</li>
<li>Timestamp accuracy</li>
<li>Results after source changes</li>
<li>Differences between repeated requests</li>
</ul>
<p>A provider may perform well during one test and produce inconsistent results under a recurring schedule.</p>
<h2>7. Test the handoff to your systems</h2>
<p>A data provider is part of a larger pipeline.</p>
<p>Test whether the output can be sent to:</p>
<ul>
<li>A database</li>
<li>A data warehouse</li>
<li>An internal API</li>
<li>A spreadsheet workflow</li>
<li>A cloud storage location</li>
<li>A scheduled processing job</li>
</ul>
<p>Also measure how much custom transformation is required before another team can use the data.</p>
<h2>8. Define migration and exit criteria</h2>
<p>Before committing to a provider, document:</p>
<ul>
<li>Export formats</li>
<li>Historical data access</li>
<li>Schema versioning</li>
<li>Task and request logs</li>
<li>Error visibility</li>
<li>Account and usage reporting</li>
<li>Support response process</li>
<li>Replacement or migration effort</li>
</ul>
<p>This prevents the team from discovering operational dependencies only after production adoption.</p>
<p>The current Web Scraper API pricing page lists paid plans from \(1.00 per 1,000 results, with high-volume pricing down to \)0.50 per 1,000 results. Pricing should be evaluated together with valid-record rate and post-processing cost.</p>
<p>A responsible pilot should use lawful, authorized public-data workflows and respect applicable website terms and restrictions.</p>
<p>Product page:</p>
<p><a href="https://www.thordata.com/products/web-scraper">https://www.thordata.com/products/web-scraper</a></p>
<p>Documentation:</p>
<p><a href="https://doc.thordata.com/doc/scraping/web-scraper-api.md">https://doc.thordata.com/doc/scraping/web-scraper-api.md</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[From URL Lists to Structured Data: Designing a Multi-Site Scraping Workflow]]></title><description><![CDATA[A web scraping project often starts with a list of URLs. The real engineering work begins after that list is created. A production workflow must decide…]]></description><link>https://thordata-notes.hashnode.dev/from-url-lists-to-structured-data-designing-a-multi-site-scraping-workflow</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/from-url-lists-to-structured-data-designing-a-multi-site-scraping-workflow</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Mon, 19 Oct 2026 05:49:02 GMT</pubDate><content:encoded><![CDATA[<p>A web scraping project often starts with a list of URLs.</p>
<p>The real engineering work begins after that list is created.</p>
<p>A production workflow must decide which fields to collect, how to submit requests in batches, how to handle failures, how to validate records, and how to deliver structured output to another system.</p>
<p>This article outlines a practical workflow for multi-site public data extraction.</p>
<h2>1. Define the output schema first</h2>
<p>Do not start by writing selectors.</p>
<p>Start by defining the fields the business actually needs.</p>
<p>For an e-commerce product workflow, the schema might include:</p>
<ul>
<li><p>Product title</p>
</li>
<li><p>Product ID</p>
</li>
<li><p>Brand</p>
</li>
<li><p>Description</p>
</li>
<li><p>Price</p>
</li>
<li><p>Currency</p>
</li>
<li><p>Availability</p>
</li>
<li><p>Rating</p>
</li>
<li><p>Review count</p>
</li>
<li><p>Product URL</p>
</li>
<li><p>Collection timestamp</p>
</li>
</ul>
<p>For a social media workflow, the schema may include:</p>
<ul>
<li><p>Post ID</p>
</li>
<li><p>Author</p>
</li>
<li><p>Caption</p>
</li>
<li><p>Publication time</p>
</li>
<li><p>Engagement metrics</p>
</li>
<li><p>Media URL</p>
</li>
<li><p>Source URL</p>
</li>
</ul>
<p>The schema should also define which fields are required, which may be null, and how values should be normalized.</p>
<h2>2. Choose a source-specific scraper</h2>
<p>Different websites expose different page structures and data models.</p>
<p>A source-specific scraper can handle the expected parameters and return fields that are already structured for that platform.</p>
<p>Thordata Web Scraper API provides preconfigured templates for Amazon, YouTube, Facebook, Zillow, Instagram, and more than 120 public domains.</p>
<p>The API and dashboard workflow support both code-based and no-code scraping.</p>
<h2>3. Submit URLs in batches</h2>
<p>Batch submission is useful when a workflow has many independent URLs.</p>
<p>A batch should have:</p>
<ul>
<li><p>A clear task identifier</p>
</li>
<li><p>A defined source scraper</p>
</li>
<li><p>A consistent parameter set</p>
</li>
<li><p>A predictable file naming rule</p>
</li>
<li><p>A maximum request scope</p>
</li>
<li><p>An error-handling policy</p>
</li>
</ul>
<p>The Web Scraper API documentation describes adding multiple URLs through the API Request Builder. URLs can be entered in batches, separated by line breaks or commas.</p>
<p>The product page also lists bulk request handling for up to 10,000 URLs.</p>
<h2>4. Separate collection from validation</h2>
<p>A successful request does not automatically mean a valid record.</p>
<p>After receiving the result, validate:</p>
<ol>
<li><p>The source URL matches the requested URL.</p>
</li>
<li><p>Required fields are present.</p>
</li>
<li><p>Numeric fields use the expected format.</p>
</li>
<li><p>Currency and units are correctly identified.</p>
</li>
<li><p>The record belongs to the expected page or item.</p>
</li>
<li><p>Duplicate records are handled consistently.</p>
</li>
<li><p>The response is not an error or incomplete page.</p>
</li>
</ol>
<p>Invalid records should be marked for review instead of being silently added to the final dataset.</p>
<h2>5. Keep errors with the task</h2>
<p>A useful pipeline does not discard failures.</p>
<p>Store:</p>
<ul>
<li><p>URL</p>
</li>
<li><p>Task ID</p>
</li>
<li><p>Error category</p>
</li>
<li><p>Retry count</p>
</li>
<li><p>Response timestamp</p>
</li>
<li><p>Parser or scraper version</p>
</li>
<li><p>Final status</p>
</li>
</ul>
<p>This makes it easier to distinguish a temporary network issue from a missing field, unsupported page type, or changed source structure.</p>
<h2>6. Deliver data in a usable format</h2>
<p>Different teams need different outputs.</p>
<p>JSON is useful for APIs and application workflows. CSV is convenient for spreadsheets and simple data analysis. XLSX can be useful for business users who need a ready-to-review file.</p>
<p>Thordata’s Web Scraper API supports structured data output in JSON, CSV, and XLSX. The product also describes scheduled collection and automatic delivery to a designated cloud storage service.</p>
<p>The delivery step should preserve the task ID and collection timestamp so downstream users can trace each file back to its source run.</p>
<h2>7. Monitor quality over time</h2>
<p>Track more than request volume.</p>
<p>Useful workflow metrics include:</p>
<ul>
<li><p>Required-field completion rate</p>
</li>
<li><p>Valid records per task</p>
</li>
<li><p>Duplicate rate</p>
</li>
<li><p>Error rate by source</p>
</li>
<li><p>Average processing time</p>
</li>
<li><p>Retry rate</p>
</li>
<li><p>Cost per valid record</p>
</li>
<li><p>Data freshness</p>
</li>
</ul>
<p>These metrics reveal whether the workflow is producing usable data or simply generating more responses.</p>
<h2>8. Start with a controlled pilot</h2>
<p>Before scaling to every source, test a small group of URLs.</p>
<p>Review:</p>
<ul>
<li><p>Schema completeness</p>
</li>
<li><p>Field consistency</p>
</li>
<li><p>Null values</p>
</li>
<li><p>Duplicate behavior</p>
</li>
<li><p>Pagination</p>
</li>
<li><p>Error handling</p>
</li>
<li><p>Output format</p>
</li>
<li><p>Delivery process</p>
</li>
</ul>
<p>Once the pilot is stable, increase batch size and add scheduled collection.</p>
<p>The current Web Scraper API pricing page lists a 7-day free trial with 5,000 credits. Paid plans start at \(1.00 per 1,000 credits, with the listed high-volume plan priced at \)0.50 per 1,000 credits.</p>
<p>Use the service only for lawful, authorized public-data workflows and respect applicable website terms and restrictions.</p>
<p>Official documentation:</p>
<p><a href="https://doc.thordata.com/doc/scraping/web-scraper-api.md">https://doc.thordata.com/doc/scraping/web-scraper-api.md</a></p>
<p>API Request Builder:</p>
<p><a href="https://doc.thordata.com/doc/scraping/web-scraper-api/api-request-builder.md">https://doc.thordata.com/doc/scraping/web-scraper-api/api-request-builder.md</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[Building a Website Change Monitoring Workflow with Field-Level Diffs]]></title><description><![CDATA[Website change monitoring is often implemented as a simple loop: The difficult part is deciding whether the difference is meaningful. A rotating banner…]]></description><link>https://thordata-notes.hashnode.dev/building-a-website-change-monitoring-workflow-with-field-level-diffs</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/building-a-website-change-monitoring-workflow-with-field-level-diffs</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Fri, 16 Oct 2026 01:33:35 GMT</pubDate><content:encoded><![CDATA[<p>Website change monitoring is often implemented as a simple loop:</p>
<ol>
<li>Fetch a page.</li>
<li>Compare it with the previous version.</li>
<li>Send an alert.</li>
</ol>
<p>The difficult part is deciding whether the difference is meaningful.</p>
<p>A rotating banner, timestamp, cookie value, tracking parameter, or regional template can trigger an alert even when the business information has not changed.</p>
<p>A reliable monitoring workflow needs normalization, structured observations, and clear alert states.</p>
<h2>Define the change you care about</h2>
<p>Start by describing the alert in business terms.</p>
<p>Examples include:</p>
<ul>
<li>A product price changed.</li>
<li>A product became unavailable.</li>
<li>A public policy page was updated.</li>
<li>A competitor added or removed a plan.</li>
<li>A regional landing page changed its offer.</li>
<li>A public status page changed its incident message.</li>
</ul>
<p>Each use case needs a different comparison method.</p>
<p>A full-page diff may work for a legal document. A field-level diff is usually better for price, availability, plan, or status monitoring.</p>
<h2>Store a normalized observation</h2>
<p>Keep both the raw source reference and a cleaned representation.</p>
<p>A monitoring record may include:</p>
<ul>
<li>URL</li>
<li>Requested country</li>
<li>Requested city</li>
<li>Page title</li>
<li>Important extracted fields</li>
<li>Content hash</li>
<li>Collection timestamp</li>
<li>Parser version</li>
<li>Validation status</li>
</ul>
<p>The requested location matters because a page can be unchanged in one market and different in another.</p>
<p>Without location and timestamp fields, the monitor may interpret a routing difference as a website change.</p>
<h2>Normalize dynamic content</h2>
<p>Before comparing page versions, remove elements that change frequently but are not part of the alert:</p>
<ul>
<li>Current timestamps</li>
<li>Rotating promotional modules</li>
<li>Personalized recommendations</li>
<li>Tracking parameters</li>
<li>Random IDs</li>
<li>Unrelated navigation elements</li>
<li>Session-specific tokens</li>
</ul>
<p>Do not remove too much. A changing eligibility message, shipping note, or plan limitation may be the exact information the business needs to monitor.</p>
<p>Normalization rules should be versioned so that an alert can be reproduced later.</p>
<h2>Compare stable fields</h2>
<p>For a pricing page, compare:</p>
<ul>
<li>Plan name</li>
<li>Price</li>
<li>Currency</li>
<li>Billing period</li>
<li>Availability</li>
<li>Promotion status</li>
</ul>
<p>For a policy page, compare cleaned body text and generate a line-level diff.</p>
<p>For a status page, compare:</p>
<ul>
<li>Incident title</li>
<li>Incident status</li>
<li>Affected service</li>
<li>Latest update</li>
<li>Resolution message</li>
</ul>
<p>Field-level comparison makes alerts easier to explain than a raw HTML hash alone.</p>
<h2>Use proxies as controlled variables</h2>
<p>A proxy can help when the page is location-sensitive or when the team needs to compare responses from different markets.</p>
<p>Configure the monitoring job deliberately:</p>
<ol>
<li>Select the required country or city.</li>
<li>Use rotating sessions for independent checks.</li>
<li>Use a short sticky session when related requests need continuity.</li>
<li>Record the requested location and session mode.</li>
<li>Keep the URL, parser, and normalization rules consistent across routes.</li>
</ol>
<p>Thordata documents country, city, state, continent, and ASN targeting, as well as rotating and sticky residential sessions over HTTP/HTTPS.</p>
<p>The goal is not to generate more traffic. The goal is to make each observation representative of the market being monitored.</p>
<h2>Add alert confidence states</h2>
<p>A useful monitoring pipeline should distinguish between different outcomes:</p>
<ul>
<li><code>unchanged</code>: Important fields match the previous valid observation.</li>
<li><code>changed</code>: A meaningful field changed and validation passed.</li>
<li><code>needs_review</code>: The response is incomplete or unexpected.</li>
<li><code>unavailable</code>: The page or required field could not be observed.</li>
<li><code>parse_failed</code>: The page loaded but extraction failed.</li>
</ul>
<p>Only confirmed <code>changed</code> records should trigger urgent alerts.</p>
<p>Route <code>needs_review</code> and <code>parse_failed</code> to a lower-priority queue. This reduces alert fatigue and gives engineers a clearer maintenance signal.</p>
<h2>Use bounded retries</h2>
<p>A timeout may justify one or two retries with backoff.</p>
<p>A wrong-region response should not be treated as a network failure. A missing price may indicate a parser change, an unavailable offer, or a page that requires a different location setting.</p>
<p>Record the final reason even when a later retry succeeds. Otherwise, the monitoring dashboard may look healthy while hiding an unstable collection path.</p>
<h2>Test with a replay set</h2>
<p>Keep a small set of representative pages and replay them after changing:</p>
<ul>
<li>Proxy route</li>
<li>Parser</li>
<li>Normalization rules</li>
<li>Alert thresholds</li>
<li>Session configuration</li>
</ul>
<p>Include at least one page for each alert type. Review the diffs manually before enabling high-priority notifications.</p>
<p>This replay set is especially useful after a website redesign.</p>
<h2>Respect the monitoring boundary</h2>
<p>Monitor only public pages and data that you are authorized to access. Follow applicable terms, reasonable request rates, privacy requirements, and copyright rules.</p>
<p>Thordata's Residential Proxies pricing page currently lists plans from $0.65/GB. The product page lists rotating and sticky sessions, location targeting, and coverage across more than 190 locations.</p>
<p>Official related article:</p>
<p><a href="https://www.thordata.com/blog/proxies/website-change-monitoring-with-proxies-how-to-reduce-false-alerts">https://www.thordata.com/blog/proxies/website-change-monitoring-with-proxies-how-to-reduce-false-alerts</a></p>
<p>Residential proxy documentation:</p>
<p><a href="https://doc.thordata.com/doc/proxies/residential-proxies/making-request.md">https://doc.thordata.com/doc/proxies/residential-proxies/making-request.md</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[Regional Price Monitoring: A Data Quality Framework for Public E-commerce Checks]]></title><description><![CDATA[Regional price monitoring is often described as a proxy problem. In practice, it is first a data-quality problem. A request can succeed technically and…]]></description><link>https://thordata-notes.hashnode.dev/regional-price-monitoring-a-data-quality-framework-for-public-e-commerce-checks</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/regional-price-monitoring-a-data-quality-framework-for-public-e-commerce-checks</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Sat, 10 Oct 2026 02:02:32 GMT</pubDate><content:encoded><![CDATA[<p>Regional price monitoring is often described as a proxy problem. In practice, it is first a data-quality problem.</p>
<p>A request can succeed technically and still produce an unreliable business result if the collection context is missing. A price observed from the wrong market, currency, session, or page state can create a false comparison.</p>
<h2>Record the context of every observation</h2>
<p>A useful price record should include at least:</p>
<ul>
<li><p>Product ID or SKU</p>
</li>
<li><p>Product URL</p>
</li>
<li><p>Target country and city when relevant</p>
</li>
<li><p>Collection timestamp</p>
</li>
<li><p>Currency</p>
</li>
<li><p>Observed price</p>
</li>
<li><p>Availability status</p>
</li>
<li><p>Session identifier</p>
</li>
<li><p>Parser version</p>
</li>
<li><p>Response or page-state indicators</p>
</li>
</ul>
<p>This context makes it possible to explain why two observations differ. It also helps separate a real market change from a localization issue, an unavailable product, or a parser failure.</p>
<h2>Use location deliberately</h2>
<p>A regional check should define its target before the request is sent. Depending on the workflow, this may include a country, state, city, continent, or ASN.</p>
<p>Thordata Residential Proxies support location targeting and rotating or sticky sessions. The documentation also describes session durations from 1 to 90 minutes. These controls can help teams reproduce a localized request pattern during authorized public-data collection.</p>
<p>The important point is consistency. If one run targets a city and another uses a random location, their prices should not be compared as if they represented the same market.</p>
<h2>Match the proxy type to the workload</h2>
<p>Different collection jobs have different infrastructure requirements:</p>
<ul>
<li><p>Residential proxies are suited to complex public-web data collection and granular location targeting.</p>
</li>
<li><p>Static ISP proxies are useful when a workflow needs a stable identity and a longer session.</p>
</li>
<li><p>Datacenter proxies are often a practical choice for high-volume, routine requests where speed and cost are the primary considerations.</p>
</li>
<li><p>Mobile proxies are designed for mobile-network and mobile-content workflows.</p>
</li>
</ul>
<p>The correct choice depends on the target, the required continuity, the geographic scope, and the expected request volume.</p>
<h2>Validate before storing a price</h2>
<p>A parser should not treat every successful HTTP response as a valid observation. Before storing a price, check:</p>
<ol>
<li><p>The response belongs to the expected product.</p>
</li>
<li><p>The page is in the intended language and currency.</p>
</li>
<li><p>The product is available or clearly marked as unavailable.</p>
</li>
<li><p>The required price field is present.</p>
</li>
<li><p>The response is not an error, consent, login, or unexpected page.</p>
</li>
<li><p>The target location matches the requested location where location matters.</p>
</li>
</ol>
<p>If one of these checks fails, store the event as a validation failure instead of silently writing an incorrect price.</p>
<h2>Measure data quality over time</h2>
<p>Useful monitoring metrics include:</p>
<ul>
<li><p>Price-field completion rate</p>
</li>
<li><p>Unexpected-page rate</p>
</li>
<li><p>Availability-state accuracy</p>
</li>
<li><p>Requested versus observed location</p>
</li>
<li><p>Currency mismatch rate</p>
</li>
<li><p>Retry rate per valid record</p>
</li>
<li><p>Cost per validated observation</p>
</li>
<li><p>Time since the last successful observation</p>
</li>
</ul>
<p>These metrics are more useful than request count alone. A large number of requests does not guarantee a reliable price dataset.</p>
<p>The current Thordata Residential Proxies pricing page lists plans from $0.65/GB. Product and pricing details should always be verified before procurement because plan terms can change.</p>
<p>Read the technical documentation:</p>
<p><a href="https://doc.thordata.com/doc/proxies/residential-proxies/making-request">https://doc.thordata.com/doc/proxies/residential-proxies/making-request</a></p>
<p>Read the related Thordata analysis:</p>
<p><a href="https://www.thordata.com/blog/serp-api/one-product-forty-prices-a-methods-paper-on-measuring-geo-pricing-honestly">https://www.thordata.com/blog/serp-api/one-product-forty-prices-a-methods-paper-on-measuring-geo-pricing-honestly</a></p>
<p>Explore Thordata:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
<p>Use proxy infrastructure only for lawful, authorized workflows involving permitted public data.</p>
]]></content:encoded></item><item><title><![CDATA[Designing Session Budgets for Reliable Public-Web Data Collection]]></title><description><![CDATA[Proxy rotation is often treated as a simple rule: change the IP as frequently as possible. That approach can work for some workloads, but it is not a…]]></description><link>https://thordata-notes.hashnode.dev/designing-session-budgets-for-reliable-public-web-data-collection</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/designing-session-budgets-for-reliable-public-web-data-collection</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Fri, 09 Oct 2026 07:32:16 GMT</pubDate><content:encoded><![CDATA[<p>Proxy rotation is often treated as a simple rule: change the IP as frequently as possible.</p>
<p>That approach can work for some workloads, but it is not a complete data-collection strategy. The right session model depends on whether requests are independent, whether the workflow needs continuity, and whether location affects the response.</p>
<h2>Match the session model to the workflow</h2>
<h3>Independent public pages</h3>
<p>Product pages, public listings, and independent search-result URLs can usually be collected with rotating sessions. Each request can be processed separately, which makes rotation simple and easier to scale.</p>
<h3>Short stateful sequences</h3>
<p>Pagination, short navigation flows, and localized checks may need a sticky session. Keeping the same IP for a limited period can make the response easier to reproduce and the workflow easier to debug.</p>
<h3>Long-running workflows</h3>
<p>Long sessions should have a clearly documented purpose, an expiry condition, and an approved scope. A proxy does not remove the need for rate limits, validation, or compliance review.</p>
<h2>Define a session budget</h2>
<p>Before a collection job starts, define:</p>
<ul>
<li><p>Maximum requests per session</p>
</li>
<li><p>Maximum session duration</p>
</li>
<li><p>Target country, state, city, or ASN</p>
</li>
<li><p>Condition for creating a new session</p>
</li>
<li><p>Condition for stopping the entire job</p>
</li>
</ul>
<p>For example, a collector might use one sticky session for ten pagination requests or fifteen minutes, whichever comes first. The exact values depend on the workflow, but the principle is the same: a session should have a reason and a limit.</p>
<h2>Separate retries from rotation</h2>
<p>Rotation is not a retry policy.</p>
<p>A timeout may justify a limited retry with backoff. A missing field may indicate a parser problem rather than a network problem. A response from the wrong region usually means the target configuration needs to be checked before trying again.</p>
<p>Changing the parser, request rate, headers, and proxy location at the same time makes failures difficult to diagnose. Change one variable at a time and record the result.</p>
<h2>Measure data quality, not only connectivity</h2>
<p>Useful metrics include:</p>
<ul>
<li><p>Required-field completion</p>
</li>
<li><p>Empty-page and unexpected-page rate</p>
</li>
<li><p>Timeout rate</p>
</li>
<li><p>Retries per valid record</p>
</li>
<li><p>Median and p95 response time</p>
</li>
<li><p>Requested versus observed location</p>
</li>
<li><p>Cost per valid record</p>
</li>
</ul>
<p>A lower error rate is useful, but complete and correctly localized records are the real output of the pipeline.</p>
<p>Thordata Residential Proxies documentation covers rotating sessions, sticky sessions, HTTP/HTTPS requests, and location parameters such as country, state, city, and ASN:</p>
<p><a href="https://doc.thordata.com/doc/proxies/residential-proxies/making-request">https://doc.thordata.com/doc/proxies/residential-proxies/making-request</a></p>
<p>Use proxies only for lawful, authorized workflows involving permitted public data. Do not collect nonpublic information or use automated access in ways that violate applicable law or a website's terms.</p>
<p>Learn more about Thordata's proxy infrastructure:</p>
<p><a href="https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode">https://www.thordata.com/?ls=Hashnode&amp;lk=Hashnode</a></p>
]]></content:encoded></item><item><title><![CDATA[Put Your Proxy Pool's Diversity in CI: a Drift Gate]]></title><description><![CDATA[A one-off audit of "how diverse is my proxy pool today" is a nice exercise. It tells you what you already suspected, and you close the terminal. Six weeks…]]></description><link>https://thordata-notes.hashnode.dev/put-your-proxy-pool-s-diversity-in-ci-a-drift-gate</link><guid isPermaLink="true">https://thordata-notes.hashnode.dev/put-your-proxy-pool-s-diversity-in-ci-a-drift-gate</guid><dc:creator><![CDATA[thordata]]></dc:creator><pubDate>Wed, 23 Sep 2026 09:19:05 GMT</pubDate><content:encoded><![CDATA[<h2>Why this exists</h2>
<p>A one-off audit of "how diverse is my proxy pool today" is a nice exercise. It tells you what you already suspected, and you close the terminal. Six weeks later the provider has quietly re-sourced a chunk of their allocation, your diversity numbers have collapsed by half, and nothing told you until a specific downstream target started returning 403 at a rate you had never seen. The failure is not invisible; it is only invisible on the days you do not run the audit.</p>
<p>The fix is boring. You put the number in CI. You record a floor, and you fail the build when it drifts. Below is the smallest thing I could write that does that with the Python standard library and no external service calls at the base layer.</p>
<h2>The shape of the check</h2>
<p>Every night (or every build, depending on how aggressive your provider is about throttling sampling traffic) you capture a fixed-size sample of exit IPs from your pool into a JSON file. That is the sampling side, and it is provider-specific; you know your gateway's session model, and the code here does not need to. What the code does care about is the second step: given the sample, compute a diversity metric, compare it to the previous N samples, and fail loudly when the current one drifts.</p>
<p>I keep the base check at the /24 layer for CI because it costs nothing to compute and has no rate limits. If your /24 diversity is healthy, you can optionally upgrade to a BGP announced-prefix check with an external lookup; that costs a call per unique /24 and is best done weekly, not on every build.</p>
<h2>The gate</h2>
<pre><code class="language-python"># diversity_gate.py - stdlib only.
# Reads a captured sample, computes /24 diversity, fails the build on drift.
import ipaddress, json, sys
from collections import Counter
from pathlib import Path

HISTORY_FILE = Path("diversity_history.json")
MIN_SAMPLE = 200            # do not trust the number below this
MIN_RATIO = 0.05            # at least 5% of sessions should occupy unique /24s
MAX_TOP24_SHARE = 0.30      # no single /24 may cover more than 30% of sessions
DRIFT_TOLERANCE = 0.20      # ratio may not fall more than 20% below trailing mean


def slash24(ip: str) -&gt; str:
    return str(ipaddress.ip_network(ip + "/24", strict=False))


def score(sample: list[str]) -&gt; dict:
    n = len(sample)
    by24 = Counter(slash24(ip) for ip in sample if _is_v4(ip))
    if not by24:
        return {"sample": n, "ratio": 0.0, "top_share": 1.0, "unique_24": 0}
    shares = [c / n for _, c in by24.most_common()]
    return {
        "sample": n,
        "unique_24": len(by24),
        "ratio": len(by24) / max(n, 1),
        "top_share": shares[0],
    }


def _is_v4(ip: str) -&gt; bool:
    try:
        return isinstance(ipaddress.ip_address(ip), ipaddress.IPv4Address)
    except ValueError:
        return False


def load_history() -&gt; list[dict]:
    if not HISTORY_FILE.exists():
        return []
    return json.loads(HISTORY_FILE.read_text())


def append_history(entry: dict) -&gt; None:
        hist = load_history()
        hist.append(entry)
        HISTORY_FILE.write_text(json.dumps(hist[-90:], indent=2))   # keep 90 days


def gate(today: dict, hist: list[dict]) -&gt; list[str]:
    errors = []
    if today["sample"] &lt; MIN_SAMPLE:
        errors.append(f"sample size {today['sample']} below floor {MIN_SAMPLE}")
        return errors
    if today["ratio"] &lt; MIN_RATIO:
        errors.append(
            f"/24-to-session ratio {today['ratio']:.3f} below {MIN_RATIO}")
    if today["top_share"] &gt; MAX_TOP24_SHARE:
        errors.append(
            f"top /24 covers {today['top_share']:.2%} of sessions (limit "
            f"{MAX_TOP24_SHARE:.0%})")
    recent = [h["ratio"] for h in hist[-10:]]
    if len(recent) &gt;= 3:
        baseline = sum(recent) / len(recent)
        if baseline &gt; 0 and today["ratio"] &lt; baseline * (1 - DRIFT_TOLERANCE):
            errors.append(
                f"ratio {today['ratio']:.3f} drifted &gt;20% below trailing mean "
                f"{baseline:.3f}")
    return errors


if __name__ == "__main__":
    sample = json.loads(Path(sys.argv[1]).read_text())
    today = score(sample)
    errors = gate(today, load_history())
    append_history(today)
    print(json.dumps(today, indent=2))
    for e in errors:
        print("FAIL:", e, file=sys.stderr)
    sys.exit(1 if errors else 0)
</code></pre>
<h2>The nightly sampler</h2>
<p>The other half is capturing the sample file. This one <em>is</em> provider-specific, so keep it short. Fire N single-shot requests through your gateway (or N sessions through a sticky gateway with a rotation window), and log the IP each returned from an echo endpoint. Save the list. A sketch:</p>
<pre><code class="language-python"># capture_sample.py - provider-neutral sketch.
import json
from concurrent.futures import ThreadPoolExecutor
import requests    # the one non-stdlib dependency

SESSION_URL = "http://USER:PASS@gw.example:PORT"     # provider rotates for you,
                                                     # or append ?sessionid={i}
ECHO = "https://icanhazip.com/"

def one(i: int):
    try:
        r = requests.get(ECHO, proxies={"http": SESSION_URL, "https": SESSION_URL},
                          timeout=15)
        return r.text.strip()
    except requests.RequestException:
        return None

def main(n: int = 500) -&gt; None:
    with ThreadPoolExecutor(max_workers=20) as ex:
        ips = [x for x in ex.map(one, range(n)) if x]
    json.dump(ips, open("sample_tonight.json", "w"))
    print(f"captured {len(ips)} of {n}")

if __name__ == "__main__":
    main()
</code></pre>
<p>Wire both into your scheduler of choice: capture, then gate. In CI that looks like two lines in a cron job or a GitHub Action step:</p>
<pre><code class="language-plaintext">python capture_sample.py &amp;&amp; python diversity_gate.py sample_tonight.json
</code></pre>
<h2>Thresholds are the hard part</h2>
<p><code>MIN_RATIO</code>, <code>MAX_TOP24_SHARE</code>, <code>DRIFT_TOLERANCE</code> - the code does not care about their values, your team does. What I have above is deliberately conservative: it flags real collapses, not wobble. Do not tune them before you have twenty or thirty samples, because on day one you cannot tell drift from noise. Once you have history, plot the ratio column against time and pick numbers that would have caught the incidents you remember and left you alone otherwise.</p>
<p>Two honest limits. First, the /24 layer is a lower bound. If your provider announces supernet routes (which large carriers and some hosting providers do), two different /24s can still collapse into one announced prefix and this gate will not see it. That is what the optional weekly announced-prefix pass is for; the code is not much different, you just add a lookup that returns the routed prefix instead of slicing the first three octets. Second, the drift test uses a trailing mean of ten samples, which is smooth enough for a daily job but will miss a two-day dip that recovers. That trade-off is fine for a diversity regression, because diversity regressions tend to be structural, but if your use-case is more sensitive, use a trailing median or a shorter window.</p>
<p>If you want somewhere to test the sampler without wiring a full provider integration first, <a href="https://www.thordata.com/products/residential-proxies">Thordata's residential proxy line</a> has a straightforward gateway model you can point this at. The gate itself is provider-neutral and works on any sample you can produce.</p>
<p>Disclosure: I work with Thordata on content.</p>
]]></content:encoded></item></channel></rss>