Skip to main content

Building Observability into a Web Data Pipeline

Updated 4 min readMarkdown

A scraping pipeline can show healthy request logs while producing incomplete or incorrect data.

Responses may contain an access page, a CAPTCHA challenge, an empty JavaScript shell, a changed page structure, or a partial record.

This is why web data observability must continue after the HTTP response is received.

Separate transport success from data success

A transport-success metric answers:

Did the request receive a response?

A data-success metric answers:

Did the response produce a valid record?

These are different measurements.

A useful pipeline should track both.

For each task, record:

  • Request status
  • Response type
  • Source URL
  • Parser status
  • Required-field completion
  • Validation result
  • Collection timestamp
  • Task ID
  • Scraper or parser version

Monitor the right metrics

Important metrics include:

Valid-record rate

How many requests produced records that passed validation?

Field completeness

How often are required fields present and usable?

Challenge-page rate

How often did the response contain an access page, CAPTCHA, or unexpected page type?

Duplicate rate

Are repeated records entering the dataset?

Schema-drift rate

Are fields missing, renamed, or changing type compared with previous runs?

Cost per usable record

How much did it cost to produce one validated record?

This is usually more useful than cost per request.

Add response validation

A response should not be marked successful only because it returned HTTP 200.

Validate:

  1. The source URL matches the requested URL.
  2. The page belongs to the expected source and page type.
  3. Required fields are present.
  4. Values have the expected format.
  5. The record is not duplicated.
  6. The response is not an access, login, consent, or challenge page.
  7. The schema matches the current version.

Invalid records should be stored with a clear failure state instead of being silently discarded.

Detect schema drift

Websites change.

A field may disappear, change from a number to a string, move to a nested object, or become unavailable for only some page types.

Keep a schema contract for each scraper. Compare incoming records with the expected contract and alert when:

  • A required field disappears.
  • A field changes type.
  • Null values increase sharply.
  • A new field appears unexpectedly.
  • The number of extracted records drops.
  • A parser returns empty values for a previously stable field.

Schema alerts should be separate from ordinary network alerts because they usually require parser or source review.

Classify failures

Use separate states for different problems:

  • transport_failed
  • access_challenge
  • render_failed
  • parse_failed
  • validation_failed
  • schema_changed
  • duplicate_record
  • valid_record

This makes recovery more precise.

A timeout may justify a limited retry. A parser failure needs investigation. A schema change may require a code update. Retrying all of them in the same way increases cost without improving data quality.

Monitor scheduled tasks

Thordata Web Scraper API supports scheduled tasks using visual settings or five-field Cron expressions.

A scheduled workflow should record:

  • Expected run time
  • Actual start time
  • Completion time
  • Number of inputs
  • Number of valid records
  • Number of failed records
  • Delivery status

A task that runs on schedule but produces zero valid records should be treated as a pipeline failure.

Build an operational dashboard

A practical dashboard can show:

  • Valid-record rate over time
  • Required-field completeness
  • Errors by source
  • Schema changes
  • Duplicate rate
  • Average latency
  • Cost per valid record
  • Last successful task
  • Last known parser version

These metrics help teams identify whether a problem belongs to access, parsing, validation, source changes, or downstream delivery.

The current Web Scraper API pricing page lists a 7-day free trial with 5,000 credits. Paid plans start at $1.00 per 1,000 results, with high-volume pricing down to $0.50 per 1,000 results.

Use Web Scraper API only for lawful, authorized public-data workflows and respect applicable website terms and restrictions.

Web Scraper API documentation:

https://doc.thordata.com/doc/scraping/web-scraper-api.md

Scheduler documentation:

https://doc.thordata.com/doc/scraping/web-scraper-api/scheduler.md

Explore Thordata:

https://www.thordata.com/?ls=Hashnode&lk=Hashnode

No comments yet. Be the first to comment.