Observability

Monitor your data pipeline in real time. Every event, error, and schema change is tracked and surfaced in the dashboard and API.

Dashboard Overview

The Observability page in the dashboard provides:

• Live histogram — minute-by-minute event volume for the last hour
• Hourly trend — 24-hour view with per-hour aggregates
• Stream filter — drill into a specific stream
• Error breakdown — errors by type (validation, processing, rate limit, schema rejection)
• Recent requests — individual request details with payloads

Navigate to Dashboard → Observability or click the chart icon next to any stream.

---

Real-Time Stats

Account-Level Stats

curl https://enrich.sh/stats \
  -H "Authorization: Bearer sk_live_your_key"

Response:

{
  "status": "ok",
  "customer_id": "cust_abc123",
  "hour": "2026-02-07T09",
  "requests": 1523,
  "events": 45690,
  "errors": 2,
  "histogram": [
    { "minute": "2026-02-07T09:30", "events": 1200, "requests": 40, "errors": 0 }
  ],
  "hourly": [
    { "hour": "2026-02-07T08", "events": 52000, "requests": 1700, "errors": 5 }
  ]
}

Field	Description
status	ok when data is flowing, idle when quiet
histogram	Minute-by-minute breakdown (last hour)
hourly	Hour-by-hour aggregates (last 24 hours)

Per-Stream Stats

curl https://enrich.sh/stats/events \
  -H "Authorization: Bearer sk_live_your_key"

Returns the same histogram/hourly structure scoped to one stream, plus recent errors for that stream.

---

Error Tracking

Every failed request is logged with the full error context — type, message, endpoint, and request payload.

API Errors

curl "https://enrich.sh/errors?limit=20" \
  -H "Authorization: Bearer sk_live_your_key"

Response:

{
  "errors_24h": 12,
  "by_type": {
    "validation": 8,
    "schema_rejection": 2,
    "rate_limit": 1,
    "processing": 1
  },
  "recent": [
    {
      "type": "validation",
      "message": "stream_id is required",
      "request_path": "/ingest",
      "request_body": "{ ... }",
      "created_at": "2026-02-07T09:45:00.000Z"
    }
  ]
}

Error Types

Type	Description	Common Cause
validation	Request failed validation	Missing stream_id, empty data, bad format
schema_rejection	Event didn't match strict schema	Extra/missing fields, type mismatch
rate_limit	Exceeded request or event quota	Burst traffic, plan limit reached
processing	Internal processing failure	Rare — contact support if persistent
flush	Error writing Parquet to storage	Storage issue — automatically retried

---

Schema Change Detection

When a stream uses evolve mode, Enrich.sh automatically detects and logs schema changes. This catches upstream data changes before they break your queries.

Schema Events API

curl "https://enrich.sh/schema-events?stream_id=erp_events&limit=50" \
  -H "Authorization: Bearer sk_live_your_key"

Response:

{
  "events": [
    {
      "event_type": "new_field",
      "field_name": "discount_code",
      "new_value": "string",
      "created_at": "2026-02-15T10:30:00Z"
    },
    {
      "event_type": "type_change",
      "field_name": "amount",
      "old_value": "int64",
      "new_value": "string",
      "created_at": "2026-02-15T11:00:00Z"
    }
  ]
}

Change Types

Event	Description	Severity
new_field	A field appeared that isn't in the schema	ℹ️ Info
missing_field	A previously-present field stopped appearing	⚠️ Warning
type_change	A field's data type changed	🔴 Error

TIP

Review schema events regularly when integrating SaaS APIs or ERP systems — these sources often change their payloads without notice.

---

Dead Letter Queue

When strict mode rejects an event, it's routed to the stream's Dead Letter Queue instead of being dropped. See the full DLQ documentation for details.

Quick check:

curl "https://enrich.sh/streams/transactions/dlq?days=7" \
  -H "Authorization: Bearer sk_live_your_key"

DLQ events include the original payload plus rejection metadata (_rejected_at, _reason, _field), so you can fix and re-ingest them.

---

What to Monitor

Signal	Where to Look	Action
Event volume drop	Stats → histogram	Check upstream source
Error spike	Errors → by_type	Review recent errors for pattern
Schema drift	Schema Events	Verify upstream API change
DLQ growth	DLQ count	Fix rejected events, adjust schema
Latency increase	Stats → requests/events ratio	Check payload size, reduce batch volume