Trace propagation, metrics, dashboards, and alerting.
Observability
OpenTelemetry export
GetRatchet emits OpenTelemetry traces and bounded-cardinality metrics. Set OTEL_EXPORTER_OTLP_ENDPOINT to an OTLP/HTTP collector base URL to enable export from the Next.js API. Set OTEL_EXPORTER_OTLP_HEADERS if the collector requires authentication. Store both as server-side environment variables; never put collector credentials in NEXT_PUBLIC_* values. The API does not require a collector to serve requests.
The TypeScript SDK injects W3C trace context into API requests. A durable enqueue stores traceparent and tracestate on the queued job; the claim response carries them to the worker, which starts an attempt span under the producer trace. Worker applications can opt in to export by calling startWorkerTelemetry() before creating the Ratchet client and calling await telemetry.shutdown() on graceful exit. The worker process needs the same OTLP environment variables and network access to the collector. Applications with their own OpenTelemetry provider can omit this helper and keep their provider.
import { createRatchet, startWorkerTelemetry } from '@getratchet/sdk';
const telemetry = startWorkerTelemetry('my-agent-worker');
const ratchet = createRatchet({ baseUrl: process.env.GETRATCHET_URL!, apiKey: process.env.GETRATCHET_API_KEY! });
// Register handlers and run the worker as usual.
// On graceful shutdown: await telemetry.shutdown();The Python SDK does not provide startWorkerTelemetry(). Configure your own OpenTelemetry provider in the Python worker if it needs OTLP export.
API metrics include accepted runs and steps, attempt outcomes and duration, retries and scheduled delay, lease expirations, circuit transitions, exhausted steps, recovery item outcomes, queue depth and oldest age, time since the most recent worker heartbeat, and request counts and handler duration by static route, method, and status class. Thrown route errors count as 5xx. Tool and service names are hashed into 128 buckets; no raw names, inputs, outputs, error text, account IDs, API keys, or unbounded URLs are metric labels or span attributes. The queue and heartbeat gauges aggregate across organizations and must not be used to inspect an individual tenant. The original OTLP request and worker spans are exported only when a collector is configured; Sentry has a separate opt-in export pipeline. The Grafana dashboard template assumes an OTLP collector forwards metrics to a Prometheus-compatible data source. Select that data source after import.
The alert rule guide gives PromQL, evaluation windows, contact-point setup, and an end-to-end retry verification procedure. Use the authenticated /api/v1/health endpoint for tenant-specific queue age and worker heartbeat; a developer-owned worker may be intentionally offline. Dashboard panels remain empty if no OTLP destination is configured or the collector has not received metrics. API handlers schedule a post-response metric flush because short-lived serverless instances may stop before the periodic reader fires. Export can still fail during a collector outage; the database-backed incident history remains the durable source for operational events.
Sentry application monitoring
The application uses Sentry JavaScript SDK 11.4.0, compatible with installed Next.js 15.5, React 19 and TypeScript 5.9. Node must be 20.19+ or 22.12+ (the deployed worker uses Node 22). Browser initialization is in instrumentation-client.ts, including navigation capture. Node and Edge initialization are loaded from instrumentation.ts; Node initialization runs even without OTLP configuration. No current route declares Edge runtime, and there is no middleware. The Edge configuration covers a future actual Edge entry point without importing Node modules.
app/error.tsx and app/global-error.tsx capture browser rendering failures. Server error digests are skipped by browser boundaries because the original server exception is already reported by Next's onRequestError hook. SDK auto-wrappers cover Server Components, actions and unhandled route exceptions. instrumentApi and reportFailure cover caught infrastructure failures. The helper reports the original error object once; the SDK also deduplicates the original object across auto-wrappers. Expected plan limits, Next navigation signals and normal cancellation are excluded. Customer handler errors, retry results, lease expiry, validation and authorization responses remain workload outcomes, not application defects. Email/provider and notification/database failures are infrastructure errors; notification retries and response contracts are unchanged.
The Next.js Node runtime uses Sentry's makeFetchTransport with native fetch for error envelopes. This avoids the native streaming transport's EPIPE failure observed during Vercel startup verification. The maintenance worker keeps its verified Node transport. A local HTTP test initializes the real server configuration, requires a fetch request, and checks that the delivered envelope contains only the sanitized exception.
Ownership and initialization order
In the web process, the existing @vercel/otel provider exclusively owns OpenTelemetry context, instrumentation, propagators, sampler, OTLP export and API metrics. It is registered first, with the original automatic OTLP processor and metric reader when a collector is configured. Queue gauges and bounded-cardinality labels are unchanged. Sentry initializes second with enableOpenTelemetrySetup: false, openTelemetryIntegration(), and no Sentry tracesSampleRate/tracesSampler on Node. This follows Sentry's custom OpenTelemetry pipeline guidance.
The separate maintenance process owns a NodeTracerProvider, initialized before Sentry. It preserves standard W3C context and optional OTLP export, and creates manual maintenance spans without auto-instrumenting customer endpoints.
Sentry receives spans through a separate batched OTLP exporter. It samples deterministically by trace ID (default 5%), strips attributes, events, links, status messages and resource data, and includes only static operation/route names, timing, trace IDs, release and environment. The original OTLP copy is untouched. Thus Sentry cannot expose database parameters, SQL statements or customer payloads. The collector's existing sampler still bounds spans that exist; Sentry's export rate only reduces the Sentry copy. Setting Sentry sampling to zero keeps error capture active. Sentry does not register a competing provider or duplicate OpenTelemetry instrumentation. Node default integrations are restricted to error filtering, deduplication and fatal-error handling; HTTP, database, context-line and other automatic instrumentation are excluded. Browser tracing uses Sentry's own browser instrumentation.
Vercel after() schedules bounded Sentry delivery and span flushing without delaying the response, separately from the existing metric flush. SDK auto-wrappers handle framework lifecycle delivery. Reporting failures cannot change API responses. Fetch instrumentation never automatically propagates context to external endpoints; durable job traceparent/tracestate storage and claim propagation are unchanged. Browser propagation is restricted to relative /api/ destinations; no arbitrary customer origin receives Sentry baggage.
Configuration and isolation
Set SENTRY_ENABLED=true and SENTRY_DSN on the Vercel server; set NEXT_PUBLIC_SENTRY_ENABLED=true and NEXT_PUBLIC_SENTRY_DSN at build time for browser reporting. Railway uses SENTRY_WORKER_DSN (falls back to server DSN only outside previews). Use the separate getratchet-web and getratchet-maintenance projects in the EU getratchet organization. These DSNs identify ingestion destinations; they are not upload credentials.
Sentry's OTLP transport uses explicit configuration and only its DSN authentication. It never inherits OTEL_EXPORTER_OTLP_HEADERS or OTEL_EXPORTER_OTLP_TRACES_HEADERS; those remain exclusive to the existing collector. The standard OTLP exporter merges these headers even with explicit configuration, which would leak collector credentials and cause Sentry to reject duplicate authentication. A local HTTP regression test checks the actual transmitted headers.
Production/development/preview are derived from deployment context, separate from release. Non-production requires the respective SENTRY_ALLOW_NON_PRODUCTION or NEXT_PUBLIC_SENTRY_ALLOW_NON_PRODUCTION=true. Preview additionally requires SENTRY_PREVIEW_DSN / NEXT_PUBLIC_SENTRY_PREVIEW_DSN; inherited production DSNs never enable preview reporting. Existing preview database, email, OAuth and OTLP isolation stays in place. Browser deployment context is explicitly compiled by next.config. Disable reporting with the enablement flags; missing DSNs never break builds or development.
SENTRY_TRACES_SAMPLE_RATE and NEXT_PUBLIC_SENTRY_TRACES_SAMPLE_RATE accept 0–1, default 0.05. Start with 0.05 in production and adjust against usage. Routine health/poll routes are filtered to reduce volume. Invalid rates use the conservative default. Use SENTRY_RELEASE=<Git SHA> identically for web, source-map upload and worker; Vercel/Railway Git SHA variables are fallback sources. CI supplies github.sha. Do not use deployment environment as a release identifier. The source-map build also injects NEXT_PUBLIC_SENTRY_RELEASE.
Errors retain the SDK's default 100% capture independently of trace sampling. sentryClientOptions removes the internal trace sampleRate before SDK initialization: Sentry's option with that name would otherwise sample errors. Production verification checks the actual browser client options and received exception envelopes, alongside trace delivery.
Privacy, Replay, profiling and logs
sendDefaultPii: false is the declared policy. In SDK v11 that option was replaced by dataCollection: the effective controls explicitly disable user info, cookies, headers, bodies, query parameters, database values, queues, AI/GraphQL payloads, local variables and source context lines. See v11 migration. The final event callback reconstructs an allowlisted event with generic exception messages, source locations, safe trace IDs and a fixed failure-operation tag. Request/user/tenant context, breadcrumbs, raw messages, SQL, provider responses and arbitrary extras never leave the process. Organization-level ingestion scrubbing and IP-storage prevention add a second boundary; automatic public JavaScript source fetching is disabled in favor of private uploaded maps. Browser logout clears user context and stops Replay. No global tenant/user identity is set; concurrent request scopes are isolated. Error payloads are deliberately less descriptive; use release, stacks and static operation tags for investigation.
Replay is off by default. Enable only after reviewing the product's privacy needs: NEXT_PUBLIC_SENTRY_REPLAY_ENABLED=true plus a nonzero NEXT_PUBLIC_SENTRY_REPLAY_SAMPLE_RATE. Text and inputs are masked, media is blocked, the entire document is blocked, incremental/custom/network/console recording events are dropped and metadata URLs are scrubbed at the final transport boundary and response bodies are never recorded. This deliberately provides limited Replay detail; unblocking console/job views requires a separate privacy review. Error-triggered Replay is off. Structured Sentry logs and custom metrics are dropped; Grafana remains the metric destination. Profiling is excluded to avoid native runtime compatibility, cost and sensitive function context; no profiling package is installed.
CSP already allows outbound connections (it restricts frame ancestors, objects, base URI and forms). No tunnel is required; security headers and redirects are preserved. Ad blockers can block direct browser ingestion. A future tunnel must have rate limiting and strict DSN destination validation; never add an open forwarding proxy.
Private source maps and build credentials
withSentryConfig wraps next-intl and preserves tracing includes, Prisma WASM assets, redirects and security headers. Set build-only SENTRY_ORG, SENTRY_PROJECT and SENTRY_AUTH_TOKEN for private web source-map upload. Uploads fail the build on an actual upload failure; a missing token leaves builds usable and produces a visible warning for enabled Vercel production reporting. Source maps generated for upload are deleted after upload; productionBrowserSourceMaps stays off. Do not put an auth token in NEXT_PUBLIC_*, app runtime secrets, Docker ARG/ENV or a checked-in file. Vercel project environment variables are also available to runtime functions, so do not store the upload token there. Supply it only to a CI vercel build process and deploy the resulting .vercel/output with vercel deploy --prebuilt after removing the token from the process environment. The normal CI web build uploads maps with a step-scoped token; a separate hosted Git build must upload its own matching artifacts through that prebuilt flow.
pnpm worker:build creates dist/platform-worker/index.cjs and a private map, and injects debug IDs. Docker builds that same artifact without an upload token. CI runs pnpm sentry:upload-worker with SENTRY_WORKER_PROJECT and release SHA when a main-branch push has the build token; missing CI credentials emit a warning. Keep the worker map inside the private container/filesystem, never a public CDN. Use the same lockfile and commit for CI upload and Railway build. SENTRY_AUTH_TOKEN must never be present in a Railway runtime image.
Worker sweeps and alerts
Worker instrumentation loads before application modules. Startup, per-task/page and fatal errors are captured; operations have static OpenTelemetry spans. SIGTERM/SIGINT stops new work and preserves existing database cleanup and health behavior, then flushes Sentry with a deadline. Fatal errors retain exit code 1 and have a hard shutdown deadline. Failed tasks continue independent tasks and keep the original maintenance failure count. No customer-owned worker automatically reports to GetRatchet.
SENTRY_MAINTENANCE_MONITOR_ENABLED=true enables the fixed getratchet-maintenance cron check-in. It spans a complete cursor sweep, accumulates failure across pages, and uses an interval of ceil(configured idle milliseconds / 60000), minimum one minute, plus five minutes' check-in margin and a 15-minute maximum sweep runtime. Three failing sweeps open an issue, one successful sweep recovers. This accommodates healthy idle pauses up to 15 minutes. Do not enable this until a production worker is actually running; verify monitor availability on the existing plan without purchasing features. One replica remains the default. If a complete sweep takes more than 15 minutes, revise the maximum runtime against measured workloads.
The configured defect/regression alert is 1328453 and sustained production errors alert is 1328620. Both cover web and maintenance with 30-minute throttling and suggested assignees / active-member notifications. The overlapping generated maintenance alert is disabled. Use production-scoped issue alerts for new defects and regressions, with at least 30-minute per-issue throttling. Add a sustained increase rule (e.g. more than ten error events in five minutes) for production infrastructure errors and a worker failure rule for the maintenance project. Alerts must not include expected customer retry outcomes, already excluded at capture. Inspect generated project alerts before adding rules to avoid duplicates. Route to the existing team/member notification destination; do not buy or enable paid add-ons.
Verification and troubleshooting
Focused tests exercise outbound envelope redaction, concurrent isolation, object deduplication, expected-error exclusions, preview gating, Sentry-only OTLP filtering and idle monitor scheduling. Run pnpm typecheck, pnpm test, pnpm sdk:build, pnpm worker:build, pnpm build and Prettier for changed files.
For local live testing, create a gitignored .env.sentry.local with the chosen test DSNs, explicit non-production enablement, the exact Git SHA and sampling 1. Run node --env-file=.env.sentry.local --import tsx scripts/verify-sentry.ts for an isolated synthetic worker event/span. For browser/server tests, add SENTRY_VERIFY_ENABLED=true and a temporary SENTRY_VERIFY_TOKEN, run Next dev with that env file, and open /sentry-verification. Buttons test browser exceptions, rejected promises and React rendering. POST /api/sentry-verification with x-sentry-verify and ?handled=true to test converted responses; omit handled for an unhandled server exception. The page also tests Server Component and Server Action failures. These mechanisms are disabled in production, on any Vercel deployment, or without explicit local enablement; unauthorized calls return 404. Never load production job/customer data to test reporting. Set verification flags back off afterwards.
Confirm in Sentry the event ID, static operation tag, environment, commit release, safe trace IDs, absence of the redaction canary, and readable stack locations. A successful SDK flush is delivery-attempt evidence only: confirm the event in the project. Test private source maps with an uploaded build and confirm original file/line/source context. Confirm deployed Vercel and Railway capture separately; local events do not prove deployment operation. If errors do not arrive, check enablement, DSN, non-production/preview flags, outbound network and post-response lifecycle. If traces do not arrive, check both the provider sampler and Sentry copy rate. If stacks stay minified, verify upload/debug ID/release and the precise deployed commit.
For a controlled production rollout, temporarily set server-only SENTRY_DEPLOYMENT_VERIFY=true. Each worker captures one synthetic, redacted startup event. Each server process captures one event from the first instrumented API request using Next�s post-response lifecycle; initialization-time network uploads can stall on Vercel. Both use a static verification span and bounded flushing. It does not read customer data, change health semantics, or expose a throwing HTTP route. Temporarily setting server tracing to 1 makes the verification trace observable; restore 0.05 and disable the probe with a subsequent rollout after confirming events and private source locations. Never compile a public browser verification flag. Browser delivery can be checked by an operator injecting a synthetic exception through browser developer tools on the deployed app.
Rollback: disable runtime/browser enablement (rebuild browser configuration), disable maintenance monitoring before stopping the worker, and disable Replay. OTLP/Grafana continue independently. Remove upload credentials from the build environment if disabling uploads; do not remove existing OpenTelemetry configuration.
Verified setup and delivery (2026-10-03)
The EU organization and both projects are provisioned. Enhanced privacy, default ingestion scrubbers and IP-storage prevention are enabled; public JavaScript source fetching is disabled. Global sensitive fields include city, subdivision, region, country_code and ip_address: Sentry can derive geography during ingestion even without storing IP addresses. Sentry's guidance notes a 5–15 minute propagation delay; verify a fresh event after that delay, because older events retain previously derived geography. The restricted org:ci upload token is stored in gitignored local build configuration and the wauul/getratchet Actions secret SENTRY_AUTH_TOKEN. It is scoped to build/upload steps, with no Vercel or Railway runtime copy.
Safe local live tests delivered browser exceptions/rejections, React boundary errors, handled/unhandled API errors, Server Component and Server Action errors, and worker startup/page/fatal errors. Examples: handled server error, browser error, Server Action error and source-mapped maintenance error. The last issue resolves private compiled frames to lib/platform-maintenance.ts and platform-worker/runtime.ts. Sentry also received trace 57cb0f2c1b03eb90981ab5a972e639e3 with a maintenance.page span, release attribution and the development environment. These initial test events use base SHA e7410705da25d4799c828d94c5c686799c6913ba; release deployments must use the final integrated commit SHA.
The optimized Next build also delivered event 7318d77f92ea4750a365cd1f320c4fee, tagged preview, release f535c0a8f034f5928a6cfb9752cd64f7c8785e5d and database.health. It resolves uploaded frames to lib/health.ts, app/api/health/route.ts and lib/telemetry.ts. A deliberately unavailable local database tested the original 503 health response without touching a deployed database. The production-mode verification API returned 404 and the page rendered Next's not-found boundary (streaming may return HTTP 200). The Linux worker image was healthy against disposable Postgres, completed maintenance and exited 0 with database cleanup on a real SIGTERM. Linux and Windows worker builds produced the same debug ID, 3744b5a2-27d7-566d-a6b9-61cbbdbb344d.
All 303 tests passed with disposable Postgres. Typecheck, SDK build, worker build/image, dependency security check, changed-file formatting and optimized Next build passed. The credentialed build uploaded private Node/Edge/browser maps; a scan of 776 generated files found no upload-token occurrence and no public source maps. GitHub CI also passed its database, restore, Python, SDK and build checks.
These local tests do not establish deployed production operation. Production DSNs, enablement and 5% sampling are configured on Vercel and Railway; Replay and the worker cron flag are off, and Railway variable changes did not trigger a rollout. Activation still requires the matching private web artifact upload/prebuilt deployment, Railway rollout and remote event verification. Enable the cron monitor only after a healthy production worker is running. Keep production verification routes disabled and use a controlled operator procedure for remote verification.
Customer-owned worker opt-in
Published TypeScript/Python SDKs and the operator CLI do not import Sentry or initialize a reporting account. The following integrations belong in the customer's application and use their own DSN. The customer must supply their own privacy filters. Application initialization is never bundled into the SDK.
TypeScript with an existing OpenTelemetry provider (Sentry v11):
// Register the customer's existing provider first; keep its exporter/propagators.
import * as Sentry from '@sentry/node';
Sentry.init({
dsn: process.env.MY_SENTRY_DSN,
enableOpenTelemetrySetup: false,
integrations: [Sentry.openTelemetryIntegration()],
dataCollection: { userInfo: false, cookies: false, httpHeaders: false,
httpBodies: [], urlQueryParams: false, databaseQueryData: false },
beforeSend: event => { delete event.request; delete event.extra; delete event.breadcrumbs; return event; },
});
// Keep the existing Ratchet worker's own onError policy and classify infrastructure
// versus customer handler failures before captureException. Shutdown with close(2000).Python error-only with an existing OpenTelemetry provider:
import os
import sentry_sdk
sentry_sdk.init(
dsn=os.environ.get('MY_SENTRY_DSN'),
send_default_pii=False,
default_integrations=False,
traces_sample_rate=None, # preserve the customer's tracer provider
)
# Capture only errors the customer explicitly chooses to report, with their
# own before_send scrubber. Call sentry_sdk.flush(timeout=2) during shutdown.