Run a local fail-once destination, inspect retries and preserve destination-side idempotency.
Recover a webhook delivery without duplicating its effect
The producer, worker and sample destination run on your computer; GetRatchet coordinates the jobs remotely: a producer queues a webhook, a local destination returns HTTP 503 once, and your worker retries it. The destination remembers the operation key, so a repeated delivery returns the stored result. Only the harmless sample payload is sent to GetRatchet. No email, payment or request to an external webhook destination is made.
Prepare an isolated development environment
Use a development project/environment, not production. Create separate INGEST and WORKER keys in Settings, both scoped to that environment. Allow only deliver_webhook@1 on the WORKER key. Quickstart keys cannot execute this handler. Keep credentials outside Git. Requires Node.js 22+.
npm install @getratchet/sdkCreate a private .env:
GETRATCHET_INGEST_KEY=replace_with_development_ingest_key
GETRATCHET_WORKER_KEY=replace_with_development_worker_keySave this as webhook.mjs and run node --env-file=.env webhook.mjs:
import { createServer } from 'node:http';
import { randomUUID } from 'node:crypto';
import { createRatchet, RetryableError, NonRetryableError } from '@getratchet/sdk';
// Local simulation only. Production needs a durable, atomic deduplication store.
const results = new Map();
let failOnce = true;
const destination = createServer((req, res) => {
const key = req.headers['idempotency-key'];
if (req.method !== 'POST' || req.url !== '/webhook' || !key) {
res.writeHead(400).end(); return;
}
req.resume();
req.on('end', () => {
if (results.has(key)) { res.writeHead(200).end(results.get(key)); return; }
if (failOnce) {
failOnce = false;
res.writeHead(503, { 'retry-after': '2' }).end(); return;
}
// The simulated business effect and its receipt share this operation key.
const receipt = JSON.stringify({ accepted: true });
results.set(key, receipt);
res.writeHead(200, { 'content-type': 'application/json' }).end(receipt);
});
});
await new Promise((resolve, reject) => {
destination.once('error', reject);
destination.listen(0, '127.0.0.1', resolve);
});
const target = `http://127.0.0.1:${destination.address().port}/webhook`;
const baseUrl = 'https://getratchet.app';
const producer = createRatchet({ baseUrl, apiKey: process.env.GETRATCHET_INGEST_KEY });
const worker = createRatchet({ baseUrl, apiKey: process.env.GETRATCHET_WORKER_KEY });
function retryAfterMs(value) {
if (!value) return undefined;
const seconds = Number(value);
const delay = Number.isFinite(seconds) ? seconds * 1000 : Date.parse(value) - Date.now();
if (!Number.isFinite(delay) || delay <= 0) return undefined;
// A provider wait over the API's 24-hour maximum needs manual rescheduling.
if (delay > 86400000) throw new NonRetryableError('Provider wait exceeds the supported retry window');
return Math.max(1000, Math.ceil(delay));
}
worker.registerTool({
name: 'deliver_webhook', version: '1',
handler: async (input, context) => {
const response = await fetch(target, {
method: 'POST', signal: context.signal,
headers: { 'content-type': 'application/json', 'idempotency-key': context.idempotencyKey },
body: JSON.stringify(input),
});
if (response.status === 429 || response.status === 503) {
throw new RetryableError('Destination temporarily unavailable', {
retryAfterMs: retryAfterMs(response.headers.get('retry-after')),
});
}
if (response.status === 400) throw new NonRetryableError('Destination rejected the input');
if (response.status === 401 || response.status === 403) throw new NonRetryableError('Check destination credentials');
if (!response.ok) throw new Error('Unexpected destination failure');
// Never send provider error bodies, credentials or response headers to GetRatchet.
return { delivered: true };
},
});
try {
const operationId = randomUUID(); // Persist/reuse this identity for the same business event.
const runId = await producer.startRun('Webhook delivery recovery');
await producer.enqueue({ event: 'customer.created', customerId: 'demo-customer' }, {
name: 'deliver_webhook', version: '1', endpoint: 'Development webhook recipe',
idempotencyKey: operationId, maxAttempts: 2, timeoutMs: 10000,
});
await producer.finishRun();
console.log('Inspect both attempts:', `${baseUrl}/runs/${runId}`);
await worker.worker.start({ concurrency: 1, pollIntervalMs: 1000,
signal: AbortSignal.timeout(90000), onError: () => console.error('Worker error; inspect configuration and run status') });
} finally {
destination.closeAllConnections();
await new Promise(resolve => destination.close(resolve));
}What you should observe
The first attempt fails with a retryable 503. The stored retry date respects the greater of the normal jittered policy delay and the destination's two-second minimum. With Standard backoff, expect roughly 24–36 seconds before the second attempt becomes eligible. The next attempt succeeds; the run shows both attempts. Circuit, pause, rate and concurrency gates can extend that wait. Change the simulated 503 to 429 to exercise rate limiting, or 400 to verify immediate exhaustion. Unknown thrown errors remain retryable by default.
This is normal durable work, with two billable attempts; it is not a synthetic test (synthetic tests allow only one attempt). Revoke the recipe keys when finished. If the process stops early, accepted work remains queued; restart a worker with the same handler version to finish it.
Before using a real destination
- Replace the local receiver with your authenticated HTTPS endpoint; keep the URL and credentials in worker configuration, never in untrusted job input.
- Verify a signature or bearer credential at the destination. Keep transport timeouts and propagate cancellation.
- Store operation identity, the business mutation and its receipt atomically in your destination database. The in-memory Map above is a demonstration, not production deduplication.
- Define what happens if the same key arrives with a different payload. Reject conflicting reuse. Keep receipts at least as long as your retry/replay horizon.
- Reuse the business event's stable identity across producer retries and recovery. A new key means a new operation.
- Inspect errors before replay. At-least-once execution means the destination may finish before a worker report is lost; GetRatchet cannot make an external side effect exactly once.
For deployment, follow customer worker deployment. For Python error classification, see Python SDK and provider retry decisions. The same destination-side idempotency requirement applies to either SDK.