> ## Documentation Index > Fetch the complete documentation index at: https://docs.virtuous.org/llms.txt > Use this file to discover all available pages before exploring further. # Retry Behavior > What Raise does when a webhook delivery fails — what's known from the webhook delivery logs, what's not documented in the spec, and the defensive patterns to use given the gaps. When a webhook delivery doesn't succeed — the partner endpoint returns a non-`2xx` status code, times out, or is unreachable — Raise retries the delivery. The webhook log endpoints expose retry history so partners can investigate, but the OpenAPI spec doesn't document the specific retry schedule, the maximum number of attempts, the backoff strategy, or the timeout that triggers a retry. This page covers what's known from the log endpoints, what's not in the spec, and the defensive patterns that partner integrations should use until the spec documents the retry contract explicitly. **⚠️ Spec gap:** The Raise OpenAPI spec does not document the webhook retry schedule, maximum retry attempts, backoff strategy, or the timeout for a delivery attempt. The patterns on this page are general defensive guidance based on common webhook system behavior; partners building production integrations should confirm specific behaviors with the platform team if their integration design depends on them. ## What the spec confirms Looking only at what's documented: * The `WebhookLogListModel` schema includes `success` (string), `httpStatusCode` (string), and `serverResponse` (string) fields — meaning Raise records the outcome of each delivery attempt. * Multiple log entries can exist for the same `contextId` (the ID of the entity that triggered the event) — meaning Raise can attempt delivery more than once for the same event. * The log endpoints support pagination — meaning delivery history is retained for at least some retention window. What this *doesn't* tell partners: * How many retry attempts Raise makes per failed delivery * How long Raise waits between attempts * What status codes from the partner endpoint trigger retries vs. permanent failures * How long Raise waits for a response before considering a delivery timed out These specifics matter for partner integrations designing their endpoints — but they aren't in the spec. The defensive patterns below assume the most common webhook system behavior and code conservatively. *** ## The retry-from-the-partner-perspective view Even without spec-level retry details, partners can observe retries through the webhook logs. Multiple log entries with the same `contextId` and different `createdDate` values indicate Raise retried the delivery. ### Inspecting retry history for a specific event ```bash cURL theme={null} curl -G "https://prod-api.raisedonors.com/api/Webhook/123/log/list" \ -H "Authorization: Bearer YOUR_API_TOKEN" \ --data-urlencode "Filter=12345" \ --data-urlencode "Take=20" \ --data-urlencode "SortBy=createddatetime" ``` If multiple log entries return for the same gift, donor, or other resource ID, those are the retry attempts. The `httpStatusCode` and `success` fields show whether each attempt eventually succeeded. ```javascript JavaScript theme={null} async function getRetryHistoryForEvent(webhookId, contextId) { const params = new URLSearchParams({ Take: '50', SortBy: 'createddatetime', Descending: 'false', }); const response = await fetch( `https://prod-api.raisedonors.com/api/Webhook/${webhookId}/log/list?${params}`, { headers: { Authorization: `Bearer ${process.env.RAISE_API_TOKEN}` } } ); const logs = await response.json(); // Filter to entries for this specific entity const eventLogs = logs.items.filter((log) => log.contextId === contextId); return eventLogs.map((log) => ({ attempt: log.id, attemptedAt: log.createdDate, httpStatusCode: log.httpStatusCode, success: log.success, response: log.serverResponse?.substring(0, 200), })); } ``` For a delivery that succeeded on the first try, you'll see one entry. For one that failed and was eventually retried successfully, you'll see multiple entries — the earlier ones with non-`2xx` status codes and the later one with `2xx`. ### Distinguishing "currently retrying" from "permanently failed" If retries continue for an extended period — minutes to hours — the delivery is likely in an active retry loop. If no more retries appear after a sustained gap, Raise has likely given up on the delivery. Without a spec-documented retry schedule, the practical heuristic is: | Pattern | Likely state | | ------------------------------------------------------------- | --------------------------------------- | | One entry, `2xx` status | Delivered successfully on first attempt | | Multiple entries, final one `2xx` | Delivered successfully after retries | | Multiple entries, all non-`2xx`, increasing time gaps | Likely still retrying | | Multiple entries, all non-`2xx`, no entries for several hours | Likely abandoned — needs investigation | For integrations that need to know when an event was permanently lost, the most reliable signal is a periodic reconciliation check that compares expected events (from polling the underlying resources) against observed webhook deliveries. See [Reconcile with CRM+: pattern 3 (periodic backstop)](/raise/workflows/reconcile-with-crm-plus#pattern-3-periodic-reconciliation-backstop). *** ## How the partner endpoint controls retry behavior The partner endpoint's HTTP response shapes what Raise does next. The general behavior most webhook systems use (which partners should assume Raise also follows until the spec confirms otherwise): | Partner response | Likely treated as | | -------------------------------------------------- | -------------------------------------------------------------------------------------- | | `200`–`299` | Success — no retry | | `400` (Bad Request) | Permanent failure — no retry (Raise can't fix a request the partner says is malformed) | | `401` (Unauthorized) | Permanent or transient — depends on the implementation | | `404` (Not Found) | Permanent — the endpoint is gone | | `408` (Request Timeout), `429` (Too Many Requests) | Transient — retry expected | | `500`–`599` | Transient — retry expected | | Connection refused, DNS error | Transient — retry expected | | No response within timeout | Transient — retry expected | The implication for partner endpoints: **return the right status code for the situation**. * If your endpoint received the request successfully and intends to process it, return `200 OK` immediately. Don't wait for processing to complete before responding. * If your endpoint can't accept the request right now (rate limited, in maintenance, etc.), return `503 Service Unavailable` and Raise will retry later. * If the request is genuinely malformed (signature verification failed, body parsing failed), return `400` or `401` — but make sure your endpoint really can't handle the request, because Raise typically won't retry these. ### Don't return 200 for failures A tempting anti-pattern: return `200 OK` always, then log errors internally and hope someone notices. This breaks Raise's retry mechanism — every delivery looks successful to Raise even when the partner actually failed to process it. ```javascript JavaScript theme={null} // ❌ Anti-pattern app.post('/raise-webhooks', (req, res) => { try { processEvent(req.body); } catch (err) { console.error('Processing failed:', err); } res.status(200).send('OK'); // Always returns success }); // ✅ Correct pattern app.post('/raise-webhooks', async (req, res) => { // 1. Verify signature if (!verifySignature(req)) { return res.status(401).send('Invalid signature'); } try { // 2. Queue for async processing await eventQueue.publish(req.body); // 3. Acknowledge receipt res.status(200).send('OK'); } catch (err) { // 4. Queue infrastructure failed — let Raise retry res.status(503).send('Service Unavailable'); } }); ``` The correct pattern queues the event for processing and acknowledges receipt. The async processor — running outside the webhook handler — handles errors during actual processing, using its own retry and dead-letter mechanisms separate from the Raise retry mechanism. *** ## Idempotency is non-negotiable Because Raise retries failed deliveries, the same event can arrive at the partner endpoint more than once. **The partner side must handle duplicates safely** — without producing duplicate side effects. See [Idempotency and Safe Reprocessing](/raise/webhooks/idempotency-and-safe-reprocessing) for the patterns. The summary: * Identify each event uniquely (typically by `contextId` + event type, or by a delivery ID if available). * Track already-processed events in a deduplication store. * Check the store before processing; skip if already processed. * Update the store after successful processing. Without this, retries can produce double-charged donors (if your integration triggers payment flows), duplicate thank-you emails (annoying but not catastrophic), or doubled records in downstream systems (catastrophic for accounting). *** ## Endpoint design that survives retries gracefully A few patterns that make the partner endpoint robust against Raise's retry behavior: ### Acknowledge quickly Return `200 OK` within a few seconds. Most webhook systems use timeouts in the 10–30 second range; some shorter. An endpoint that takes 30 seconds to respond produces unnecessary retries even when the request is being handled correctly. ```javascript JavaScript theme={null} app.post('/raise-webhooks', async (req, res) => { // ... verify signature ... // Acknowledge first res.status(200).send('OK'); // Process after — outside the request lifecycle setImmediate(() => processEventAsync(req.body)); }); ``` `setImmediate` (or equivalent in other frameworks) lets the response complete before processing begins. ### Queue-based decoupling A more robust pattern: use a message queue (SQS, Cloud Tasks, Redis-backed queue, etc.) between the webhook receiver and the event processor. ```mermaid theme={null} graph LR Raise["Raise"] Receiver["Webhook
Receiver"] Queue["Message
Queue"] Worker["Event
Worker"] DB["Downstream
System"] Raise -->|POST /raise-webhooks| Receiver Receiver -->|enqueue| Queue Receiver -->|200 OK| Raise Queue -->|drain| Worker Worker -->|process| DB ``` Benefits: * The receiver is small and fast — it does only signature verification and enqueueing. * Processing failures are handled by the queue's retry mechanism, not by relying on Raise to retry. * During traffic spikes, the queue absorbs load instead of overwhelming the processor. ### Handle the timeout-but-succeeded edge case A subtle issue: your endpoint processes the event successfully but is slow to respond. Raise times out, returns the delivery as failed, and retries. Your endpoint receives the same event a second time. Without idempotency, you'd process it twice. The defense: * Acknowledge quickly (as above) so timeouts are rare. * Implement idempotency (the next page) so duplicate processing is harmless when retries do occur. These two defenses together produce a system that's resilient to retries without depending on knowing the exact retry timing. *** ## Investigating delivery problems When the partner reports "we missed event X" or "we got the same event twice": Use `GET /api/Webhook/{id}/log/list` filtered to the relevant time range or context ID. One entry = single successful delivery. Multiple entries = retries. `2xx` on the final attempt = Raise considers it delivered. Non-`2xx` on the latest = Raise may still be retrying or have given up. `serverResponse` shows what the partner endpoint returned. A `200 OK` with an error message in the body is a partner-side processing issue, not a delivery issue. If no log entries exist for an event you expected, the event may not have fired at all. Verify the underlying resource was actually created/updated in Raise. The combination of webhook logs and the underlying resource records lets partner integrations triage most delivery issues without needing platform-team escalation. Save escalation for the cases where the logs show a permanent failure and the cause isn't clear from the response. *** ## Common scenarios ### Scenario: partner endpoint was deployed during a maintenance window If the partner endpoint was down for an hour during a deploy or outage, events that fired during that window arrived `5xx` from the unreachable endpoint. Raise retried those events. **What to do:** * Check the webhook logs after the partner endpoint is back up to confirm retries succeeded. * If some retries are still failing after the partner endpoint is back, check whether Raise has given up on those specific events (no more log entries in the past hour). * For permanently lost events, run a reconciliation pass to catch the missed records and process them through your normal ingestion path. ### Scenario: signature verification was broken in a deploy If a partner deployment introduced a signature verification bug, every incoming webhook returned `401 Unauthorized`. Raise may have stopped retrying these as permanent failures. **What to do:** * Fix the verification bug. * Run a reconciliation pass to catch all events that were rejected during the window. * Add an integration test for signature verification so this doesn't recur. ### Scenario: persistent 500 errors from the partner endpoint The partner endpoint has been returning `500 Internal Server Error` for an extended period. Raise has likely been retrying with increasing delays and may have given up. **What to do:** * Identify and fix the root cause of the `5xx` errors. * After the fix, run a reconciliation pass to catch missed events. Don't rely on Raise to retry events from days ago. * Set up monitoring on the partner endpoint's error rate to catch this faster next time. ### Scenario: rate-limited at the partner endpoint If the partner endpoint applies rate limiting and starts returning `429 Too Many Requests`, Raise's retries may make the situation worse — backing off, then retrying, then being rate-limited again. **What to do:** * Raise the partner endpoint's rate limit (this is a partner-side concern). * Or, add a buffering queue in front so the partner can absorb bursts without rate-limiting Raise. *** ## Monitoring webhook delivery health For partner integrations operating in production, two metrics are worth tracking: | Metric | What it shows | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Delivery success rate** | The fraction of webhook log entries with `success: "Yes"` (or equivalent). Should be near 100%. | | **Time from event to processing complete** | The end-to-end latency from when the underlying resource changed to when your processor finished handling the event. Spikes indicate a queue backlog or processor issue. | A periodic job can poll the webhook log endpoints and emit these as metrics: ```javascript JavaScript theme={null} async function emitWebhookHealthMetrics() { const oneHourAgo = new Date(Date.now() - 60 * 60 * 1000).toISOString(); const params = new URLSearchParams({ Take: '1000', SortBy: 'createddatetime', Descending: 'true', }); const response = await fetch( `https://prod-api.raisedonors.com/api/Webhook/log/list?${params}`, { headers: { Authorization: `Bearer ${process.env.RAISE_API_TOKEN}` } } ); const logs = await response.json(); const recent = logs.items.filter((log) => log.createdDate >= oneHourAgo); const successful = recent.filter((log) => log.success === 'Yes').length; const failed = recent.filter((log) => log.success !== 'Yes').length; metrics.gauge('raise.webhook.successful_per_hour', successful); metrics.gauge('raise.webhook.failed_per_hour', failed); metrics.gauge('raise.webhook.success_rate', successful / (successful + failed)); } ``` Run this on a periodic cadence (every 5 minutes is a reasonable default) and alert on sustained drops in success rate. The exact value of the `success` field (string `"Yes"`, `"True"`, boolean, etc.) is not documented in the spec. Inspect actual log entries to confirm the value, and adjust the metric calculation accordingly. *** ## Where to go next The companion pattern to retry handling — process duplicate deliveries safely. Test the retry-and-acknowledge pattern locally before production. The full subscription lifecycle including log endpoint use. Broader patterns for handling failures across all of the partner integration.