Idempotency and Safe Reprocessing

Raise webhook delivery is at-least-once: the same event can arrive at the partner endpoint more than once. Retries on transient failures, timing edge cases where the partner endpoint processes but acknowledges slowly, and (rarely) platform-level redeliveries all produce duplicates that the partner side must handle. Without explicit idempotency handling, duplicate deliveries produce duplicate side effects — duplicate emails to donors, duplicate gift records in downstream systems, duplicate Slack notifications. This page covers the keying strategies, the deduplication storage patterns, and the operational practices that make at-least-once delivery safe.

Why duplicates happen

Three scenarios produce duplicate webhook deliveries:

Scenario	Description
Retry after timeout	Partner endpoint processes successfully but is slow to respond. Raise times out, retries. Partner now receives the same event twice.
Retry after partial failure	Partner endpoint accepts the event but a downstream system fails. Partner returns `5xx`. Raise retries. Partner has to ensure the second attempt doesn’t double-process.
Platform-level redelivery	Rare, but possible during platform maintenance or recovery scenarios. The platform may re-fire events that were marked as delivered.

The first two are common. The third is rare but worth handling defensively. The partner integration’s job: detect “I’ve already processed this event” before producing any side effects. The simplest reliable mechanism is a deduplication store.

The deduplication store

A dedup store records the unique identifier of every event the integration has processed. Before processing a new event, check the store. If the event is already there, skip processing. If not, add it and proceed.

JavaScript

class DeduplicationStore {
  async hasProcessed(eventKey) {
    return Boolean(await this.db.findByKey(eventKey));
  }

  async recordProcessed(eventKey, metadata = {}) {
    await this.db.insert({
      key: eventKey,
      processedAt: new Date(),
      ...metadata,
    });
  }
}

// Usage in the event processor
async function processEvent(event) {
  const key = buildEventKey(event);

  if (await dedupStore.hasProcessed(key)) {
    console.log(`Already processed ${key} — skipping`);
    return;
  }

  await performSideEffects(event);
  await dedupStore.recordProcessed(key, { eventType: event.eventType });
}

The store’s choice of database is flexible — Postgres, DynamoDB, Redis with persistence, anything that supports atomic key lookups with reasonable latency. The key requirement: the store must persist across the integration’s restarts (in-memory caches don’t count) and provide consistent reads within the deduplication window.

Choosing the deduplication key

The key uniquely identifies an event. The right key depends on what “the same event” means for the integration.

Option 1: contextId + event type

The simplest reliable key. The contextId from the webhook log is the ID of the entity that triggered the event (the Gift ID, Donor ID, etc.); combined with the event type integer, it uniquely identifies the event:

JavaScript

function buildEventKey(event) {
  return `${event.eventType}:${event.contextId}`;
}

This treats a single gift created event (eventType 10, contextId 9876) as the same event regardless of how many times it’s delivered. Strengths: simple, durable, works without depending on undocumented event payload structure. Weakness: doesn’t distinguish multiple legitimate updates to the same resource. A gift updated event for gift 9876 today is the same key as a gift updated event for gift 9876 tomorrow — they’ll be treated as duplicates. To handle multiple legitimate updates, see Option 3 below.

Option 2: payload hash

Compute a stable hash over the relevant fields of the event payload:

JavaScript

import crypto from 'crypto';

function buildEventKey(event) {
  const stableSubset = {
    eventType: event.eventType,
    resourceId: event.payload?.id,
    modifiedDate: event.payload?.modifiedDate,
  };
  return crypto
    .createHash('sha256')
    .update(JSON.stringify(stableSubset))
    .digest('hex');
}

Strengths: distinguishes events by their actual content, so multiple legitimate updates produce distinct keys. Weakness: depends on field names in the payload (id, modifiedDate) being stable. If Raise changes the payload shape, the key changes. Use a small, stable subset of fields rather than hashing the whole payload.

Option 3: contextId + event type + modifiedDate

A hybrid that distinguishes both the resource and the version of the resource:

JavaScript

function buildEventKey(event) {
  const ts = event.payload?.modifiedDate ?? event.payload?.modifiedDateTime ?? '';
  return `${event.eventType}:${event.contextId}:${ts}`;
}

This treats each update of a resource as a distinct event (different modifiedDate) but treats duplicate deliveries of the same update as the same event. Recommended for most integrations. Distinguishes legitimate updates while still deduplicating delivery retries.

Option 4: delivery-level identifier (if available)

If Raise’s webhook payload or headers include a delivery ID (a unique identifier per delivery attempt), use that. This is the cleanest key — each delivery is uniquely identified.

The Raise OpenAPI spec doesn’t document a per-delivery identifier in the webhook payload or headers. Inspect real webhook deliveries (via the log endpoints or your endpoint’s request logs) to check whether one is included. If so, use it as the dedup key.

Time-bounded vs. permanent deduplication

The dedup store needs to remember processed events. Two approaches to retention:

Time-bounded retention

Keep dedup records for a bounded window (e.g., 7 days) and discard older entries. Acceptable when:

Retry attempts complete within the window (typically minutes to hours, well under 7 days).
Storage cost matters and the integration processes high event volume.

JavaScript

async function recordProcessed(key) {
  await db.insert({
    key,
    processedAt: new Date(),
    expiresAt: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000),
  });
}

// Periodic cleanup
async function cleanupExpiredDedupEntries() {
  await db.deleteWhere({ expiresAt: { lt: new Date() } });
}

Set the retention window longer than Raise’s maximum retry duration. If retries can continue for up to 24 hours, keep dedup records for at least 48 hours.

Permanent retention

Keep dedup records indefinitely. Acceptable when:

Storage cost is negligible.
Re-processing risk extends beyond the retry window (e.g., manual replays during incident recovery).
The integration’s audit requirements include “we processed event X at time T.”

For most partner integrations, permanent retention is simpler and avoids edge cases. A dedup table that grows by ~1 row per gift is manageable — even a million gifts a year is a small table.

Atomicity: check-and-record as a unit

A subtle race condition: two duplicate deliveries arrive simultaneously, both check the dedup store (neither finds the key), and both proceed to process. To prevent this, the check-and-record needs to be atomic.

Pattern 1: insert-then-process

Insert the dedup record first, with a unique constraint on the key. If the insert succeeds, you “won” the race — proceed to process. If the insert fails (unique constraint violation), another worker is already processing — skip.

JavaScript

async function processEvent(event) {
  const key = buildEventKey(event);

  try {
    await db.insert('dedup', { key, processedAt: new Date() });
  } catch (err) {
    if (isUniqueConstraintError(err)) {
      console.log(`Event ${key} already being processed — skipping`);
      return;
    }
    throw err;
  }

  // We won the insert — safe to process
  await performSideEffects(event);
}

Strengths: race-condition-safe with no explicit locks. Weakness: if processing fails after the insert, the dedup record stays and prevents legitimate retries. Handle by adding a status field and clearing on failure:

JavaScript

async function processEvent(event) {
  const key = buildEventKey(event);

  try {
    await db.insert('dedup', { key, status: 'processing', startedAt: new Date() });
  } catch (err) {
    if (isUniqueConstraintError(err)) return;
    throw err;
  }

  try {
    await performSideEffects(event);
    await db.update('dedup', { key }, { status: 'completed', completedAt: new Date() });
  } catch (err) {
    // Processing failed — remove the dedup record so retries can succeed
    await db.delete('dedup', { key });
    throw err;
  }
}

Pattern 2: transactional check-and-process

If your downstream side effects are themselves transactional, wrap the dedup check and the side effects in the same transaction:

JavaScript

async function processEvent(event) {
  const key = buildEventKey(event);

  await db.transaction(async (tx) => {
    const existing = await tx.findByKey('dedup', key);
    if (existing) {
      console.log(`Already processed ${key}`);
      return;
    }

    await performSideEffects(event, tx);
    await tx.insert('dedup', { key, processedAt: new Date() });
  });
}

Strengths: cleanest semantics — either everything happens (dedup record + side effects) or nothing. Weakness: requires side effects that work within the transaction. External calls (email sends, third-party API calls) typically don’t.

Idempotent side effects

The dedup store handles the “I’ve seen this before” check. But for side effects that touch external systems, idempotency at the side-effect layer adds robustness.

Idempotency keys on downstream API calls

If your integration writes to a downstream system that supports idempotency keys (e.g., Stripe, many modern APIs), pass the event key as the idempotency key:

JavaScript

async function notifySlack(event) {
  const eventKey = buildEventKey(event);

  await slackApi.postMessage({
    channel: '#donations',
    text: `New gift: ${event.payload.formattedAmount}`,
    idempotency_key: `raise-${eventKey}`,
  });
}

This adds a second line of defense — even if the dedup store check is bypassed somehow, the downstream system rejects the duplicate request.

Database upserts instead of inserts

If your integration writes records to a database, use upserts keyed by a stable identifier (the Raise resource ID, typically):

JavaScript

async function syncGiftToDownstream(gift) {
  // Upsert keyed by Raise gift ID — duplicate calls produce a single row
  await db.upsert('gifts', {
    raiseGiftId: gift.id,
    amount: gift.amount,
    date: gift.date,
    donorEmail: gift.donor?.email,
    lastSyncedAt: new Date(),
  });
}

This way, even if the same event is processed twice, the database ends up with one record per gift rather than two.

Email and notification deduplication

Email and Slack notifications are typically not idempotent at the platform level — sending the same email twice produces two emails in the donor’s inbox. For these side effects, the dedup store check is the only defense. Make sure it runs before the notification is sent.

JavaScript

async function sendThankYouEmail(event) {
  const key = buildEventKey(event);

  if (await emailDedup.hasSent(key)) return;

  await emailService.send({ to: event.payload.donor.email, ... });
  await emailDedup.recordSent(key);
}

A separate emailDedup store (distinct from the main event dedup store) lets you track “we already sent this specific email” independent of the broader event processing.

Reprocessing for recovery

Sometimes you want to reprocess an event — for example, recovering from a bug in the original processing logic, or replaying events after a downstream system restoration.

Selective reprocessing

For one-off recoveries, fetch the event from the webhook log and reprocess directly:

JavaScript

async function reprocessEvent(webhookId, logId) {
  // 1. Get the original payload
  const logResponse = await fetch(
    `https://prod-api.raisedonors.com/api/Webhook/${webhookId}/log/${logId}`,
    { headers: { Authorization: `Bearer ${process.env.RAISE_API_TOKEN}` } }
  );
  const log = await logResponse.json();
  const payload = JSON.parse(log.payLoad);

  // 2. Remove the dedup record so processing can proceed
  const key = buildEventKey({ eventType: log.eventType, contextId: log.contextId });
  await dedupStore.remove(key);

  // 3. Reprocess
  await processEvent({ eventType: log.eventType, payload });
}

This is operational tooling, not a partner-facing API surface — use it judiciously for specific recovery scenarios.

Bulk reprocessing for a time window

For wider recoveries, query the webhook logs across a time window and reprocess each:

JavaScript

async function reprocessWindow(webhookId, startDate, endDate) {
  // Paginate through logs for the time window
  const logs = await streamLogsForWindow(webhookId, startDate, endDate);

  for (const log of logs) {
    if (log.success !== 'Yes' && log.success !== 'True') continue;

    try {
      await reprocessEvent(webhookId, log.id);
    } catch (err) {
      console.error(`Reprocess failed for log ${log.id}:`, err);
    }
  }
}

For very large windows, throttle the reprocessing — bulk reprocessing of thousands of events can produce a thundering-herd effect on downstream systems. Process at a controlled pace.

Operational practices

A few practices that make idempotent processing robust in production:

Monitor the duplicate rate

Track how often the dedup store rejects duplicates. A small steady rate is normal (occasional retries). A sudden spike indicates either:

Partner-side issues causing slow responses and retry storms.
Raise-side issues causing redeliveries.
A bug in your event processing.

A sudden zero rate is also worth investigating — it suggests the dedup store may not be functioning.

Log the dedup decisions

Log each “skipping duplicate” decision with enough context to investigate later:

JavaScript

console.log('Skipping duplicate event', {
  key: eventKey,
  eventType: event.eventType,
  contextId: event.contextId,
  originalProcessedAt: existingRecord.processedAt,
  thisAttemptAt: new Date(),
});

The gap between originalProcessedAt and thisAttemptAt reveals the retry pattern — useful for understanding webhook delivery behavior.

Test the dedup path

Include tests that explicitly fire the same event twice and verify the second processing is skipped. This is easy to forget — the happy path of single-delivery events doesn’t exercise the dedup logic — but it’s the most important test for production reliability.

JavaScript

test('processing the same event twice produces no duplicate side effects', async () => {
  const event = buildTestEvent({ giftId: 9876 });

  await processEvent(event);
  await processEvent(event); // Same event again

  expect(downstream.giftRecords).toHaveLength(1);
  expect(emailService.sentEmails).toHaveLength(1);
});

A test of this shape, run in CI, catches dedup regressions before they reach production.

Where to go next

Local Testing

Test the dedup logic locally by replaying captured events.

Retry Behavior

The retry pattern that makes dedup necessary.

Webhooks Overview

The webhook log endpoints used for inspection and reprocessing.

Error Recovery Patterns

The broader retry and dead-letter patterns this idempotency story fits into.

​Why duplicates happen

​The deduplication store

​Choosing the deduplication key

​Option 1: contextId + event type

​Option 2: payload hash

​Option 3: contextId + event type + modifiedDate

​Option 4: delivery-level identifier (if available)

​Time-bounded vs. permanent deduplication

​Time-bounded retention

​Permanent retention

​Atomicity: check-and-record as a unit

​Pattern 1: insert-then-process

​Pattern 2: transactional check-and-process

​Idempotent side effects

​Idempotency keys on downstream API calls

​Database upserts instead of inserts

​Email and notification deduplication

​Reprocessing for recovery

​Selective reprocessing

​Bulk reprocessing for a time window

​Operational practices

​Monitor the duplicate rate

​Log the dedup decisions

​Test the dedup path

​Where to go next

Local Testing

Retry Behavior

Webhooks Overview

Error Recovery Patterns

Why duplicates happen

The deduplication store

Choosing the deduplication key

Option 1: contextId + event type

Option 2: payload hash

Option 3: contextId + event type + modifiedDate

Option 4: delivery-level identifier (if available)

Time-bounded vs. permanent deduplication

Time-bounded retention

Permanent retention

Atomicity: check-and-record as a unit

Pattern 1: insert-then-process

Pattern 2: transactional check-and-process

Idempotent side effects

Idempotency keys on downstream API calls

Database upserts instead of inserts

Email and notification deduplication

Reprocessing for recovery

Selective reprocessing

Bulk reprocessing for a time window

Operational practices

Monitor the duplicate rate

Log the dedup decisions

Test the dedup path

Where to go next