Idempotency & Safe Reprocessing

A webhook handler will receive duplicate deliveries of the same event. This is not a bug in your code or a bug in Virtuous — it is a structural property of any reliable webhook system, and any handler that doesn’t anticipate it will eventually produce double-counted gifts, duplicate emails, or corrupted donor records. This page covers why duplicates happen, why ordinary database writes are not enough to defend against them, and the idempotency patterns that make a handler safe to re-run.

Why duplicates happen

Four scenarios produce duplicate webhook deliveries to your endpoint:

Scenario	Description
Virtuous retries after a timeout	Your handler processed the event successfully but took longer than the delivery timeout to respond. Virtuous treats the delivery as failed and retries — your handler will receive and process the same event again.
Network glitches between Virtuous and you	The acknowledgement response is lost in transit. Virtuous never sees the `2xx` and retries.
Your service crashes mid-acknowledgement	Your handler processed the event, then crashed before responding. Same outcome as a timeout.
Replay attacks or accidental re-delivery	A captured webhook delivery is replayed by an attacker or by a misbehaving proxy. Signature verification confirms authenticity but not freshness — see Signature Verification.

The first three are common and unavoidable. The fourth is rare but possible. All four produce the same observable behavior on your end: the same event ID arriving more than once.

Receiving a duplicate is the normal case, not the exceptional one. Plan for every event to be delivered at least twice over the lifetime of your integration, even on a perfectly healthy system. A handler that produces double-counted side effects on a duplicate delivery is not “robust under unusual conditions” — it is unreliable under normal conditions.

Why ordinary database writes are not enough

A common first attempt at idempotency relies on database uniqueness — “I’ll just upsert the record, so a second delivery is a no-op.” This works for some cases but fails in others, depending on what side effects the handler produces.

Operation	Idempotent via database uniqueness alone?
Upsert a record by Virtuous ID (`UPDATE WHERE id = ?` or `INSERT ... ON CONFLICT DO UPDATE`)	Yes — multiple writes produce the same final state.
Insert a record with auto-generated ID	No — each delivery creates a new row.
Increment a counter (`UPDATE ... SET count = count + 1`)	No — each delivery increments.
Send an email	No — each delivery sends another email.
Charge a credit card	No — each delivery is a separate charge.
Call a downstream API that creates a record	Usually no — depends on the API.
Append an event to an audit log	No — each delivery appends.

The pattern: idempotency through database state works for “make the row look like this” operations and fails for “do this thing once” operations. Most real partner integrations have at least some “do this thing once” side effects — sending a notification, posting to an accounting system, triggering an automation. Those need an explicit idempotency mechanism.

The idempotency-key pattern

The canonical solution is to track which events you have already processed. Before processing an event, check whether you have seen its identifier; if you have, skip processing. If you have not, process the event and then record the identifier.

JavaScript

async function processWebhookEvent(event) {
  // 1. Try to claim the event for processing.
  const claim = await idempotencyStore.claim(event.eventId);
  if (claim.alreadyProcessed) {
    console.info('Skipping duplicate event', { eventId: event.eventId });
    return;
  }

  try {
    // 2. Process the event — including any side effects.
    await handleEvent(event);

    // 3. Mark the event as fully processed.
    await idempotencyStore.complete(event.eventId);
  } catch (err) {
    // 4. On failure, release the claim so a retry can attempt again.
    await idempotencyStore.release(event.eventId);
    throw err;
  }
}

Three states for each event ID:

State	Meaning
Unclaimed	Event has not been seen yet. The first delivery transitions to `claimed`.
Claimed	Event is currently being processed. Subsequent deliveries should wait or skip.
Completed	Event was fully processed including side effects. Subsequent deliveries are no-ops.

This three-state pattern matters because of the concurrent-delivery edge case: two deliveries of the same event arriving in parallel. With only two states (“seen” / “unseen”), both deliveries see “unseen” simultaneously, both process the event, and you get double side effects. With three states, the first delivery atomically transitions unclaimed → claimed, the second sees claimed and skips.

Implementation choices

The idempotency store needs to support an atomic compare-and-set operation. Several options work: Redis with SET ... NX:

JavaScript

async function claimEvent(eventId) {
  const result = await redis.set(`idempotency:${eventId}`, 'claimed', 'NX', 'EX', 86400);
  return { alreadyProcessed: result === null };
}

PostgreSQL with INSERT ... ON CONFLICT DO NOTHING:

INSERT INTO webhook_events (event_id, status, claimed_at)
VALUES ($1, 'claimed', NOW())
ON CONFLICT (event_id) DO NOTHING;

If INSERT affected zero rows, the event was already claimed. A managed queue with deduplication: AWS SQS FIFO queues support content-based deduplication via a MessageDeduplicationId. Pass the Virtuous event ID as the dedup ID; SQS drops duplicate enqueue attempts within the dedup window. Choose based on the durability and consistency requirements of your handler. For most partner integrations, a relational database table is the simplest, most observable option. Redis is faster but requires careful TTL management. Managed queue dedup is convenient but the dedup window is finite (5 minutes on SQS) which is not long enough to defend against late retries.

Retention

Keep idempotency records for at least the maximum retry window. A safe default is 30 days — long enough to outlast the longest realistic retry sequence and to defend against replay attacks within a reasonable window. For very high-volume integrations where storage cost matters, 7 days is also defensible if you have good observability around retry-driven duplicates.

Side-effect ordering

A subtler pitfall: the order in which you mark the event as completed and perform side effects matters. Consider this code:

JavaScript

// ❌ Side effect before completion
await sendThankYouEmail(event.contact);
await idempotencyStore.complete(event.eventId);

If the process crashes between the two lines, the next delivery will re-run both — sending a second email. Reversing the order doesn’t help either:

JavaScript

// ❌ Completion before side effect
await idempotencyStore.complete(event.eventId);
await sendThankYouEmail(event.contact);

Now if the process crashes between the lines, the next delivery sees completed and skips — the email is never sent. The robust pattern is to perform side effects through the same atomic mechanism as the completion record. Two options: Option A: Transactional outbox. Write the side-effect intent to a database table in the same transaction that marks the event complete. A separate worker reads the outbox and actually performs the side effects, marking them done after success.

JavaScript

await db.transaction(async (tx) => {
  await tx.complete(event.eventId);
  await tx.outbox.insert({
    type: 'send_email',
    contactId: event.data.id,
    template: 'thank_you',
  });
});
// A background worker drains the outbox separately, with its own idempotency.

Option B: Idempotency keys on downstream APIs. Many downstream services accept an idempotency key (Stripe, SendGrid, Mailchimp, Twilio). Derive the key from the Virtuous event ID and the side-effect type — ${eventId}-email for the email, ${eventId}-stripe-charge for the charge. The downstream service handles dedup.

JavaScript

await sendgrid.send({
  to: event.data.email,
  templateId: 'thank-you',
  customArgs: { idempotency_key: `${event.eventId}-email` },
});

For partner integrations with multiple side effects per event, the transactional outbox pattern is the most reliable.

Handling out-of-order deliveries

Webhooks can arrive out of chronological order — particularly after a retry sequence. A contact.updated event delivered in the first attempt may arrive after a later contact.updated event that succeeded on its first try. For most integrations, the right defense is to use the timestamp in the event payload (not the time you received it) for ordering decisions. When processing a Contact update, compare the event’s timestamp against the modifiedDateTimeUtc already stored for the Contact in your database:

JavaScript

async function handleContactUpdated(event) {
  const stored = await db.contacts.find({ virtuousId: event.data.id });

  if (stored && new Date(stored.modifiedDateTimeUtc) >= new Date(event.data.modifiedDateTimeUtc)) {
    // The stored record is newer than the event — skip.
    console.info('Skipping stale update', { eventId: event.eventId });
    return;
  }

  await db.contacts.upsert({ virtuousId: event.data.id, ...event.data });
}

This pattern is naturally idempotent on its own — a re-delivery of the same event will see the stored modifiedDateTimeUtc matches and skip the update. But it works only for state-update events, not for side-effect events. Use it alongside the idempotency-key pattern, not as a replacement.

If your handler does anything beyond updating a row in your database — sending an email, posting to a downstream API, charging a card — use both the timestamp check and the idempotency-key store. The timestamp catches stale state-updates; the idempotency key catches duplicate side effects.

Cross-source duplicates

A specifically partner-relevant duplicate scenario: your integration both submits Transactions and subscribes to webhooks. When you submit a Gift Transaction via POST /api/v2/Gift/Transaction, the nightly batch eventually creates the real Gift — and fires a giftCreate webhook back to your endpoint. If your handler also creates a record for the Gift on your side, you can end up with two records of the same gift unless you deduplicate. The defense is the transactionSource / transactionId pair you sent on the original submission. The webhook payload carries the same pair back to you — match against it before creating a new record on your side:

JavaScript

async function handleGiftCreated(event) {
  // If this gift originated from one of our own submissions, we already
  // know about it. Update our record to reflect the canonical Virtuous Gift ID
  // instead of creating a new row.
  if (event.data.transactionSource === 'YourPlatform') {
    await db.gifts.update(
      { transactionId: event.data.transactionId },
      { virtuousGiftId: event.data.id, status: 'confirmed' }
    );
    return;
  }

  // Otherwise, this gift was created by someone else (manual entry, another
  // integration, etc.) — capture it as a new record.
  await db.gifts.insert({ ... });
}

This pattern matters most for Gifts but applies anywhere your integration writes records that may also generate webhook events. See Transactions for the upstream side of this pattern.

Testing idempotency

Build a test fixture that replays the same event twice through your handler. The handler’s observable behavior — database rows, downstream API calls, emails, audit log entries — should be identical to a single delivery:

JavaScript

test('handleGiftCreated is idempotent', async () => {
  const event = loadFixture('gift-created.json');

  await processWebhookEvent(event);
  const stateAfterFirst = await captureState();

  await processWebhookEvent(event);
  const stateAfterSecond = await captureState();

  expect(stateAfterSecond).toEqual(stateAfterFirst);
  expect(mockEmailService.sentCount).toBe(1); // Not 2.
});

Run this test for every event handler. The most common partner-integration bug found this way is the missing idempotency key on a side effect — typically an email or a downstream API call.

Where to go next

Retry Behavior

The retry mechanism that produces most of the duplicate deliveries your idempotency layer needs to handle.

Signature Verification

The replay-protection layer that complements idempotency for security-sensitive scenarios.

Transactions

The cross-source duplicate scenario where your own submissions generate webhooks you also receive.

Local Testing

Test idempotency by replaying captured fixtures through your local handler.

Getting Started

Core Concepts

Workflows

Webhooks

Recipes

Best Practices

Idempotency & Safe Reprocessing

Why duplicates happen

Why ordinary database writes are not enough

The idempotency-key pattern

Implementation choices

Retention

Side-effect ordering

Handling out-of-order deliveries

Cross-source duplicates

Testing idempotency

Where to go next

Retry Behavior

Signature Verification

Transactions

Local Testing

​Why duplicates happen

​Why ordinary database writes are not enough

​The idempotency-key pattern

​Implementation choices

​Retention

​Side-effect ordering

​Handling out-of-order deliveries

​Cross-source duplicates

​Testing idempotency

​Where to go next

Retry Behavior

Signature Verification

Transactions

Local Testing

Why duplicates happen

Why ordinary database writes are not enough

The idempotency-key pattern

Implementation choices

Retention

Side-effect ordering

Handling out-of-order deliveries

Cross-source duplicates

Testing idempotency

Where to go next