Sync Architecture Patterns - Virtuous API Docs

Most partner integrations against Raise are fundamentally about moving data — out of Raise into something else, occasionally from something else into Raise, sometimes both. The architecture of that movement determines whether the integration scales, handles failures gracefully, recovers from outages, and produces a reliable picture downstream. This page covers the five common sync architecture patterns, when each is the right fit, and the trade-offs between them. The audience is partner integration architects designing or auditing a Raise integration’s high-level structure.

The fundamental shape

For most partner integrations, Raise is the source and an external system is the destination: The customer’s fundraising team operates Raise (collecting donations, managing donors, configuring campaigns). The customer’s other teams need that data in their respective systems. The partner integration’s job is to keep those downstream systems aligned with what’s happening in Raise. The patterns on this page describe different ways to design that pipeline. Most production integrations use a combination of two or three.

Pattern 1: webhook-driven push (the default)

The simplest pattern — Raise pushes events to the partner, the partner writes to the destination. The pattern in code:

JavaScript

app.post('/raise-webhooks/:customerId', async (req, res) => {
  const { customerId } = req.params;

  if (!verifySignature(req, customerSettings[customerId].webhookSecret)) {
    return res.status(401).send('Invalid signature');
  }

  res.status(200).send('OK');

  // Queue for async processing
  const event = JSON.parse(req.body.toString('utf8'));
  await syncQueue.publish({ customerId, event });
});

async function processSync(message) {
  const { customerId, event } = message;
  const mapped = mapToDestination(event.payload);
  await destinationApi.upsert(mapped);
}

When this pattern fits

Scenario	Why this pattern
Real-time sync is acceptable	Webhook events deliver in seconds
The destination has an upsert API	Idempotency comes from upsert semantics
The customer’s volume is reasonable	Webhook fan-out scales linearly with event count
The partner can host a public webhook receiver	HTTPS endpoint, signature verification, queue infrastructure

For most partner integrations, this is the default pattern. Start here unless something specific rules it out.

When this pattern doesn’t fit alone

Issue	Implication
Webhook events can be missed	Need a reconciliation backstop (pattern 4)
Initial customer state isn’t covered	Need a backfill pattern (pattern 5)
Destination has poor idempotency	Need dedup layer in the worker
Customer needs historical reporting before integration was live	Backfill required before steady-state matters

Webhook-driven push alone is rarely a complete solution. It typically pairs with reconciliation (pattern 4) and backfill (pattern 5).

Pattern 2: polled pull

For destinations that can’t receive webhooks (legacy systems, on-premise tools, batch-oriented warehouses), the partner integration polls Raise on a schedule: The pattern in code:

JavaScript

async function pollSync(customerId) {
  const settings = customerSettings[customerId];
  const lastSync = await checkpointStore.get(`gift_sync:${customerId}`);

  const newGifts = await streamGiftsCursor(settings.raiseApiToken, [
    {
      conditions: [
        { parameter: 'modifieddatetimeutc', operator: GT_OPERATOR, value: lastSync },
      ],
    },
  ]);

  for (const gift of newGifts) {
    await processOneGift(customerId, gift);
  }

  // Advance checkpoint to the latest processed record's timestamp
  const latest = newGifts[newGifts.length - 1]?.modifiedDateTime;
  if (latest) {
    await checkpointStore.set(`gift_sync:${customerId}`, latest);
  }
}

// Run every 15 minutes
setInterval(() => pollSync(customerId), 15 * 60 * 1000);

When polled pull fits

Scenario	Why this pattern
Destination is a batch-oriented data warehouse	Aligns with the destination’s natural cadence
Partner can’t host a public webhook receiver (firewall, on-premise)	No inbound port needed
Sync lag of 15–60 minutes is acceptable	Polling cadence determines latency
Customer’s volume is low enough that polling stays cheap	A few hundred to a few thousand records per cycle

Trade-offs vs. webhook-driven

Aspect	Webhook-driven	Polled pull
Latency	Seconds	Minutes (poll interval)
Rate-limit cost	Near-zero for detection	Each poll consumes budget
Missed events	Possible — needs backstop	Detected at next poll
Setup complexity	Subscribe to webhook	Schedule the cron job
Deletion detection	Catches delete events	Hard — deleted records don’t appear in queries

Polled pull misses deletions if the destination doesn’t track its own state — a record that exists in the destination but no longer in Raise won’t be detected by a “give me everything modified since X” query. For destinations that need delete-handling, use webhooks or pair polling with a periodic full inventory comparison.

Advancing the checkpoint correctly

A common bug in polled pull: advancing the checkpoint to “now” rather than to the last successful record’s timestamp:

JavaScript

// ❌ Anti-pattern: advance to current time
await checkpointStore.set(`gift_sync:${customerId}`, new Date());
// If the sync crashed mid-stream, the next run misses records

// ✅ Advance to the last successfully processed record's timestamp
const latest = newGifts[newGifts.length - 1]?.modifiedDateTime;
if (latest) {
  await checkpointStore.set(`gift_sync:${customerId}`, latest);
}
// Next run resumes exactly where this one left off

Always anchor the checkpoint to a record’s actual timestamp, not to wall-clock time. The slight overlap on the next poll (re-fetching the boundary record) is fine — idempotency in the worker handles it.

Pattern 3: hybrid (webhook + periodic reconciliation)

The most robust production pattern. Webhooks handle steady-state real-time sync; periodic reconciliation catches anything webhooks missed. The reconciler runs daily (or hourly for higher-stakes integrations) and verifies that every Raise record from yesterday made it to the destination. Gaps go back into the queue for re-processing. The pattern in code:

JavaScript

async function dailyReconcile(customerId, date) {
  // 1. Read all Raise records modified yesterday
  const raiseRecords = await queryRaiseGiftsForDay(customerId, date);

  // 2. Check each against the destination
  const gaps = [];
  for (const record of raiseRecords) {
    const exists = await destinationApi.exists(`raise-${record.id}`);
    if (!exists) gaps.push(record);
  }

  // 3. Re-queue gaps for sync
  for (const record of gaps) {
    await syncQueue.publish({
      customerId,
      event: { payload: record, eventType: 'reconciliation' },
    });
  }

  return { totalRecords: raiseRecords.length, gapsFound: gaps.length };
}

Why hybrid is the production default

Without reconciliation	With reconciliation
One missed webhook = one permanently missed record	Missed records detected within 24 hours
No way to verify the integration is working	Daily metric proves end-to-end correctness
Customer issues require ad-hoc investigation	Confidence to answer “did this gift make it?”

Webhooks are reliable for ~99.5% of events. Reconciliation closes the gap to ~100%. For partner integrations where data accuracy matters (and it almost always does), the hybrid pattern is worth the extra complexity.

Tuning reconciliation cadence

Cadence	When to use
Daily	Most partner integrations — catches gaps with 1-day lag
Hourly	High-stakes integrations where multi-hour gaps matter
Weekly	Low-stakes integrations or supplementary to other safeguards
Real-time	Not feasible — would consume too much rate-limit budget

Daily reconciliation is the right default. Move to hourly only if the integration’s SLA genuinely requires it.

Pattern 4: backfill + steady-state

For customers with existing data when the integration starts, a backfill pulls historical records before steady-state sync takes over. The full sequence:

Customer onboarding triggers initial backfill

Pull historical records via POST /api/Gift/query with pagination.

Subscribe to webhooks (in parallel)

Set up the webhook subscription early so events fire to a queue from the start.

Backfill streams records to the same queue as the webhook

Both backfill and live events flow through one worker — single processing path.

When backfill completes, mark the customer as live

The queue continues to drain, now fed only by webhooks.

Daily reconciliation begins after cutover

Reconciliation verifies steady-state sync correctness over time.

Critical detail: both backfill and steady-state events go through the same queue and worker. Having separate backfill and live paths creates drift in edge-case handling. See Sync Raise Gifts to an External System: The same queue for backfill and steady-state.

When backfill is needed

Scenario	Backfill?
Customer has years of historical data; downstream needs it	Yes — backfill all
Customer is brand new to Raise	No — steady-state alone is sufficient
Customer only needs forward-looking sync	No — set the checkpoint to “now” and skip backfill
Customer wants partial history (e.g., last 2 years)	Backfill with a date filter

Backfill performance

A customer with 100,000 historical gifts is a substantial backfill — at Take=1000 with 1-second throttling, that’s ~100 minutes of constant API calls. Plan for this:

JavaScript

async function backfillGifts(customerId) {
  let skip = 0;
  const take = 1000;
  let total = null;
  const startTime = Date.now();

  do {
    const page = await fetchGiftsPage(customerId, skip, take);
    if (total === null) total = page.total;

    for (const gift of page.items) {
      await syncQueue.publish({ customerId, event: { payload: gift } });
    }

    skip += take;

    // Log progress every 10 pages
    if (skip % 10000 === 0) {
      const elapsed = (Date.now() - startTime) / 1000;
      const recordsPerSec = skip / elapsed;
      const remaining = (total - skip) / recordsPerSec;
      console.log(
        `Backfill: ${skip}/${total} (${recordsPerSec.toFixed(0)} r/s, ` +
        `~${(remaining / 60).toFixed(0)} min remaining)`
      );
    }

    await sleep(1000); // Throttle between pages
  } while (skip < total);
}

For very large backfills, consider running in parallel across multiple workers — each handling a non-overlapping ID range — to reduce wall-clock time.

Resumable backfill

If the backfill crashes partway through, it should resume from where it left off rather than starting over. Track progress in a checkpoint:

JavaScript

async function backfillResumable(customerId) {
  let lastIdProcessed = await getBackfillCheckpoint(customerId) ?? 0;

  while (true) {
    const page = await queryGifts({
      groups: [
        {
          conditions: [
            { parameter: 'id', operator: GT_OPERATOR, value: lastIdProcessed.toString() },
          ],
        },
      ],
      sortBy: 'id',
      descending: false,
      take: 1000,
    });

    if (page.items.length === 0) break;

    for (const gift of page.items) {
      await syncQueue.publish({ customerId, event: { payload: gift } });
    }

    lastIdProcessed = page.items[page.items.length - 1].id;
    await setBackfillCheckpoint(customerId, lastIdProcessed);

    await sleep(1000);
  }
}

Resumability turns a multi-hour backfill from a single fragile operation into one that can fail and restart without losing progress.

Pattern 5: two-way coordination (rare)

When the partner integration writes back to Raise — for example, syncing donor preferences from an external system into Raise — coordination between the two directions becomes important. The pattern in code:

JavaScript

async function syncDonorFromExternal(donorId, externalData) {
  // Mark this write as partner-originated
  await writeAttribution.recordIntent({
    type: 'donor_update',
    raiseId: donorId,
    expectedFields: Object.keys(externalData),
    submittedAt: new Date(),
  });

  // Submit the update
  await fetch(`https://prod-api.raisedonors.com/api/Donor/${donorId}`, {
    method: 'PATCH',
    headers: { /* ... */ },
    body: JSON.stringify(externalData),
  });
}

async function handleDonorWebhook(event) {
  const donor = event.payload;

  // Check whether this update was triggered by us
  const recentIntent = await writeAttribution.findRecent(donor.id, 'donor_update');
  if (recentIntent && fieldsMatch(recentIntent.expectedFields, donor)) {
    // This is the echo of our own write — don't re-sync
    return;
  }

  // External update — sync to the external system
  await externalApi.syncDonor(donor);
}

The echo problem

When the partner writes to Raise, Raise fires a webhook back. Without coordination, the partner integration would treat the echo as an external change and try to sync it back to the external system — producing a loop. The “write attribution” pattern records the partner’s intent before writing so the echo can be recognized and ignored.

When two-way sync is needed

Scenario	Two-way needed?
Customer uses Raise as the source of truth	No — one-way out of Raise is enough
Customer’s email platform is canonical for opt-in state	Yes — opt-in changes need to flow into Raise
Customer’s CRM holds richer donor data not in Raise	Often — partner integration writes updates back to Raise
Customer’s accounting system is canonical for revenue	No — accounting reads from Raise; doesn’t write back

Most integrations are one-way out of Raise. Two-way is the exception. When you do need it, plan the echo-handling explicitly.

Combining patterns

Real production integrations combine these patterns. The most common combination:

Phase	Pattern
Initial customer onboarding	Backfill (pattern 4)
Ongoing real-time sync	Webhook-driven push (pattern 1)
Verifying correctness	Daily reconciliation (pattern 3)

For destinations that can’t receive webhooks:

Phase	Pattern
Initial customer onboarding	Backfill (pattern 4)
Ongoing sync	Polled pull (pattern 2)
Verifying correctness	Daily reconciliation against the destination’s state

For partner integrations with bidirectional needs:

Phase	Pattern
Initial customer onboarding	Backfill in both directions
Ongoing sync	Webhook-driven push (pattern 1) + two-way coordination (pattern 5)
Verifying correctness	Daily reconciliation with attribution-aware comparison

The combination matters more than any single pattern. A partner integration with all three (backfill, steady-state, reconciliation) handles 99%+ of customers’ production needs.

Destination-specific considerations

The patterns above apply broadly, but specific destination types have particular needs:

Accounting destinations (QuickBooks, Xero, NetSuite)

Consideration	Implication
Strong idempotency required	Accounting can’t tolerate double-recorded revenue
Refunds need separate records, not mutations	Maps to credit notes / credit memos
Hard deletes typically not supported	Use void-with-reason instead
Period close requires sync to be “final” by month-end	Reconciliation timing matters

BI / data warehouse destinations (Snowflake, BigQuery, Redshift)

Consideration	Implication
Flat wide rows preferred over nested structures	Schema mapping flattens nested JSON
Deletes can be soft (set a `deleted_at` flag)	Preserves historical data for analysis
High latency is acceptable	Daily or hourly batch sync often sufficient
Volume tends to be substantial	ID-cursor iteration becomes important

External CRM destinations (HubSpot, Salesforce, ActiveCampaign)

Consideration	Implication
Contact records keyed by email	Maps to Raise’s email-as-primary-key model
Two-way sync sometimes needed	Especially for opt-in state
Custom field mapping varies	Document the mapping per-customer
API rate limits often stricter than Raise’s	The destination may be the bottleneck

Marketing automation destinations (Mailchimp, Klaviyo, Iterable)

Consideration	Implication
List/audience membership is the primary state	Map donors to list members
Tag-based segmentation is common	Translate Raise attributes to tags
Deliverability matters	Don’t sync test-mode donors
GDPR / opt-in compliance is strict	Honor `donorEmailOptIn` and similar flags

For each destination type, the partner integration’s architect should review the destination’s specific constraints and adapt the patterns above to fit.

Operational practices

A few practices that apply to any sync architecture:

Monitor everything end-to-end

Track the full journey: gift creation in Raise → webhook delivery → queue depth → worker processing → destination write → reconciliation verification. Latency at each step. Failure rate at each step. The end-to-end view catches issues that any single stage’s metrics miss.

Per-customer dashboards

For partner integrations with many customers, build per-customer dashboards that show sync health for each:

Last successful sync timestamp
Records synced today / this week / this month
Open dead-letter entries
Webhook subscription status
Recent reconciliation results

A customer asking “is our integration working?” should be answerable in seconds, not hours.

Customer-facing audit trail

Expose the sync history to the customer’s team. They should be able to look up any Raise gift ID and see where it is in the sync pipeline — synced to which destinations, with what destination ID, at what time. Turns “we’ll have to investigate” into “I can see exactly what happened.”

Graceful degradation

When a destination is unavailable, the sync should pause for that destination rather than failing the whole pipeline. A workflow that writes to three destinations should continue to write to the two that are healthy when the third is down.

JavaScript

for (const dest of destinations) {
  try {
    await syncToDestination(customerId, dest, payload);
  } catch (err) {
    console.error(`Sync to ${dest} failed:`, err);
    // Continue to next destination
  }
}

The customer’s BI dashboard being down shouldn’t prevent the customer’s accounting from being up-to-date.

Choosing an architecture

For a new integration, walk through these questions:

Where does the data flow?

Out of Raise (most common), into Raise, or both?

What's the acceptable latency?

Seconds → webhook-driven. Minutes → either webhook or polled. Hours → polled is fine.

Can the partner host a public webhook receiver?

Yes → webhook-driven preferred. No → polled.

Is there existing data to backfill?

Yes → backfill + steady-state pattern.

How critical is data accuracy?

High → add reconciliation as a backstop. Low → steady-state alone may suffice.

Are there multiple destinations?

Yes → fan-out from one worker to each destination, with independent failure handling.

Is bidirectional sync needed?

Yes → plan the echo-handling and write attribution explicitly. No → simpler architecture.

The answers map directly to the patterns. Most integrations land on backfill + webhook-driven push + daily reconciliation — the production-default combination that handles the largest share of real-world needs.

Where to go next

Sync Raise Gifts to an External System

The end-to-end recipe that combines several of these patterns for a real implementation.

Reconcile with CRM+

The reconciliation workflow that the hybrid pattern depends on.

Error Recovery Patterns

The error-handling patterns that the sync worker uses for resilience.

API Performance Tips

The performance patterns that make sync workloads efficient.

​The fundamental shape

​Pattern 1: webhook-driven push (the default)

​When this pattern fits

​When this pattern doesn’t fit alone

​Pattern 2: polled pull

​When polled pull fits

​Trade-offs vs. webhook-driven

​Advancing the checkpoint correctly

​Pattern 3: hybrid (webhook + periodic reconciliation)

​Why hybrid is the production default

​Tuning reconciliation cadence

​Pattern 4: backfill + steady-state

​When backfill is needed

​Backfill performance

​Resumable backfill

​Pattern 5: two-way coordination (rare)

​The echo problem

​When two-way sync is needed

​Combining patterns

​Destination-specific considerations

​Accounting destinations (QuickBooks, Xero, NetSuite)

​BI / data warehouse destinations (Snowflake, BigQuery, Redshift)

​External CRM destinations (HubSpot, Salesforce, ActiveCampaign)

​Marketing automation destinations (Mailchimp, Klaviyo, Iterable)

​Operational practices

​Monitor everything end-to-end

​Per-customer dashboards

​Customer-facing audit trail

​Graceful degradation

​Choosing an architecture

​Where to go next

Sync Raise Gifts to an External System

Reconcile with CRM+

Error Recovery Patterns

API Performance Tips

The fundamental shape

Pattern 1: webhook-driven push (the default)

When this pattern fits

When this pattern doesn’t fit alone

Pattern 2: polled pull

When polled pull fits

Trade-offs vs. webhook-driven

Advancing the checkpoint correctly

Pattern 3: hybrid (webhook + periodic reconciliation)

Why hybrid is the production default

Tuning reconciliation cadence

Pattern 4: backfill + steady-state

When backfill is needed

Backfill performance

Resumable backfill

Pattern 5: two-way coordination (rare)

The echo problem

When two-way sync is needed

Combining patterns

Destination-specific considerations

Accounting destinations (QuickBooks, Xero, NetSuite)

BI / data warehouse destinations (Snowflake, BigQuery, Redshift)

External CRM destinations (HubSpot, Salesforce, ActiveCampaign)

Marketing automation destinations (Mailchimp, Klaviyo, Iterable)

Operational practices

Monitor everything end-to-end

Per-customer dashboards

Customer-facing audit trail

Graceful degradation

Choosing an architecture

Where to go next