> ## Documentation Index
> Fetch the complete documentation index at: https://docs.virtuous.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Sync Architecture Patterns

> Architectural patterns for partner integrations that move data between Raise and external systems — the five patterns, when each fits, and how to combine them for production reliability.

Most partner integrations against Raise are fundamentally about *moving data* — out of Raise into something else, occasionally from something else into Raise, sometimes both. The architecture of that movement determines whether the integration scales, handles failures gracefully, recovers from outages, and produces a reliable picture downstream.

This page covers the five common sync architecture patterns, when each is the right fit, and the trade-offs between them. The audience is partner integration architects designing or auditing a Raise integration's high-level structure.

## The fundamental shape

For most partner integrations, Raise is the **source** and an external system is the **destination**:

```mermaid theme={null}
graph LR
  Raise["Raise"]
  Pipeline["Partner integration<br/>(pipeline)"]
  Dest1["Accounting tool"]
  Dest2["BI platform"]
  Dest3["External CRM"]

  Raise --> Pipeline
  Pipeline --> Dest1
  Pipeline --> Dest2
  Pipeline --> Dest3
```

The customer's fundraising team operates Raise (collecting donations, managing donors, configuring campaigns). The customer's other teams need that data in their respective systems. The partner integration's job is to keep those downstream systems aligned with what's happening in Raise.

The patterns on this page describe different ways to design that pipeline. Most production integrations use a combination of two or three.

***

## Pattern 1: webhook-driven push (the default)

The simplest pattern — Raise pushes events to the partner, the partner writes to the destination.

```mermaid theme={null}
graph LR
  Raise["Raise"]
  Webhook["Partner<br/>webhook receiver"]
  Queue["Sync queue"]
  Worker["Sync worker"]
  Dest["Destination"]

  Raise -->|event| Webhook
  Webhook --> Queue
  Queue --> Worker
  Worker --> Dest
```

The pattern in code:

```javascript JavaScript theme={null}
app.post('/raise-webhooks/:customerId', async (req, res) => {
  const { customerId } = req.params;

  if (!verifySignature(req, customerSettings[customerId].webhookSecret)) {
    return res.status(401).send('Invalid signature');
  }

  res.status(200).send('OK');

  // Queue for async processing
  const event = JSON.parse(req.body.toString('utf8'));
  await syncQueue.publish({ customerId, event });
});

async function processSync(message) {
  const { customerId, event } = message;
  const mapped = mapToDestination(event.payload);
  await destinationApi.upsert(mapped);
}
```

### When this pattern fits

| Scenario                                       | Why this pattern                                             |
| ---------------------------------------------- | ------------------------------------------------------------ |
| Real-time sync is acceptable                   | Webhook events deliver in seconds                            |
| The destination has an upsert API              | Idempotency comes from upsert semantics                      |
| The customer's volume is reasonable            | Webhook fan-out scales linearly with event count             |
| The partner can host a public webhook receiver | HTTPS endpoint, signature verification, queue infrastructure |

For most partner integrations, **this is the default pattern.** Start here unless something specific rules it out.

### When this pattern doesn't fit alone

| Issue                                                           | Implication                                   |
| --------------------------------------------------------------- | --------------------------------------------- |
| Webhook events can be missed                                    | Need a reconciliation backstop (pattern 4)    |
| Initial customer state isn't covered                            | Need a backfill pattern (pattern 5)           |
| Destination has poor idempotency                                | Need dedup layer in the worker                |
| Customer needs historical reporting before integration was live | Backfill required before steady-state matters |

Webhook-driven push alone is rarely a complete solution. It typically pairs with reconciliation (pattern 4) and backfill (pattern 5).

***

## Pattern 2: polled pull

For destinations that can't receive webhooks (legacy systems, on-premise tools, batch-oriented warehouses), the partner integration polls Raise on a schedule:

```mermaid theme={null}
graph LR
  Cron["Scheduled job<br/>(every N minutes)"]
  Raise["Raise"]
  Checkpoint["Sync checkpoint<br/>(last_sync_time)"]
  Worker["Sync worker"]
  Dest["Destination"]

  Cron --> Worker
  Worker -->|query: modifiedDateTime > checkpoint| Raise
  Raise --> Worker
  Worker --> Dest
  Worker -->|advance| Checkpoint
```

The pattern in code:

```javascript JavaScript theme={null}
async function pollSync(customerId) {
  const settings = customerSettings[customerId];
  const lastSync = await checkpointStore.get(`gift_sync:${customerId}`);

  const newGifts = await streamGiftsCursor(settings.raiseApiToken, [
    {
      conditions: [
        { parameter: 'modifieddatetimeutc', operator: GT_OPERATOR, value: lastSync },
      ],
    },
  ]);

  for (const gift of newGifts) {
    await processOneGift(customerId, gift);
  }

  // Advance checkpoint to the latest processed record's timestamp
  const latest = newGifts[newGifts.length - 1]?.modifiedDateTime;
  if (latest) {
    await checkpointStore.set(`gift_sync:${customerId}`, latest);
  }
}

// Run every 15 minutes
setInterval(() => pollSync(customerId), 15 * 60 * 1000);
```

### When polled pull fits

| Scenario                                                            | Why this pattern                                  |
| ------------------------------------------------------------------- | ------------------------------------------------- |
| Destination is a batch-oriented data warehouse                      | Aligns with the destination's natural cadence     |
| Partner can't host a public webhook receiver (firewall, on-premise) | No inbound port needed                            |
| Sync lag of 15–60 minutes is acceptable                             | Polling cadence determines latency                |
| Customer's volume is low enough that polling stays cheap            | A few hundred to a few thousand records per cycle |

### Trade-offs vs. webhook-driven

| Aspect             | Webhook-driven            | Polled pull                                    |
| ------------------ | ------------------------- | ---------------------------------------------- |
| Latency            | Seconds                   | Minutes (poll interval)                        |
| Rate-limit cost    | Near-zero for detection   | Each poll consumes budget                      |
| Missed events      | Possible — needs backstop | Detected at next poll                          |
| Setup complexity   | Subscribe to webhook      | Schedule the cron job                          |
| Deletion detection | Catches delete events     | Hard — deleted records don't appear in queries |

Polled pull misses deletions if the destination doesn't track its own state — a record that exists in the destination but no longer in Raise won't be detected by a "give me everything modified since X" query. For destinations that need delete-handling, use webhooks or pair polling with a periodic full inventory comparison.

### Advancing the checkpoint correctly

A common bug in polled pull: advancing the checkpoint to "now" rather than to the last successful record's timestamp:

```javascript JavaScript theme={null}
// ❌ Anti-pattern: advance to current time
await checkpointStore.set(`gift_sync:${customerId}`, new Date());
// If the sync crashed mid-stream, the next run misses records

// ✅ Advance to the last successfully processed record's timestamp
const latest = newGifts[newGifts.length - 1]?.modifiedDateTime;
if (latest) {
  await checkpointStore.set(`gift_sync:${customerId}`, latest);
}
// Next run resumes exactly where this one left off
```

Always anchor the checkpoint to a record's actual timestamp, not to wall-clock time. The slight overlap on the next poll (re-fetching the boundary record) is fine — idempotency in the worker handles it.

***

## Pattern 3: hybrid (webhook + periodic reconciliation)

The most robust production pattern. Webhooks handle steady-state real-time sync; periodic reconciliation catches anything webhooks missed.

```mermaid theme={null}
graph LR
  Raise["Raise"]
  Webhook["Webhook<br/>receiver"]
  Queue["Sync queue"]
  Worker["Sync worker"]
  Reconciler["Daily<br/>reconciler"]
  Dest["Destination"]

  Raise -->|events| Webhook
  Webhook --> Queue
  Queue --> Worker
  Worker --> Dest

  Reconciler -->|query yesterday| Raise
  Reconciler -->|verify| Dest
  Reconciler -->|gaps re-queued| Queue
```

The reconciler runs daily (or hourly for higher-stakes integrations) and verifies that every Raise record from yesterday made it to the destination. Gaps go back into the queue for re-processing.

The pattern in code:

```javascript JavaScript theme={null}
async function dailyReconcile(customerId, date) {
  // 1. Read all Raise records modified yesterday
  const raiseRecords = await queryRaiseGiftsForDay(customerId, date);

  // 2. Check each against the destination
  const gaps = [];
  for (const record of raiseRecords) {
    const exists = await destinationApi.exists(`raise-${record.id}`);
    if (!exists) gaps.push(record);
  }

  // 3. Re-queue gaps for sync
  for (const record of gaps) {
    await syncQueue.publish({
      customerId,
      event: { payload: record, eventType: 'reconciliation' },
    });
  }

  return { totalRecords: raiseRecords.length, gapsFound: gaps.length };
}
```

### Why hybrid is the production default

| Without reconciliation                             | With reconciliation                           |
| -------------------------------------------------- | --------------------------------------------- |
| One missed webhook = one permanently missed record | Missed records detected within 24 hours       |
| No way to verify the integration is working        | Daily metric proves end-to-end correctness    |
| Customer issues require ad-hoc investigation       | Confidence to answer "did this gift make it?" |

Webhooks are reliable for \~99.5% of events. Reconciliation closes the gap to \~100%. For partner integrations where data accuracy matters (and it almost always does), the hybrid pattern is worth the extra complexity.

### Tuning reconciliation cadence

| Cadence   | When to use                                                  |
| --------- | ------------------------------------------------------------ |
| Daily     | Most partner integrations — catches gaps with 1-day lag      |
| Hourly    | High-stakes integrations where multi-hour gaps matter        |
| Weekly    | Low-stakes integrations or supplementary to other safeguards |
| Real-time | Not feasible — would consume too much rate-limit budget      |

Daily reconciliation is the right default. Move to hourly only if the integration's SLA genuinely requires it.

***

## Pattern 4: backfill + steady-state

For customers with existing data when the integration starts, a backfill pulls historical records before steady-state sync takes over.

```mermaid theme={null}
graph TD
  Start([Integration starts<br/>for customer])
  Backfill["Backfill process:<br/>paginated query<br/>across all history"]
  Cutover["Cutover:<br/>backfill complete"]
  Webhook["Steady-state:<br/>webhook-driven sync"]
  Reconcile["Daily<br/>reconciliation"]

  Start --> Backfill
  Backfill --> Cutover
  Cutover --> Webhook
  Webhook --> Reconcile
  Reconcile --> Webhook
```

The full sequence:

<Steps>
  <Step title="Customer onboarding triggers initial backfill">
    Pull historical records via `POST /api/Gift/query` with pagination.
  </Step>

  <Step title="Subscribe to webhooks (in parallel)">
    Set up the webhook subscription early so events fire to a queue from the start.
  </Step>

  <Step title="Backfill streams records to the same queue as the webhook">
    Both backfill and live events flow through one worker — single processing path.
  </Step>

  <Step title="When backfill completes, mark the customer as live">
    The queue continues to drain, now fed only by webhooks.
  </Step>

  <Step title="Daily reconciliation begins after cutover">
    Reconciliation verifies steady-state sync correctness over time.
  </Step>
</Steps>

Critical detail: **both backfill and steady-state events go through the same queue and worker.** Having separate backfill and live paths creates drift in edge-case handling. See [Sync Raise Gifts to an External System: The same queue for backfill and steady-state](/raise/recipes/sync-raise-gifts-to-an-external-system#the-same-queue-for-backfill-and-steady-state).

### When backfill is needed

| Scenario                                                   | Backfill?                                          |
| ---------------------------------------------------------- | -------------------------------------------------- |
| Customer has years of historical data; downstream needs it | Yes — backfill all                                 |
| Customer is brand new to Raise                             | No — steady-state alone is sufficient              |
| Customer only needs forward-looking sync                   | No — set the checkpoint to "now" and skip backfill |
| Customer wants partial history (e.g., last 2 years)        | Backfill with a date filter                        |

### Backfill performance

A customer with 100,000 historical gifts is a substantial backfill — at `Take=1000` with 1-second throttling, that's \~100 minutes of constant API calls. Plan for this:

```javascript JavaScript theme={null}
async function backfillGifts(customerId) {
  let skip = 0;
  const take = 1000;
  let total = null;
  const startTime = Date.now();

  do {
    const page = await fetchGiftsPage(customerId, skip, take);
    if (total === null) total = page.total;

    for (const gift of page.items) {
      await syncQueue.publish({ customerId, event: { payload: gift } });
    }

    skip += take;

    // Log progress every 10 pages
    if (skip % 10000 === 0) {
      const elapsed = (Date.now() - startTime) / 1000;
      const recordsPerSec = skip / elapsed;
      const remaining = (total - skip) / recordsPerSec;
      console.log(
        `Backfill: ${skip}/${total} (${recordsPerSec.toFixed(0)} r/s, ` +
        `~${(remaining / 60).toFixed(0)} min remaining)`
      );
    }

    await sleep(1000); // Throttle between pages
  } while (skip < total);
}
```

For very large backfills, consider running in parallel across multiple workers — each handling a non-overlapping ID range — to reduce wall-clock time.

### Resumable backfill

If the backfill crashes partway through, it should resume from where it left off rather than starting over. Track progress in a checkpoint:

```javascript JavaScript theme={null}
async function backfillResumable(customerId) {
  let lastIdProcessed = await getBackfillCheckpoint(customerId) ?? 0;

  while (true) {
    const page = await queryGifts({
      groups: [
        {
          conditions: [
            { parameter: 'id', operator: GT_OPERATOR, value: lastIdProcessed.toString() },
          ],
        },
      ],
      sortBy: 'id',
      descending: false,
      take: 1000,
    });

    if (page.items.length === 0) break;

    for (const gift of page.items) {
      await syncQueue.publish({ customerId, event: { payload: gift } });
    }

    lastIdProcessed = page.items[page.items.length - 1].id;
    await setBackfillCheckpoint(customerId, lastIdProcessed);

    await sleep(1000);
  }
}
```

Resumability turns a multi-hour backfill from a single fragile operation into one that can fail and restart without losing progress.

***

## Pattern 5: two-way coordination (rare)

When the partner integration writes back to Raise — for example, syncing donor preferences from an external system into Raise — coordination between the two directions becomes important.

```mermaid theme={null}
graph LR
  External["External system"]
  Partner["Partner integration"]
  Raise["Raise"]

  External -->|trigger: donor updated| Partner
  Partner -->|PATCH donor| Raise
  Raise -->|webhook: donor updated| Partner
  Partner -->|decision:<br/>this is our own write,<br/>ignore| External
```

The pattern in code:

```javascript JavaScript theme={null}
async function syncDonorFromExternal(donorId, externalData) {
  // Mark this write as partner-originated
  await writeAttribution.recordIntent({
    type: 'donor_update',
    raiseId: donorId,
    expectedFields: Object.keys(externalData),
    submittedAt: new Date(),
  });

  // Submit the update
  await fetch(`https://prod-api.raisedonors.com/api/Donor/${donorId}`, {
    method: 'PATCH',
    headers: { /* ... */ },
    body: JSON.stringify(externalData),
  });
}

async function handleDonorWebhook(event) {
  const donor = event.payload;

  // Check whether this update was triggered by us
  const recentIntent = await writeAttribution.findRecent(donor.id, 'donor_update');
  if (recentIntent && fieldsMatch(recentIntent.expectedFields, donor)) {
    // This is the echo of our own write — don't re-sync
    return;
  }

  // External update — sync to the external system
  await externalApi.syncDonor(donor);
}
```

### The echo problem

When the partner writes to Raise, Raise fires a webhook back. Without coordination, the partner integration would treat the echo as an external change and try to sync it back to the external system — producing a loop. The "write attribution" pattern records the partner's intent before writing so the echo can be recognized and ignored.

### When two-way sync is needed

| Scenario                                                | Two-way needed?                                          |
| ------------------------------------------------------- | -------------------------------------------------------- |
| Customer uses Raise as the source of truth              | No — one-way out of Raise is enough                      |
| Customer's email platform is canonical for opt-in state | Yes — opt-in changes need to flow into Raise             |
| Customer's CRM holds richer donor data not in Raise     | Often — partner integration writes updates back to Raise |
| Customer's accounting system is canonical for revenue   | No — accounting reads from Raise; doesn't write back     |

Most integrations are **one-way out of Raise.** Two-way is the exception. When you do need it, plan the echo-handling explicitly.

***

## Combining patterns

Real production integrations combine these patterns. The most common combination:

| Phase                       | Pattern                          |
| --------------------------- | -------------------------------- |
| Initial customer onboarding | Backfill (pattern 4)             |
| Ongoing real-time sync      | Webhook-driven push (pattern 1)  |
| Verifying correctness       | Daily reconciliation (pattern 3) |

For destinations that can't receive webhooks:

| Phase                       | Pattern                                              |
| --------------------------- | ---------------------------------------------------- |
| Initial customer onboarding | Backfill (pattern 4)                                 |
| Ongoing sync                | Polled pull (pattern 2)                              |
| Verifying correctness       | Daily reconciliation against the destination's state |

For partner integrations with bidirectional needs:

| Phase                       | Pattern                                                            |
| --------------------------- | ------------------------------------------------------------------ |
| Initial customer onboarding | Backfill in both directions                                        |
| Ongoing sync                | Webhook-driven push (pattern 1) + two-way coordination (pattern 5) |
| Verifying correctness       | Daily reconciliation with attribution-aware comparison             |

The combination matters more than any single pattern. A partner integration with all three (backfill, steady-state, reconciliation) handles 99%+ of customers' production needs.

***

## Destination-specific considerations

The patterns above apply broadly, but specific destination types have particular needs:

### Accounting destinations (QuickBooks, Xero, NetSuite)

| Consideration                                         | Implication                                       |
| ----------------------------------------------------- | ------------------------------------------------- |
| Strong idempotency required                           | Accounting can't tolerate double-recorded revenue |
| Refunds need separate records, not mutations          | Maps to credit notes / credit memos               |
| Hard deletes typically not supported                  | Use void-with-reason instead                      |
| Period close requires sync to be "final" by month-end | Reconciliation timing matters                     |

### BI / data warehouse destinations (Snowflake, BigQuery, Redshift)

| Consideration                                   | Implication                                 |
| ----------------------------------------------- | ------------------------------------------- |
| Flat wide rows preferred over nested structures | Schema mapping flattens nested JSON         |
| Deletes can be soft (set a `deleted_at` flag)   | Preserves historical data for analysis      |
| High latency is acceptable                      | Daily or hourly batch sync often sufficient |
| Volume tends to be substantial                  | ID-cursor iteration becomes important       |

### External CRM destinations (HubSpot, Salesforce, ActiveCampaign)

| Consideration                               | Implication                                |
| ------------------------------------------- | ------------------------------------------ |
| Contact records keyed by email              | Maps to Raise's email-as-primary-key model |
| Two-way sync sometimes needed               | Especially for opt-in state                |
| Custom field mapping varies                 | Document the mapping per-customer          |
| API rate limits often stricter than Raise's | The destination may be the bottleneck      |

### Marketing automation destinations (Mailchimp, Klaviyo, Iterable)

| Consideration                                 | Implication                               |
| --------------------------------------------- | ----------------------------------------- |
| List/audience membership is the primary state | Map donors to list members                |
| Tag-based segmentation is common              | Translate Raise attributes to tags        |
| Deliverability matters                        | Don't sync test-mode donors               |
| GDPR / opt-in compliance is strict            | Honor `donorEmailOptIn` and similar flags |

For each destination type, the partner integration's architect should review the destination's specific constraints and adapt the patterns above to fit.

***

## Operational practices

A few practices that apply to any sync architecture:

### Monitor everything end-to-end

Track the full journey: gift creation in Raise → webhook delivery → queue depth → worker processing → destination write → reconciliation verification. Latency at each step. Failure rate at each step. The end-to-end view catches issues that any single stage's metrics miss.

### Per-customer dashboards

For partner integrations with many customers, build per-customer dashboards that show sync health for each:

* Last successful sync timestamp
* Records synced today / this week / this month
* Open dead-letter entries
* Webhook subscription status
* Recent reconciliation results

A customer asking "is our integration working?" should be answerable in seconds, not hours.

### Customer-facing audit trail

Expose the sync history to the customer's team. They should be able to look up any Raise gift ID and see where it is in the sync pipeline — synced to which destinations, with what destination ID, at what time. Turns "we'll have to investigate" into "I can see exactly what happened."

### Graceful degradation

When a destination is unavailable, the sync should pause for that destination rather than failing the whole pipeline. A workflow that writes to three destinations should continue to write to the two that are healthy when the third is down.

```javascript JavaScript theme={null}
for (const dest of destinations) {
  try {
    await syncToDestination(customerId, dest, payload);
  } catch (err) {
    console.error(`Sync to ${dest} failed:`, err);
    // Continue to next destination
  }
}
```

The customer's BI dashboard being down shouldn't prevent the customer's accounting from being up-to-date.

***

## Choosing an architecture

For a new integration, walk through these questions:

<Steps>
  <Step title="Where does the data flow?">
    Out of Raise (most common), into Raise, or both?
  </Step>

  <Step title="What's the acceptable latency?">
    Seconds → webhook-driven. Minutes → either webhook or polled. Hours → polled is fine.
  </Step>

  <Step title="Can the partner host a public webhook receiver?">
    Yes → webhook-driven preferred. No → polled.
  </Step>

  <Step title="Is there existing data to backfill?">
    Yes → backfill + steady-state pattern.
  </Step>

  <Step title="How critical is data accuracy?">
    High → add reconciliation as a backstop. Low → steady-state alone may suffice.
  </Step>

  <Step title="Are there multiple destinations?">
    Yes → fan-out from one worker to each destination, with independent failure handling.
  </Step>

  <Step title="Is bidirectional sync needed?">
    Yes → plan the echo-handling and write attribution explicitly. No → simpler architecture.
  </Step>
</Steps>

The answers map directly to the patterns. Most integrations land on backfill + webhook-driven push + daily reconciliation — the production-default combination that handles the largest share of real-world needs.

***

## Where to go next

<CardGroup cols={2}>
  <Card title="Sync Raise Gifts to an External System" icon="arrows-rotate" href="/raise/recipes/sync-raise-gifts-to-an-external-system">
    The end-to-end recipe that combines several of these patterns for a real implementation.
  </Card>

  <Card title="Reconcile with CRM+" icon="check-double" href="/raise/workflows/reconcile-with-crm-plus">
    The reconciliation workflow that the hybrid pattern depends on.
  </Card>

  <Card title="Error Recovery Patterns" icon="arrows-rotate" href="/raise/best-practices/error-recovery-patterns">
    The error-handling patterns that the sync worker uses for resilience.
  </Card>

  <Card title="API Performance Tips" icon="gauge" href="/raise/best-practices/api-performance-tips">
    The performance patterns that make sync workloads efficient.
  </Card>
</CardGroup>