> ## Documentation Index
> Fetch the complete documentation index at: https://docs.virtuous.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Sync Architecture Patterns

> The recurring architectural patterns for partner integrations syncing data with CRM+ — when to use each, how they compose, and how to evolve between them as requirements change.

Sync architecture isn't a one-size-fits-all decision. The right pattern depends on the data freshness requirement, the source platform's webhook support, the customer's operational model, and the integration's scale. This page catalogs the recurring patterns, the criteria for choosing between them, and how to evolve from one to another as requirements change.

The audience is a technical lead designing a new integration or auditing an existing one. The Workflows and Recipes pages walk through specific implementations; this page describes the patterns those implementations instantiate.

## The architectural building blocks

Most integration architectures are composed from five building blocks:

| Block                     | Role                                                                                                        |
| ------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **Source event capture**  | Webhook receiver, polling worker, or change-data-capture stream that detects changes on the source platform |
| **Outbound queue**        | Durable storage of pending changes between source capture and Virtuous submission                           |
| **Submitter**             | Worker that drains the queue and calls Virtuous write endpoints                                             |
| **Inbound event handler** | Webhook receiver and/or polling worker that detects changes on the Virtuous side                            |
| **State store**           | Persistent record of sync state, idempotency keys, checkpoints, and dead-letter entries                     |

Every recipe in the docs is a specific assembly of these blocks. The patterns below describe the most common assemblies.

***

## Pattern 1: one-way push (event-driven)

The default pattern for partner integrations that capture data on the source platform and synchronize it into Virtuous.

```mermaid theme={null}
graph LR
  Source["Source Platform"]
  Capture["Source Event<br/>Capture"]
  Queue["Outbound Queue"]
  Submitter["Submitter"]
  Virtuous["Virtuous CRM+"]
  State["State Store"]

  Source -->|webhook events| Capture
  Capture --> Queue
  Queue --> Submitter
  Submitter -->|writes| Virtuous
  Submitter --> State
  State -.->|checkpoint, dead-letter| Submitter
```

### When to use

* The source platform supports webhooks for the events you care about.
* The customer wants near-real-time data in Virtuous.
* The integration is a payment processor, fundraising platform, ESP, or similar event-producing source.

### Implementations

[Stripe to Virtuous](/crm/recipes/stripe-to-virtuous), [Mailchimp to Virtuous](/crm/recipes/mailchimp-to-virtuous), [Fundraising Platform to Virtuous](/crm/recipes/fundraising-platform-to-virtuous), [Auction/Event Platform to Virtuous](/crm/recipes/auction-event-platform-to-virtuous).

The foundational reference is [Sync External Donations into Virtuous](/crm/workflows/sync-external-donations).

***

## Pattern 2: one-way push (polled)

The same destination, different source-event model. Used when the source platform doesn't support webhooks or supports only a partial event set.

```mermaid theme={null}
graph LR
  Source["Source Platform"]
  Poller["Polling Worker<br/>(scheduled)"]
  Queue["Outbound Queue"]
  Submitter["Submitter"]
  Virtuous["Virtuous CRM+"]
  Checkpoint["Sync Checkpoint"]

  Poller -->|read since checkpoint| Source
  Poller --> Queue
  Queue --> Submitter
  Submitter -->|writes| Virtuous
  Poller --> Checkpoint
```

### When to use

* The source doesn't support webhooks.
* The source has webhooks but for a tier the customer doesn't subscribe to.
* Customer's freshness requirement is tolerant (hourly or daily acceptable).

### Implementations

[Build a Nightly Data Sync](/crm/recipes/build-a-nightly-data-sync) is the canonical pattern.

### The hybrid variant

[Constant Contact to Virtuous](/crm/recipes/constant-contact-to-virtuous) is a hybrid — webhooks for the events available, polling for the rest. The same outbound queue receives from both capture paths.

***

## Pattern 3: one-way pull (Virtuous → partner)

The reverse direction: a partner platform pulls data from Virtuous for its own consumption.

```mermaid theme={null}
graph LR
  Virtuous["Virtuous CRM+"]
  VWebhook["Virtuous Webhook<br/>Receiver"]
  Inbound["Inbound Processor"]
  Partner["Partner Platform"]
  Reconciler["Periodic Reconciler"]

  Virtuous -->|webhook events| VWebhook
  VWebhook --> Inbound
  Inbound -->|writes| Partner
  Reconciler -.->|Query backstop| Virtuous
  Reconciler --> Partner
```

### When to use

* The partner platform is a reporting tool, BI integration, or data warehouse.
* The customer's data flows from Virtuous outward (Virtuous is the source of truth).
* A reverse-direction integration of a Pattern 1 architecture.

### Implementations

The inbound side of [Build a Two-Way Sync](/crm/workflows/build-a-two-way-sync) instantiates this pattern. The reconciler component pattern comes from [Reconcile Failed Syncs](/crm/workflows/reconcile-failed-syncs).

***

## Pattern 4: two-way sync

The composition of Pattern 1 and Pattern 3 — bidirectional sync between two systems.

```mermaid theme={null}
graph LR
  Source["Source Platform"]
  SCapture["Source Capture"]
  OutQueue["Outbound Queue"]
  Submitter["Submitter"]
  Virtuous["Virtuous CRM+"]
  VCapture["Virtuous Capture"]
  InQueue["Inbound Queue"]
  Inbound["Inbound Processor"]
  State["Sync State<br/>(per-record)"]

  Source --> SCapture
  SCapture --> OutQueue
  OutQueue --> Submitter
  Submitter --> Virtuous
  Virtuous --> VCapture
  VCapture --> InQueue
  InQueue --> Inbound
  Inbound --> Source
  Submitter <-.-> State
  Inbound <-.-> State
```

### When to use

* The customer maintains data in both systems and expects them to stay consistent.
* Changes on either side should propagate to the other.
* The customer needs a "single view" across the two platforms.

### Implementations

[Build a Two-Way Sync](/crm/workflows/build-a-two-way-sync) is the canonical pattern. The two-way Mailchimp and Constant Contact recipes are specific instances.

### The hardest problem

Two-way sync introduces the **sync loop**: a write on one side triggers a webhook that triggers a write on the other side that triggers a webhook back, ad infinitum. The defense is source identification + per-record sync state — see [Build a Two-Way Sync: The hardest problem](/crm/workflows/build-a-two-way-sync#the-hardest-problem-sync-loops).

***

## Pattern 5: bulk load + steady-state

The composition of a bulk-load pattern (for initial data backfill) with an ongoing sync pattern (for steady-state operation). Both phases share the same architecture.

```mermaid theme={null}
graph LR
  Source["Source Platform"]
  Historical["Bulk Loader<br/>(one-time)"]
  EventCapture["Event Capture<br/>(steady-state)"]
  Queue["Shared Outbound Queue"]
  Submitter["Shared Submitter"]
  Virtuous["Virtuous CRM+"]

  Source --> Historical
  Source --> EventCapture
  Historical -->|backfill| Queue
  EventCapture -->|live events| Queue
  Queue --> Submitter
  Submitter --> Virtuous
```

### When to use

* A new customer is onboarding with existing historical data.
* The integration needs to import "everything before today" and then start syncing "everything after today."

### Implementations

[Import Historical Gifts](/crm/recipes/import-historical-gifts) is the canonical bulk loader, with explicit guidance on how it composes with the steady-state architecture from [Sync External Donations](/crm/workflows/sync-external-donations).

### Sharing the queue

The architectural choice that makes this pattern work: **share the queue between bulk and steady-state**. The submitter doesn't distinguish; it drains everything. This means:

* The same idempotency logic protects both bulk and steady-state submissions.
* The same throttling keeps both within the rate limit.
* The same monitoring catches issues in both.

Avoid separate "bulk-only" and "steady-state-only" paths — they tend to drift in their handling of edge cases.

***

## State management patterns

Every sync architecture needs state. Three classes of state, each with its own storage pattern:

### Sync state per record

For each entity (Contact, Gift, RecurringGift) you sync, persist:

```sql theme={null}
CREATE TABLE sync_records (
  partner_id TEXT NOT NULL,
  customer_id TEXT NOT NULL,
  virtuous_id INTEGER,
  status TEXT NOT NULL,                       -- 'pending' | 'submitted' | 'confirmed' | 'failed'
  last_synced_at TIMESTAMPTZ,
  partner_modified_at TIMESTAMPTZ,
  virtuous_modified_at TIMESTAMPTZ,
  PRIMARY KEY (customer_id, partner_id)
);
```

The per-record state enables:

* Idempotency (don't re-submit something already confirmed).
* Reconciliation (find records in inconsistent states).
* Audit (what state was this record in at time T).

### Checkpoint per integration

For incremental sync, track the high-water mark of what's been processed:

```sql theme={null}
CREATE TABLE sync_checkpoints (
  customer_id TEXT NOT NULL,
  resource_type TEXT NOT NULL,                -- 'contact' | 'gift' | etc.
  direction TEXT NOT NULL,                    -- 'inbound' | 'outbound'
  last_processed_timestamp TIMESTAMPTZ,
  last_processed_id TEXT,
  PRIMARY KEY (customer_id, resource_type, direction)
);
```

The checkpoint scopes by customer, resource type, and direction — each independent sync has its own progress.

### Dead-letter store

For records that exhausted retries — see [Error Recovery Patterns](/crm/best-practices/error-recovery-patterns#dead-letter-queue-for-permanent-failures) for the schema and pattern.

***

## Multi-tenant isolation

Partner integrations serving multiple customers need isolation along several axes:

| Axis                  | Why                                                                                  |
| --------------------- | ------------------------------------------------------------------------------------ |
| **Credentials**       | Each customer has their own Virtuous API token and source-platform credentials.      |
| **Queues**            | Each customer's queue is independent — a busy customer doesn't starve others.        |
| **Workers**           | Each customer has their own submitter and reconciler instances (or scheduled slots). |
| **State**             | Per-record sync state is scoped by `customer_id`.                                    |
| **Rate-limit budget** | Each customer has their own 5,000/hour budget against Virtuous — they don't share.   |
| **Alerting**          | A failure in one customer's sync alerts on that customer specifically.               |

The simplest pattern: scope every table and queue by `customer_id`, and run per-customer worker instances (or per-customer scheduled cron slots). This produces clear blast-radius isolation: any failure in one customer's sync is contained to that customer.

For very small partner integrations (10–20 customers), a single multi-tenant worker that processes one customer at a time is acceptable. For larger scale, per-customer dedicated workers are simpler to operate.

***

## Initial reconciliation patterns

When an integration first connects to a customer, neither side has knowledge of the other's records. The initial reconciliation establishes mutual awareness.

### Pattern A: full bulk-load (one-direction)

For Pattern 1 (one-way push), the initial reconciliation is just the bulk-load pattern — pull every relevant source-side record, queue it, let the submitter process. See [Import Historical Gifts](/crm/recipes/import-historical-gifts).

### Pattern B: cross-matching reconciliation (two-way)

For Pattern 4 (two-way sync), more nuanced. Records may already exist on both sides:

<Steps>
  <Step title="Pull all relevant records from both sides">
    Page through `POST /api/Contact/Query` (with the right filter) to enumerate Virtuous-side. Enumerate partner-side from your platform's database.
  </Step>

  <Step title="Cross-match by stable identifiers">
    For each Virtuous Contact, check its `contactReferences[]` for your platform's source. For each partner-side record, check for an existing Virtuous Contact ID. Build the matched pairs.
  </Step>

  <Step title="Resolve unmatched records">
    For Virtuous-only records, decide: create a partner-side record, or mark Virtuous-side as "external only." For partner-only records, submit a Contact Transaction.
  </Step>

  <Step title="Establish baseline state">
    Mark all matched pairs as `in_sync` in the sync state table. Future events flow through the steady-state architecture.
  </Step>
</Steps>

Initial reconciliation can take hours or days for large customers. Communicate the expected duration upfront and consider running it during off-peak hours.

***

## Evolving between patterns

Most integrations start with one pattern and need to evolve to another as requirements change. Three common evolutions:

### Polled → event-driven

If the source platform adds webhook support, migrating from Pattern 2 to Pattern 1 is straightforward because the queue and submitter are unchanged:

<Steps>
  <Step title="Add the webhook receiver">
    Deploy alongside the existing polling worker. Both feed the same outbound queue.
  </Step>

  <Step title="Validate parity">
    Run both in parallel for a sustained period (a week or two). Confirm webhook-driven captures produce the same outcomes as poll-driven ones.
  </Step>

  <Step title="Reduce polling cadence">
    Move polling from primary signal to backstop. Run it less frequently (e.g., daily for reconciliation rather than every 15 minutes for change detection).
  </Step>

  <Step title="Retire if appropriate">
    Some integrations keep polling permanently as a safety net; others retire it. Depends on the source platform's webhook reliability track record.
  </Step>
</Steps>

### One-way → two-way

Adding inbound sync to a one-way push integration requires careful attention to sync loops:

<Steps>
  <Step title="Subscribe to Virtuous webhooks">
    Add the `contactCreate`/`contactUpdate`/`giftCreate` event subscriptions.
  </Step>

  <Step title="Add per-record sync state">
    Migrate to the per-record sync state schema before turning on inbound processing.
  </Step>

  <Step title="Implement source identification">
    The outbound submitter must mark records it writes with `transactionSource` / `referenceSource` so the inbound handler can distinguish "from us" vs. "from elsewhere."
  </Step>

  <Step title="Roll out inbound processing">
    Initially process only "from elsewhere" events; ignore "from us" events. Confirm no loops produced.
  </Step>

  <Step title="Add full inbound logic">
    Extend processing to apply Virtuous changes back to the partner side. Watch for sync loops in production.
  </Step>
</Steps>

The most common failure mode of this evolution is undetected sync loops. Roll out behind a feature flag and monitor request volume on both sides during rollout.

### Single-tenant → multi-tenant

A POC integration for one customer often becomes a multi-customer offering. The migration:

<Steps>
  <Step title="Add the customer_id column everywhere">
    Every record in the state store, every queue entry, every credential record. Initially populate with a single value for the existing customer.
  </Step>

  <Step title="Refactor workers to be customer-aware">
    Instead of running one global worker, run per-customer workers (or one worker per scheduling slot that processes one customer at a time).
  </Step>

  <Step title="Move credentials to a secrets manager">
    Per-customer Virtuous tokens and source-platform tokens belong in a secrets manager, not in environment variables.
  </Step>

  <Step title="Onboard the second customer">
    The first new customer is the hardest. Many bugs only surface with the second customer (hardcoded values that turn out to be customer-specific, shared state that should have been scoped).
  </Step>
</Steps>

***

## Anti-patterns to avoid

A few common anti-patterns that cause partner integrations to fail at scale:

### The synchronous-write antipattern

```javascript JavaScript theme={null}
// ❌ Inside a webhook handler from the source platform
app.post('/source-webhook', async (req, res) => {
  const event = req.body;
  await submitToVirtuous(event);                // synchronous Virtuous call
  res.status(200).send('OK');
});
```

If Virtuous is slow or unavailable, the source platform's webhook delivery times out. The source platform retries, you process again, and so on. Capture into a queue and acknowledge — don't make the source platform wait on Virtuous.

### The shared global rate-limit antipattern

```javascript JavaScript theme={null}
// ❌ Single global rate limiter
const globalLimiter = new TokenBucket(1500, ...);

async function makeRequest(customerId, url, options) {
  await globalLimiter.acquire();
  return fetch(url, options);
}
```

The Virtuous rate limit is **per-customer-organization**, not per-partner. A global limiter under-utilizes the actual budget (you have N × 5,000/hour available, but the limiter only allows 5,000/hour total). Worse, one customer's burst can starve others. Use per-customer limiters.

### The no-state antipattern

```javascript JavaScript theme={null}
// ❌ No state, retry by re-running the entire sync
async function nightlySync() {
  const allContacts = await fetchAllSourceContacts();
  for (const contact of allContacts) {
    await submitContactTransaction(contact);
  }
}
```

Without state, an interrupted run starts over and re-processes everything. Without idempotency keys, that's a duplicate-creation event. Add at least a checkpoint and per-record state — see [Build a Nightly Data Sync](/crm/recipes/build-a-nightly-data-sync).

### The infinite-retry antipattern

```javascript JavaScript theme={null}
// ❌ Retry forever
async function retryForever(fn) {
  while (true) {
    try {
      return await fn();
    } catch (err) {
      await sleep(60000);
    }
  }
}
```

Some failures are permanent. Infinite retry of a permanent failure consumes resources, generates noise, and hides the underlying problem. Bound retries and surface persistent failures to humans — see [Error Recovery Patterns](/crm/best-practices/error-recovery-patterns).

***

## Where to go next

<CardGroup cols={2}>
  <Card title="Security and Credential Management" icon="lock" href="/crm/best-practices/security-and-credential-management">
    The companion practices for handling per-customer credentials safely.
  </Card>

  <Card title="Versioning and Backward Compatibility" icon="code-branch" href="/crm/best-practices/versioning-and-backward-compatibility">
    How CRM+'s versioning model affects long-term integration durability.
  </Card>

  <Card title="Sync External Donations into Virtuous" icon="arrow-right-arrow-left" href="/crm/workflows/sync-external-donations">
    The foundational push-pattern implementation that several patterns on this page build on.
  </Card>

  <Card title="Build a Two-Way Sync" icon="arrows-rotate" href="/crm/workflows/build-a-two-way-sync">
    The two-way sync workflow with deeper coverage of the sync-loop defense.
  </Card>
</CardGroup>