Sync Architecture Patterns - Virtuous API Docs

Sync architecture isn’t a one-size-fits-all decision. The right pattern depends on the data freshness requirement, the source platform’s webhook support, the customer’s operational model, and the integration’s scale. This page catalogs the recurring patterns, the criteria for choosing between them, and how to evolve from one to another as requirements change. The audience is a technical lead designing a new integration or auditing an existing one. The Workflows and Recipes pages walk through specific implementations; this page describes the patterns those implementations instantiate.

The architectural building blocks

Most integration architectures are composed from five building blocks:

Block	Role
Source event capture	Webhook receiver, polling worker, or change-data-capture stream that detects changes on the source platform
Outbound queue	Durable storage of pending changes between source capture and Virtuous submission
Submitter	Worker that drains the queue and calls Virtuous write endpoints
Inbound event handler	Webhook receiver and/or polling worker that detects changes on the Virtuous side
State store	Persistent record of sync state, idempotency keys, checkpoints, and dead-letter entries

Every recipe in the docs is a specific assembly of these blocks. The patterns below describe the most common assemblies.

Pattern 1: one-way push (event-driven)

The default pattern for partner integrations that capture data on the source platform and synchronize it into Virtuous.

When to use

The source platform supports webhooks for the events you care about.
The customer wants near-real-time data in Virtuous.
The integration is a payment processor, fundraising platform, ESP, or similar event-producing source.

Implementations

Stripe to Virtuous, Mailchimp to Virtuous, Fundraising Platform to Virtuous, Auction/Event Platform to Virtuous. The foundational reference is Sync External Donations into Virtuous.

Pattern 2: one-way push (polled)

The same destination, different source-event model. Used when the source platform doesn’t support webhooks or supports only a partial event set.

When to use

The source doesn’t support webhooks.
The source has webhooks but for a tier the customer doesn’t subscribe to.
Customer’s freshness requirement is tolerant (hourly or daily acceptable).

Implementations

Build a Nightly Data Sync is the canonical pattern.

The hybrid variant

Constant Contact to Virtuous is a hybrid — webhooks for the events available, polling for the rest. The same outbound queue receives from both capture paths.

Pattern 3: one-way pull (Virtuous → partner)

The reverse direction: a partner platform pulls data from Virtuous for its own consumption.

When to use

The partner platform is a reporting tool, BI integration, or data warehouse.
The customer’s data flows from Virtuous outward (Virtuous is the source of truth).
A reverse-direction integration of a Pattern 1 architecture.

Implementations

The inbound side of Build a Two-Way Sync instantiates this pattern. The reconciler component pattern comes from Reconcile Failed Syncs.

Pattern 4: two-way sync

The composition of Pattern 1 and Pattern 3 — bidirectional sync between two systems.

When to use

The customer maintains data in both systems and expects them to stay consistent.
Changes on either side should propagate to the other.
The customer needs a “single view” across the two platforms.

Implementations

Build a Two-Way Sync is the canonical pattern. The two-way Mailchimp and Constant Contact recipes are specific instances.

The hardest problem

Two-way sync introduces the sync loop: a write on one side triggers a webhook that triggers a write on the other side that triggers a webhook back, ad infinitum. The defense is source identification + per-record sync state — see Build a Two-Way Sync: The hardest problem.

Pattern 5: bulk load + steady-state

The composition of a bulk-load pattern (for initial data backfill) with an ongoing sync pattern (for steady-state operation). Both phases share the same architecture.

When to use

A new customer is onboarding with existing historical data.
The integration needs to import “everything before today” and then start syncing “everything after today.”

Implementations

Import Historical Gifts is the canonical bulk loader, with explicit guidance on how it composes with the steady-state architecture from Sync External Donations. The architectural choice that makes this pattern work: share the queue between bulk and steady-state. The submitter doesn’t distinguish; it drains everything. This means:

The same idempotency logic protects both bulk and steady-state submissions.
The same throttling keeps both within the rate limit.
The same monitoring catches issues in both.

Avoid separate “bulk-only” and “steady-state-only” paths — they tend to drift in their handling of edge cases.

State management patterns

Every sync architecture needs state. Three classes of state, each with its own storage pattern:

Sync state per record

For each entity (Contact, Gift, RecurringGift) you sync, persist:

CREATE TABLE sync_records (
  partner_id TEXT NOT NULL,
  customer_id TEXT NOT NULL,
  virtuous_id INTEGER,
  status TEXT NOT NULL,                       -- 'pending' | 'submitted' | 'confirmed' | 'failed'
  last_synced_at TIMESTAMPTZ,
  partner_modified_at TIMESTAMPTZ,
  virtuous_modified_at TIMESTAMPTZ,
  PRIMARY KEY (customer_id, partner_id)
);

The per-record state enables:

Idempotency (don’t re-submit something already confirmed).
Reconciliation (find records in inconsistent states).
Audit (what state was this record in at time T).

Checkpoint per integration

For incremental sync, track the high-water mark of what’s been processed:

CREATE TABLE sync_checkpoints (
  customer_id TEXT NOT NULL,
  resource_type TEXT NOT NULL,                -- 'contact' | 'gift' | etc.
  direction TEXT NOT NULL,                    -- 'inbound' | 'outbound'
  last_processed_timestamp TIMESTAMPTZ,
  last_processed_id TEXT,
  PRIMARY KEY (customer_id, resource_type, direction)
);

The checkpoint scopes by customer, resource type, and direction — each independent sync has its own progress.

Dead-letter store

For records that exhausted retries — see Error Recovery Patterns for the schema and pattern.

Multi-tenant isolation

Partner integrations serving multiple customers need isolation along several axes:

Axis	Why
Credentials	Each customer has their own Virtuous API token and source-platform credentials.
Queues	Each customer’s queue is independent — a busy customer doesn’t starve others.
Workers	Each customer has their own submitter and reconciler instances (or scheduled slots).
State	Per-record sync state is scoped by `customer_id`.
Rate-limit budget	Each customer has their own 5,000/hour budget against Virtuous — they don’t share.
Alerting	A failure in one customer’s sync alerts on that customer specifically.

The simplest pattern: scope every table and queue by customer_id, and run per-customer worker instances (or per-customer scheduled cron slots). This produces clear blast-radius isolation: any failure in one customer’s sync is contained to that customer. For very small partner integrations (10–20 customers), a single multi-tenant worker that processes one customer at a time is acceptable. For larger scale, per-customer dedicated workers are simpler to operate.

Initial reconciliation patterns

When an integration first connects to a customer, neither side has knowledge of the other’s records. The initial reconciliation establishes mutual awareness.

Pattern A: full bulk-load (one-direction)

For Pattern 1 (one-way push), the initial reconciliation is just the bulk-load pattern — pull every relevant source-side record, queue it, let the submitter process. See Import Historical Gifts.

Pattern B: cross-matching reconciliation (two-way)

For Pattern 4 (two-way sync), more nuanced. Records may already exist on both sides:

Pull all relevant records from both sides

Page through POST /api/Contact/Query (with the right filter) to enumerate Virtuous-side. Enumerate partner-side from your platform’s database.

Cross-match by stable identifiers

For each Virtuous Contact, check its contactReferences[] for your platform’s source. For each partner-side record, check for an existing Virtuous Contact ID. Build the matched pairs.

Resolve unmatched records

For Virtuous-only records, decide: create a partner-side record, or mark Virtuous-side as “external only.” For partner-only records, submit a Contact Transaction.

Establish baseline state

Mark all matched pairs as in_sync in the sync state table. Future events flow through the steady-state architecture.

Initial reconciliation can take hours or days for large customers. Communicate the expected duration upfront and consider running it during off-peak hours.

Evolving between patterns

Most integrations start with one pattern and need to evolve to another as requirements change. Three common evolutions:

Polled → event-driven

If the source platform adds webhook support, migrating from Pattern 2 to Pattern 1 is straightforward because the queue and submitter are unchanged:

Add the webhook receiver

Deploy alongside the existing polling worker. Both feed the same outbound queue.

Validate parity

Run both in parallel for a sustained period (a week or two). Confirm webhook-driven captures produce the same outcomes as poll-driven ones.

Reduce polling cadence

Move polling from primary signal to backstop. Run it less frequently (e.g., daily for reconciliation rather than every 15 minutes for change detection).

Retire if appropriate

Some integrations keep polling permanently as a safety net; others retire it. Depends on the source platform’s webhook reliability track record.

One-way → two-way

Adding inbound sync to a one-way push integration requires careful attention to sync loops:

Subscribe to Virtuous webhooks

Add the contactCreate/contactUpdate/giftCreate event subscriptions.

Add per-record sync state

Migrate to the per-record sync state schema before turning on inbound processing.

Implement source identification

The outbound submitter must mark records it writes with transactionSource / referenceSource so the inbound handler can distinguish “from us” vs. “from elsewhere.”

Roll out inbound processing

Initially process only “from elsewhere” events; ignore “from us” events. Confirm no loops produced.

Add full inbound logic

Extend processing to apply Virtuous changes back to the partner side. Watch for sync loops in production.

The most common failure mode of this evolution is undetected sync loops. Roll out behind a feature flag and monitor request volume on both sides during rollout.

Single-tenant → multi-tenant

A POC integration for one customer often becomes a multi-customer offering. The migration:

Add the customer_id column everywhere

Every record in the state store, every queue entry, every credential record. Initially populate with a single value for the existing customer.

Refactor workers to be customer-aware

Instead of running one global worker, run per-customer workers (or one worker per scheduling slot that processes one customer at a time).

Move credentials to a secrets manager

Per-customer Virtuous tokens and source-platform tokens belong in a secrets manager, not in environment variables.

Onboard the second customer

The first new customer is the hardest. Many bugs only surface with the second customer (hardcoded values that turn out to be customer-specific, shared state that should have been scoped).

Anti-patterns to avoid

A few common anti-patterns that cause partner integrations to fail at scale:

The synchronous-write antipattern

JavaScript

// ❌ Inside a webhook handler from the source platform
app.post('/source-webhook', async (req, res) => {
  const event = req.body;
  await submitToVirtuous(event);                // synchronous Virtuous call
  res.status(200).send('OK');
});

If Virtuous is slow or unavailable, the source platform’s webhook delivery times out. The source platform retries, you process again, and so on. Capture into a queue and acknowledge — don’t make the source platform wait on Virtuous.

The shared global rate-limit antipattern

JavaScript

// ❌ Single global rate limiter
const globalLimiter = new TokenBucket(1500, ...);

async function makeRequest(customerId, url, options) {
  await globalLimiter.acquire();
  return fetch(url, options);
}

The Virtuous rate limit is per-customer-organization, not per-partner. A global limiter under-utilizes the actual budget (you have N × 5,000/hour available, but the limiter only allows 5,000/hour total). Worse, one customer’s burst can starve others. Use per-customer limiters.

The no-state antipattern

JavaScript

// ❌ No state, retry by re-running the entire sync
async function nightlySync() {
  const allContacts = await fetchAllSourceContacts();
  for (const contact of allContacts) {
    await submitContactTransaction(contact);
  }
}

Without state, an interrupted run starts over and re-processes everything. Without idempotency keys, that’s a duplicate-creation event. Add at least a checkpoint and per-record state — see Build a Nightly Data Sync.

The infinite-retry antipattern

JavaScript

// ❌ Retry forever
async function retryForever(fn) {
  while (true) {
    try {
      return await fn();
    } catch (err) {
      await sleep(60000);
    }
  }
}

Some failures are permanent. Infinite retry of a permanent failure consumes resources, generates noise, and hides the underlying problem. Bound retries and surface persistent failures to humans — see Error Recovery Patterns.

Where to go next

Security and Credential Management

The companion practices for handling per-customer credentials safely.

Versioning and Backward Compatibility

How CRM+‘s versioning model affects long-term integration durability.

Sync External Donations into Virtuous

The foundational push-pattern implementation that several patterns on this page build on.

Build a Two-Way Sync

The two-way sync workflow with deeper coverage of the sync-loop defense.

​The architectural building blocks

​Pattern 1: one-way push (event-driven)

​When to use

​Implementations

​Pattern 2: one-way push (polled)

​When to use

​Implementations

​The hybrid variant

​Pattern 3: one-way pull (Virtuous → partner)

​When to use

​Implementations

​Pattern 4: two-way sync

​When to use

​Implementations

​The hardest problem

​Pattern 5: bulk load + steady-state

​When to use

​Implementations

​Sharing the queue

​State management patterns

​Sync state per record

​Checkpoint per integration

​Dead-letter store

​Multi-tenant isolation

​Initial reconciliation patterns

​Pattern A: full bulk-load (one-direction)

​Pattern B: cross-matching reconciliation (two-way)

​Evolving between patterns

​Polled → event-driven

​One-way → two-way

​Single-tenant → multi-tenant

​Anti-patterns to avoid

​The synchronous-write antipattern

​The shared global rate-limit antipattern

​The no-state antipattern

​The infinite-retry antipattern

​Where to go next

Security and Credential Management

Versioning and Backward Compatibility

Sync External Donations into Virtuous

Build a Two-Way Sync

The architectural building blocks

Pattern 1: one-way push (event-driven)

When to use

Implementations

Pattern 2: one-way push (polled)

When to use

Implementations

The hybrid variant

Pattern 3: one-way pull (Virtuous → partner)

When to use

Implementations

Pattern 4: two-way sync

When to use

Implementations

The hardest problem

Pattern 5: bulk load + steady-state

When to use

Implementations

Sharing the queue

State management patterns

Sync state per record

Checkpoint per integration

Dead-letter store

Multi-tenant isolation

Initial reconciliation patterns

Pattern A: full bulk-load (one-direction)

Pattern B: cross-matching reconciliation (two-way)

Evolving between patterns

Polled → event-driven

One-way → two-way

Single-tenant → multi-tenant

Anti-patterns to avoid

The synchronous-write antipattern

The shared global rate-limit antipattern

The no-state antipattern

The infinite-retry antipattern

Where to go next