Build a Nightly Data Sync - Virtuous API Docs

Event-driven sync (the architecture in Sync External Donations into Virtuous and most integration recipes) is the default recommendation for most partner integrations. But event-driven sync depends on the source platform supporting webhooks, the customer’s environment supporting persistent webhook receivers, and the data freshness requirement justifying always-on infrastructure. When any of those don’t hold, nightly sync — a scheduled batch job that pulls changes from the source platform and pushes them to Virtuous — is the right architecture. This recipe covers the full nightly sync pattern: when to choose it, how to structure the job, how to handle interruption and retry, and how to monitor it.

When nightly is the right choice

Signal	Why nightly fits
Source platform doesn’t have webhooks	An event-driven receiver is impossible; periodic polling is the only path.
Customer’s environment can’t host webhook receivers	Some customers can’t run always-on infrastructure; a scheduled job is operationally simpler.
Data freshness requirement is tolerant	If “yesterday’s data, today” is acceptable (typical for reporting, accounting reconciliation, donor analytics), the latency of a nightly run is fine.
Source platform’s API quota is more constrained than Virtuous’s	Polling once nightly consumes less source-side quota than continuous polling or webhook ingestion.
Customer’s operations are already batch-oriented	Monthly accounting close, weekly BI reports — nightly sync aligns naturally with these.

Most importantly: nightly sync is not a worse architecture than event-driven. It’s a different tradeoff. The right pattern is the one that matches the customer’s actual operational needs.

Do not choose nightly sync because event-driven seems hard. Event-driven is the right choice for most integrations because the freshness benefit is substantial for the customer’s day-to-day operations. Choose nightly only when one of the signals above genuinely applies.

Architecture

Five components:

Job scheduler — cron, Kubernetes CronJob, AWS EventBridge, or whatever scheduling primitive your environment provides.
Sync job — a single binary or script that performs the full sync.
Source platform read — pull changes since the last checkpoint.
Virtuous write — apply changes via the appropriate write endpoints.
State store — persistent storage for the checkpoint timestamp and any per-record sync metadata.

A nightly sync is simpler than event-driven because it runs in one place at one time. It’s also more demanding because that one run needs to handle the full hour-long (or longer) window in which something might go wrong.

The job’s structure

A typical nightly sync job has four phases:

JavaScript

async function runNightlySync(customerId) {
  const runId = generateRunId();
  console.log(`Starting nightly sync run ${runId} for customer ${customerId}`);

  try {
    // Phase 1: load checkpoint and confirm prerequisites
    const checkpoint = await loadCheckpoint(customerId);
    const { sourceToken, virtuousToken } = await loadCredentials(customerId);

    // Phase 2: read changes from source
    const sourceChanges = await readSourceChanges(sourceToken, checkpoint.lastSourceTimestamp);
    console.log(`Found ${sourceChanges.length} source changes`);

    // Phase 3: apply changes to Virtuous (with throttling)
    const results = await applyChangesToVirtuous(virtuousToken, sourceChanges);

    // Phase 4: write checkpoint and emit run report
    await persistCheckpoint(customerId, {
      runId,
      completedAt: new Date(),
      lastSourceTimestamp: results.highestSourceTimestamp,
      successCount: results.successes,
      failureCount: results.failures,
    });

    await emitRunReport(customerId, runId, results);
  } catch (err) {
    console.error(`Nightly sync run ${runId} failed:`, err);
    await emitFailureReport(customerId, runId, err);
    throw err;
  }
}

The four phases are independent — phase boundaries are natural retry points if the job is interrupted mid-run.

Phase 1: checkpoint and prerequisites

The checkpoint is the heart of incremental nightly sync. It tracks where the last successful run stopped so the current run knows where to start.

CREATE TABLE nightly_sync_checkpoints (
  customer_id TEXT PRIMARY KEY,
  last_run_id TEXT,
  last_run_completed_at TIMESTAMPTZ,
  last_source_timestamp TIMESTAMPTZ,         -- the highest source-side timestamp processed
  last_virtuous_timestamp TIMESTAMPTZ,        -- if doing bidirectional, the highest Virtuous timestamp
  consecutive_failures INTEGER NOT NULL DEFAULT 0,
  paused_until TIMESTAMPTZ                    -- circuit-breaker: skip runs while paused
);

Two things the checkpoint stores:

The highest source timestamp processed. Use this as the floor for the next run’s source query — pull everything modified after this value.
Failure metadata. A consecutive_failures counter and a paused_until field implement a simple circuit breaker: after N failed runs, pause the schedule until manual intervention.

Why timestamps, not “since last run started”

The checkpoint should track the highest source-side modification timestamp processed, not the wall-clock time when the last run started. The latter would miss any source-side changes that happened during the run itself. The former guarantees a record modified during the run will be picked up by the next run.

JavaScript

async function readSourceChanges(sourceToken, sinceTimestamp) {
  const changes = [];
  let cursor = null;

  do {
    const url = buildPageUrl(sinceTimestamp, cursor);
    const response = await fetch(url, { headers: { Authorization: `Bearer ${sourceToken}` } });
    const page = await response.json();
    changes.push(...page.records);
    cursor = page.nextCursor;
  } while (cursor);

  return changes;
}

The source platform’s pagination shape varies — some use cursors, some use page numbers, some use since + until ranges. Adapt the loop to the source’s API.

Confirming prerequisites

Before doing any read or write, validate that the conditions for a successful run hold:

JavaScript

async function confirmPrerequisites(customerId) {
  // 1. Source credentials valid
  const sourceToken = await loadSourceToken(customerId);
  if (!sourceToken || isExpired(sourceToken)) {
    throw new Error('Source credentials missing or expired');
  }

  // 2. Virtuous credentials valid
  const virtuousToken = await loadVirtuousToken(customerId);
  if (!virtuousToken || isExpired(virtuousToken)) {
    throw new Error('Virtuous credentials missing or expired');
  }

  // 3. Circuit breaker check
  const checkpoint = await loadCheckpoint(customerId);
  if (checkpoint.paused_until && new Date(checkpoint.paused_until) > new Date()) {
    throw new Error(`Sync paused until ${checkpoint.paused_until}`);
  }

  // 4. Virtuous reachable
  const test = await fetch('https://api.virtuoussoftware.com/api/Health', {
    headers: { Authorization: `Bearer ${virtuousToken}` },
  });
  if (!test.ok) {
    throw new Error('Virtuous API not reachable');
  }
}

Early failure here saves the cost of partial runs.

Phase 2: read changes from source

The source-read pattern depends on the source platform’s API. Three common shapes:

Source API style	Pattern
Modification-timestamp filter	Query for records with `modified_after > checkpoint`. The most common case.
Cursor-based “changes since”	Pass a cursor from the last run; the API returns everything new.
Full snapshot + diff	Pull every record, diff against your local copy to find changes. Used when the source doesn’t expose modification timestamps.

For most modern APIs, the modification-timestamp filter is what’s available. The cursor pattern is more efficient when the API supports it. Full snapshot is the fallback when neither is available — it’s expensive but works.

The full snapshot fallback

JavaScript

async function readChangesViaFullSnapshot(customerId) {
  const sourceToken = await loadSourceToken(customerId);
  const currentSnapshot = await fetchFullSourceSnapshot(sourceToken);
  const previousSnapshot = await loadPreviousSnapshot(customerId);

  const changes = [];
  const previousById = new Map(previousSnapshot.map((r) => [r.id, r]));

  for (const current of currentSnapshot) {
    const previous = previousById.get(current.id);
    if (!previous || diffRecord(current, previous)) {
      changes.push({ type: previous ? 'update' : 'create', record: current });
    }
    previousById.delete(current.id);
  }

  // Anything left in previousById is a deletion
  for (const deleted of previousById.values()) {
    changes.push({ type: 'delete', record: deleted });
  }

  // Save the current snapshot for next run
  await persistSnapshot(customerId, currentSnapshot);

  return changes;
}

The cost is the storage of the previous snapshot and the comparison time. For sources with tens of thousands of records, this is acceptable nightly; for millions, it’s not.

Phase 3: apply changes to Virtuous

The write phase submits each source change as the appropriate Virtuous operation. Three patterns matter at batch scale:

Pattern 1: throttle to stay within rate limits

The Virtuous rate limit is 1,500 requests per hour per organization — see Rate Limits. For a nightly sync with thousands of changes, throttle the submission rate:

JavaScript

async function applyChangesWithThrottling(virtuousToken, changes) {
  const REQUESTS_PER_HOUR = 1200;             // Conservative — leave 20% headroom
  const MS_BETWEEN_REQUESTS = (60 * 60 * 1000) / REQUESTS_PER_HOUR;

  let lastRequestAt = 0;
  const results = { successes: 0, failures: 0, failureDetails: [] };

  for (const change of changes) {
    // Pace the requests
    const elapsed = Date.now() - lastRequestAt;
    if (elapsed < MS_BETWEEN_REQUESTS) {
      await sleep(MS_BETWEEN_REQUESTS - elapsed);
    }
    lastRequestAt = Date.now();

    try {
      await applyChange(virtuousToken, change);
      results.successes++;
    } catch (err) {
      if (isRetryable(err)) {
        // Re-queue for the next run
        await persistDeferredChange(change, err);
      } else {
        results.failures++;
        results.failureDetails.push({ change, error: err.message });
      }
    }
  }

  return results;
}

At 1,200 requests/hour (20% headroom), a job processes 1,200 changes per hour. A 10,000-change run takes roughly 8.3 hours — typically fitting in the overnight window. For larger workloads, raise the throttle closer to the limit (1,400/hour leaves 7% headroom). Don’t run at the cap — a single rate-limited request stops the run mid-stream until the limit resets.

Pattern 2: batch where possible

Some Virtuous endpoints accept multiple records in a single request. The most relevant for sync workloads:

POST /api/Tag/Bulk for tag application across multiple Contacts.
POST /api/ContactNote/Bulk for note creation.

Single-record endpoints (POST /api/Contact/Transaction, POST /api/v2/Gift/Transaction) are the more common case and don’t support batching.

Pattern 3: separate retryable from permanent failures

Just like the event-driven submitter (see Sync External Donations):

Retryable (5xx, 429, network error): re-queue for the next nightly run.
Permanent (400, 422): log and surface for human investigation. Do not retry on the next run.

The difference from event-driven sync is the retry cadence — nightly retries are 24 hours apart, not minutes apart. For genuinely transient failures this is usually fine; for failures that look transient but are actually permanent (a misconfigured field that produces 422 every time), the slower cadence makes the misdiagnosis cheaper.

Phase 4: persist checkpoint and emit report

The checkpoint update commits the run’s progress. If anything fails after the writes succeed but before the checkpoint is updated, the next run will re-process the same changes — your idempotency layer needs to handle this (see Idempotency and Safe Reprocessing).

JavaScript

async function persistCheckpoint(customerId, runMetadata) {
  await db.nightly_sync_checkpoints.upsert({
    customer_id: customerId,
    last_run_id: runMetadata.runId,
    last_run_completed_at: runMetadata.completedAt,
    last_source_timestamp: runMetadata.lastSourceTimestamp,
    consecutive_failures: 0,                 // reset on success
  });
}

The run report is operational visibility — a record of what happened that an on-call human can read:

JavaScript

async function emitRunReport(customerId, runId, results) {
  const report = {
    customer_id: customerId,
    run_id: runId,
    completed_at: new Date(),
    total_changes: results.successes + results.failures,
    successes: results.successes,
    failures: results.failures,
    duration_seconds: results.durationSeconds,
    rate_limit_pauses: results.rateLimitPauseCount,
    failure_details: results.failureDetails.slice(0, 50), // truncate for log readability
  };

  await db.nightly_sync_runs.insert(report);

  if (results.failures > results.successes * 0.05) {
    // Failure rate above 5% — alert
    await alertOps(`Nightly sync for ${customerId} had ${results.failures} failures`);
  }
}

Handling interruption

Nightly jobs are vulnerable to interruption: the scheduler kills the job after a timeout, the host machine restarts, the network drops mid-run. Make the job resumable. The pattern: persist progress within the job, not just at the end:

JavaScript

async function applyChangesResumable(virtuousToken, changes, runId) {
  // Check if a previous attempt for this run exists
  const prevProgress = await db.partial_run_progress.find({ run_id: runId });
  const startIndex = prevProgress?.last_completed_index ?? 0;

  for (let i = startIndex; i < changes.length; i++) {
    await applyChange(virtuousToken, changes[i]);

    // Persist progress every N records
    if (i % 100 === 0) {
      await db.partial_run_progress.upsert({
        run_id: runId,
        last_completed_index: i,
        last_persisted_at: new Date(),
      });
    }
  }

  // Clean up partial-progress record after successful completion
  await db.partial_run_progress.delete({ run_id: runId });
}

A killed job restarts and resumes from the last persisted index rather than starting over. For very long-running jobs, persist progress more frequently. The tradeoff: more frequent persistence means lower replay cost after interruption but higher steady-state I/O. Every 100 records (or every 30 seconds) is a reasonable default.

Circuit breaker

If the sync fails repeatedly, the wrong response is to keep retrying every night — that just produces more failure noise. Build a circuit breaker:

JavaScript

async function applyCircuitBreaker(customerId, runFailed) {
  const checkpoint = await loadCheckpoint(customerId);

  if (runFailed) {
    const newFailureCount = checkpoint.consecutive_failures + 1;

    if (newFailureCount >= 3) {
      // Pause for 24 hours
      const pauseUntil = new Date(Date.now() + 24 * 60 * 60 * 1000);
      await db.nightly_sync_checkpoints.update({ customer_id: customerId }, {
        consecutive_failures: newFailureCount,
        paused_until: pauseUntil,
      });
      await alertOps(`Sync for ${customerId} paused after ${newFailureCount} failures`);
    } else {
      await db.nightly_sync_checkpoints.update({ customer_id: customerId }, {
        consecutive_failures: newFailureCount,
      });
    }
  } else {
    // Reset on success
    await db.nightly_sync_checkpoints.update({ customer_id: customerId }, {
      consecutive_failures: 0,
      paused_until: null,
    });
  }
}

After three consecutive failures, the sync pauses for 24 hours. An ops human must investigate, fix the root cause, and manually clear paused_until to resume. This prevents “sync has been failing for three weeks but nobody noticed” scenarios.

Multi-tenant scheduling

For partner integrations serving many customers, run a separate scheduled job per customer. The patterns to follow:

Stagger start times. Don’t run all customers’ syncs at midnight; spread them across the overnight window. This isolates rate-limit budgets and keeps any single Virtuous account from being hammered by your infrastructure.
Per-customer state. The checkpoint, credentials, and run report are scoped by customer_id.
Per-customer credentials. Each customer has their own Virtuous API token and their own source-platform credentials, loaded from secrets manager.
Per-customer alerts. A failure in one customer’s sync should alert ops about that customer specifically, not as part of a generic “sync failed” notification.

A typical setup: a cron expression that fires once per hour, each invocation processing the customers whose scheduled time slot has arrived. This naturally staggers the load.

Combining nightly with event-driven

A common hybrid pattern: event-driven sync for resources with webhook support, nightly sync for resources without. For example:

Event-driven: gifts (from Stripe), contacts (from Stripe), webhook updates from Virtuous.
Nightly: marketing platform subscriber sync (no webhooks), data warehouse export, accounting reconciliation.

The two pipelines are independent — different schedules, different code paths, different alerting. Just make sure they share idempotency keys for any resource they both touch, so a nightly run that overlaps with an event-driven write doesn’t produce duplicates.

Monitoring

Track these metrics on a nightly sync:

Metric	Healthy value	Investigate when
Run completion	100% success rate over a week	Any failed runs
Duration	Stable, well within the scheduled window	Sustained increases
Records processed	Trends with the customer’s activity	Sudden spikes or drops
Failure rate within run	< 1% of records	Above 5%
Rate-limit pause count	0 (running below the throttle limit)	Any non-zero — the throttle is too aggressive
Checkpoint age	< 24 hours	Stale checkpoints indicate stuck or paused syncs

Most nightly sync issues show up first as a duration regression — the job is doing more work than expected and starts spilling out of its window. The second-most-common issue is checkpoint staleness, which a simple “has the sync run successfully in the last 24 hours?” check catches quickly.

Production readiness checklist

Checkpoint persisted on every successful run, including the highest source timestamp processed.
Source read uses an incremental filter (modification timestamp or cursor), not full snapshots, unless the source API requires it.
Virtuous writes throttled below the 1,500/hour rate limit (1,200/hour or lower recommended).
Retryable vs. permanent failures are distinguished — retryable changes are re-queued for the next run.
The run is resumable: progress persisted within the job so an interrupted run continues where it stopped.
Circuit breaker pauses the sync after consecutive failures and alerts ops.
Multi-tenant: per-customer state, credentials, and run reports.
Run reports persisted and inspectable by ops.
Monitoring alerts on missed runs, elevated failure rates, and rate-limit pauses.
Idempotency layer ensures duplicate records aren’t created if a run partially completes and re-runs.

Where to go next

Sync External Donations into Virtuous

The event-driven alternative — preferred when the source platform supports webhooks and freshness matters.

Constant Contact to Virtuous CRM

A real-world hybrid (some events via webhook, some via polling) that uses pieces of this nightly pattern.

Reconcile Failed Syncs

The reconciliation pattern complements both nightly and event-driven sync as a safety net.

Rate Limits

The constraint that drives the throttling pattern in Phase 3.

​When nightly is the right choice

​Architecture

​The job’s structure

​Phase 1: checkpoint and prerequisites

​Why timestamps, not “since last run started”

​Confirming prerequisites

​Phase 2: read changes from source

​The full snapshot fallback

​Phase 3: apply changes to Virtuous

​Pattern 1: throttle to stay within rate limits

​Pattern 2: batch where possible

​Pattern 3: separate retryable from permanent failures

​Phase 4: persist checkpoint and emit report

​Handling interruption

​Circuit breaker

​Multi-tenant scheduling

​Combining nightly with event-driven

​Monitoring

​Production readiness checklist

​Where to go next

Sync External Donations into Virtuous

Constant Contact to Virtuous CRM

Reconcile Failed Syncs

Rate Limits

When nightly is the right choice

Architecture

The job’s structure

Phase 1: checkpoint and prerequisites

Why timestamps, not “since last run started”

Confirming prerequisites

Phase 2: read changes from source

The full snapshot fallback

Phase 3: apply changes to Virtuous

Pattern 1: throttle to stay within rate limits

Pattern 2: batch where possible

Pattern 3: separate retryable from permanent failures

Phase 4: persist checkpoint and emit report

Handling interruption

Circuit breaker

Multi-tenant scheduling

Combining nightly with event-driven

Monitoring

Production readiness checklist

Where to go next