Sync Users to External System - Virtuous API Docs

The single most common Volunteer integration pattern: mirror VOMO Users into an external system — a CRM, a data warehouse, a marketing platform, an HR roster. This recipe walks through the end-to-end implementation: initial backfill of historical data, ongoing polling for incremental changes, periodic reconciliation to catch gaps, and the operational practices that keep the integration production-grade reliable. The recipe combines workflows covered separately in the docs (listing users, upserting users, polling, reconciliation) into a single working integration architecture.

What you’ll build

A partner integration that:

Receives a Bearer token from the customer during onboarding
Performs an initial full backfill of all VOMO Users into the external system
Polls VOMO for ongoing changes and propagates them to the external system
Reconciles daily to catch gaps and deletions
Surfaces sync health visibility to the customer

By the end you’ll have a complete reference architecture and the code to implement it.

When this recipe fits

Scenario	This recipe fits
Sync VOMO users to an external CRM (Salesforce, HubSpot, etc.)	✓
Mirror VOMO users into a data warehouse (Snowflake, BigQuery, etc.)	✓
Feed marketing automation tools with volunteer rosters	✓
Sync to a customer-built reporting database	✓
Bidirectional sync (external CRM is source-of-truth for some fields)	✓ With added coordination — see the bidirectional section
Real-time sub-minute sync requirements	✗ Not feasible without webhooks

Architecture

Six components, each independent — failures in one don’t crash others:

Component	Purpose
Onboarding flow	Captures the customer’s VOMO token; configures the integration
Initial backfill	One-time full read of VOMO users at integration start
Polling worker	Ongoing incremental sync via `updated_after`
Reconciliation worker	Daily/weekly audit catching gaps
Partner state DB	Checkpoints, external-ID mappings, audit log
Dead-letter queue	Failed records for later retry

Step 1: customer onboarding

The onboarding flow handles three things: capture the Bearer token, configure sync targets, trigger the initial backfill.

JavaScript

async function onboardCustomer(customerId, onboardingData) {
  // 1. Validate the token works
  const tokenValid = await testVomoToken(onboardingData.vomoToken);
  if (!tokenValid) {
    throw new ValidationError({ vomoToken: ['Token rejected by VOMO API'] });
  }

  // 2. Store the token securely
  await credentials.setVomoToken(customerId, onboardingData.vomoToken);

  // 3. Store sync configuration
  await db.upsert('sync_configs', {
    customer_id: customerId,
    external_system_url: onboardingData.externalSystemUrl,
    poll_cadence_minutes: onboardingData.pollCadenceMinutes ?? 30,
    enabled: true,
    onboarded_at: new Date(),
  });

  // 4. Initialize the checkpoint to "epoch" — backfill will read everything
  await db.upsert('sync_checkpoints', {
    customer_id: customerId,
    resource: 'user_sync',
    checkpoint: new Date(0),
  });

  // 5. Queue the initial backfill
  await jobQueue.publish({
    type: 'initial_backfill',
    customerId,
    enqueuedAt: new Date(),
  });

  return { ok: true };
}

async function testVomoToken(token) {
  try {
    const response = await fetch(
      'https://api.vomo.org/v1/users?page=1',
      { headers: { Authorization: `Bearer ${token}` } }
    );
    return response.ok;
  } catch {
    return false;
  }
}

Why initialize the checkpoint to epoch

Setting the checkpoint to new Date(0) means the next polling cycle would query “everything updated since 1970” — which is everything. But the backfill runs first; once the backfill completes, the checkpoint advances to the latest-seen timestamp. From that point forward, polling operates incrementally. This approach unifies backfill and steady-state under one mechanism — no separate code paths.

Step 2: initial backfill

The backfill reads all VOMO Users and pushes them to the external system. For a 10,000-user customer, this is ~667 paginated requests at 15 records per page.

JavaScript

async function performInitialBackfill(customerId) {
  const token = await credentials.getVomoToken(customerId);
  const client = new ThrottledVomoClient({ token, requestsPerSecond: 3 });

  await jobLog.start(customerId, 'initial_backfill');

  let url = 'https://api.vomo.org/v1/users';
  let processedCount = 0;
  let latestSeen = new Date(0);
  const failures = [];

  while (url) {
    const response = await client.fetch(url);
    if (!response.ok) {
      await jobLog.fail(customerId, 'initial_backfill', `HTTP ${response.status}`);
      throw new Error(`Backfill failed: ${response.status}`);
    }

    const page = await response.json();

    for (const user of page.data) {
      try {
        await processUserForBackfill(customerId, user);
        processedCount++;

        const u = new Date(user.updated_at);
        if (u > latestSeen) latestSeen = u;
      } catch (err) {
        failures.push({ user, error: err.message });
      }
    }

    if (processedCount % 100 === 0) {
      await jobLog.progress(customerId, 'initial_backfill', {
        processedCount,
        total: page.meta.total,
      });
    }

    url = page.links.next;
  }

  await db.upsert('sync_checkpoints', {
    customer_id: customerId,
    resource: 'user_sync',
    checkpoint: latestSeen,
  });

  if (failures.length > 0) {
    await deadLetterQueue.publishBatch(
      failures.map((f) => ({
        type: 'backfill_failure',
        customerId,
        user: f.user,
        error: f.error,
      }))
    );
  }

  await db.upsert('sync_configs', {
    customer_id: customerId,
    backfill_completed_at: new Date(),
    backfill_processed_count: processedCount,
  });

  await jobLog.complete(customerId, 'initial_backfill', {
    processedCount,
    failureCount: failures.length,
  });

  return { processedCount, failureCount: failures.length };
}

What `processUserForBackfill` does

JavaScript

async function processUserForBackfill(customerId, vomoUser) {
  // 1. Upsert into the external system
  const externalRecord = await externalSystem.upsertUser({
    external_id: `vomo-${vomoUser.id}`,
    email: vomoUser.email,
    first_name: vomoUser.first_name,
    last_name: vomoUser.last_name,
    phone: vomoUser.phone,
    custom_fields: {
      user_status: vomoUser.user_status,
      membership_role: vomoUser.membership_role,
      vomo_created_at: vomoUser.created_at,
    },
  });

  // 2. Record the mapping
  await db.upsert('user_mappings', {
    customer_id: customerId,
    vomo_user_id: vomoUser.id,
    external_user_id: externalRecord.id,
    last_known_email: vomoUser.email,
    last_synced_at: new Date(),
    last_synced_updated_at: new Date(vomoUser.updated_at),
  });

  // 3. Audit log
  await db.insert('sync_audit', {
    customer_id: customerId,
    resource_type: 'user',
    record_id: vomoUser.id,
    operation: 'backfill',
    succeeded: true,
    processed_at: new Date(),
  });
}

The three writes — to external system, to mapping table, to audit log — happen together. If any fails, the whole record fails (caught by the backfill’s try/catch and routed to DLQ).

Backfill duration estimates

Customer size	Backfill duration (at 3 req/sec)
1,000 users	~3 minutes
10,000 users	~30 minutes
100,000 users	~5-6 hours
500,000 users	~1 day

Communicate this during onboarding so customers understand the initial sync isn’t instant. Show progress in the UI if available.

Step 3: steady-state polling

After backfill, the polling worker takes over. It runs on a schedule and processes only what’s changed.

JavaScript

class UserSyncPoller {
  constructor({ customerId, token }) {
    this.customerId = customerId;
    this.token = token;
    this.client = new ThrottledVomoClient({ token, requestsPerSecond: 3 });
  }

  async poll() {
    const lastSync = await db.getCheckpoint(this.customerId, 'user_sync');
    const traceId = generateTraceId();

    await jobLog.start(this.customerId, 'user_sync_poll', { traceId });

    const params = new URLSearchParams({
      updated_after: lastSync.toISOString(),
    });

    let url = `https://api.vomo.org/v1/users?${params}`;
    let latestSeen = lastSync;
    let processedCount = 0;
    const failures = [];

    try {
      while (url) {
        const response = await this.client.fetch(url);
        if (!response.ok) throw new Error(`Poll failed: ${response.status}`);

        const page = await response.json();
        for (const user of page.data) {
          try {
            await this._processUserChange(user, traceId);
            processedCount++;

            const u = new Date(user.updated_at);
            if (u > latestSeen) latestSeen = u;
          } catch (err) {
            failures.push({ user, error: err.message });
          }
        }

        url = page.links.next;
      }

      if (failures.length > 0) {
        await deadLetterQueue.publishBatch(
          failures.map((f) => ({
            type: 'user_sync_failure',
            customerId: this.customerId,
            user: f.user,
            error: f.error,
            traceId,
          }))
        );
      }

      await db.upsert('sync_checkpoints', {
        customer_id: this.customerId,
        resource: 'user_sync',
        checkpoint: latestSeen,
      });

      await jobLog.complete(this.customerId, 'user_sync_poll', {
        traceId,
        processedCount,
        failureCount: failures.length,
        newCheckpoint: latestSeen,
      });

      return { processedCount, failureCount: failures.length };
    } catch (err) {
      await jobLog.fail(this.customerId, 'user_sync_poll', { traceId, error: err.message });
      throw err;
    }
  }

  async _processUserChange(vomoUser, traceId) {
    const existing = await db.getUserMapping(this.customerId, vomoUser.id);
    const isNew = !existing;

    // Email-change detection
    if (existing && existing.last_known_email !== vomoUser.email?.toLowerCase()) {
      await alertOps({
        severity: 'medium',
        customerId: this.customerId,
        type: 'email_changed',
        vomoUserId: vomoUser.id,
        oldEmail: existing.last_known_email,
        newEmail: vomoUser.email,
        traceId,
      });
    }

    // Upsert into external system
    const externalRecord = await externalSystem.upsertUser({
      external_id: `vomo-${vomoUser.id}`,
      email: vomoUser.email,
      first_name: vomoUser.first_name,
      last_name: vomoUser.last_name,
      phone: vomoUser.phone,
      custom_fields: {
        user_status: vomoUser.user_status,
        membership_role: vomoUser.membership_role,
      },
    });

    // Update mapping
    await db.upsert('user_mappings', {
      customer_id: this.customerId,
      vomo_user_id: vomoUser.id,
      external_user_id: externalRecord.id,
      last_known_email: vomoUser.email?.toLowerCase(),
      last_synced_at: new Date(),
      last_synced_updated_at: new Date(vomoUser.updated_at),
    });

    // Audit log
    await db.insert('sync_audit', {
      customer_id: this.customerId,
      resource_type: 'user',
      record_id: vomoUser.id,
      operation: isNew ? 'create' : 'update',
      trace_id: traceId,
      succeeded: true,
      processed_at: new Date(),
    });

    // Fire side-effect events
    if (isNew) {
      await eventBus.emit('user.created', { customerId: this.customerId, user: vomoUser });
    } else {
      await eventBus.emit('user.updated', { customerId: this.customerId, user: vomoUser });
    }
  }
}

Why use the mapping table to detect new vs. update

The 200/201 detection works at the VOMO upsert level. Here we’re going the other direction (VOMO → external) — the mapping table is the source of truth for “have we seen this user before in this customer’s integration”:

Mapping exists?	Treatment
No	New user from the integration’s perspective; fire `user.created`
Yes	Known user; fire `user.updated`

This is more robust than time-based heuristics (created_at == updated_at). The mapping table is the integration’s persisted view of reality.

Step 4: daily reconciliation

Daily reconciliation catches gaps and re-processes them. See Reconciliation Patterns for the full pattern.

JavaScript

async function dailyUserReconciliation(customerId) {
  const yesterday = startOfDayUTC(daysAgo(1));
  const today = startOfDayUTC(daysAgo(0));

  // 1. Re-read yesterday's changes
  const params = new URLSearchParams({
    updated_after: yesterday.toISOString(),
    updated_before: today.toISOString(),
  });

  const recentUsers = await paginate(
    `https://api.vomo.org/v1/users?${params}`,
    customerId
  );

  // 2. Check each was processed
  const gaps = [];
  for (const user of recentUsers) {
    const mapping = await db.getUserMapping(customerId, user.id);
    if (!mapping) {
      gaps.push({ user, reason: 'not_in_mapping' });
      continue;
    }
    if (new Date(mapping.last_synced_updated_at) < new Date(user.updated_at)) {
      gaps.push({ user, reason: 'stale_in_mapping' });
    }
  }

  // 3. Re-process gaps
  for (const gap of gaps) {
    try {
      await processUserGapRecovery(customerId, gap.user);
    } catch (err) {
      await deadLetterQueue.publish({
        type: 'reconciliation_failure',
        customerId,
        user: gap.user,
        error: err.message,
      });
    }
  }

  // 4. Record result
  await db.insert('reconciliation_runs', {
    customer_id: customerId,
    type: 'user_daily',
    date: yesterday,
    expected_count: recentUsers.length,
    gap_count: gaps.length,
    completed_at: new Date(),
  });

  return { expectedCount: recentUsers.length, gapCount: gaps.length };
}

Weekly deletion detection

JavaScript

async function weeklyUserDeletionScan(customerId) {
  const currentUsers = await paginate('https://api.vomo.org/v1/users', customerId);
  const currentIds = new Set(currentUsers.map((u) => u.id));

  const knownMappings = await db.getAllUserMappings(customerId);
  const possiblyDeleted = knownMappings.filter((m) => !currentIds.has(m.vomo_user_id));

  const confirmedDeletions = [];
  for (const mapping of possiblyDeleted) {
    const detail = await fetchUserDetail(mapping.vomo_user_id, customerId);
    if (!detail) {
      confirmedDeletions.push(mapping); // 404 — truly deleted
    }
  }

  for (const mapping of confirmedDeletions) {
    await externalSystem.markUserDeleted(mapping.external_user_id);
    await db.deleteUserMapping(customerId, mapping.vomo_user_id);

    await db.insert('sync_audit', {
      customer_id: customerId,
      resource_type: 'user',
      record_id: mapping.vomo_user_id,
      operation: 'delete',
      succeeded: true,
      processed_at: new Date(),
    });
  }

  return {
    possiblyDeleted: possiblyDeleted.length,
    confirmedDeletions: confirmedDeletions.length,
  };
}

Step 5: customer-facing visibility

Customers ask “is our integration working?” — build a per-customer dashboard that answers in seconds:

JavaScript

async function getCustomerSyncStatus(customerId) {
  const config = await db.getSyncConfig(customerId);
  const latestPoll = await db.getLatestJobLog(customerId, 'user_sync_poll');
  const checkpoint = await db.getCheckpoint(customerId, 'user_sync');
  const dlqDepth = await deadLetterQueue.getDepthForCustomer(customerId);
  const last7DaysRecon = await db.getRecentReconciliations(customerId, 7);
  const usersTotal = await db.countUserMappings(customerId);

  return {
    integration: {
      enabled: config.enabled,
      onboardedAt: config.onboarded_at,
      backfillCompletedAt: config.backfill_completed_at,
    },
    polling: {
      lastRunAt: latestPoll?.completed_at,
      lastRunOutcome: latestPoll?.outcome,
      checkpointTimestamp: checkpoint,
      lagMinutes: Math.round((Date.now() - checkpoint.getTime()) / (60 * 1000)),
    },
    health: {
      dlqDepth,
      averageReconciliationGapRate:
        last7DaysRecon.length > 0
          ? last7DaysRecon.reduce((sum, r) => sum + r.gap_count / Math.max(r.expected_count, 1), 0) /
            last7DaysRecon.length
          : 0,
      isHealthy:
        dlqDepth < 10 &&
        latestPoll?.outcome === 'success' &&
        Date.now() - checkpoint.getTime() < 60 * 60 * 1000, // <1hr lag
    },
    inventory: {
      totalUsersMirrored: usersTotal,
    },
  };
}

Surface this through a UI in your partner product. Customers can self-diagnose “did the sync run today?” without contacting support.

Bidirectional sync

Some integrations need to also write back to VOMO — e.g., the external CRM is source-of-truth for user phone numbers.

JavaScript

async function syncFromExternalToVomo(customerId, externalUserUpdate) {
  const mapping = await db.getUserMappingByExternalId(customerId, externalUserUpdate.id);

  if (!mapping) {
    return findOrCreateUserInVomo(customerId, externalUserUpdate);
  }

  // ⚠ Email mismatch is dangerous — would create VOMO duplicate
  if (mapping.last_known_email !== externalUserUpdate.email.toLowerCase()) {
    await alertOps({
      severity: 'high',
      customerId,
      type: 'email_mismatch_bidirectional',
      vomoUserId: mapping.vomo_user_id,
      vomoEmail: mapping.last_known_email,
      externalEmail: externalUserUpdate.email,
    });
    return { skipped: true, reason: 'email_mismatch' };
  }

  const token = await credentials.getVomoToken(customerId);
  const response = await fetch('https://api.vomo.org/v1/users', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      email: externalUserUpdate.email,
      first_name: externalUserUpdate.firstName,
      last_name: externalUserUpdate.lastName,
      phone: externalUserUpdate.phone,
    }),
  });

  if (!response.ok) throw new Error(`VOMO update failed: ${response.status}`);
  return { synced: true };
}

The critical pattern: email mismatch in the external→VOMO direction is dangerous (creates VOMO duplicates). Detect, alert, don’t auto-sync — let a human resolve it. For most integrations, stay one-way (VOMO → external only) unless bidirectional is a genuine business need.

Operational concerns

Token expiration

JavaScript

async function handleAuthFailure(customerId) {
  await db.upsert('sync_configs', { customer_id: customerId, enabled: false });
  await emailService.send({
    to: customerSettings[customerId].adminEmail,
    template: 'vomo_token_invalid',
    data: { customerId },
  });
  await alertOps({ severity: 'high', customerId, type: 'auth_failed' });
}

See Authentication: Handling auth failures.

Customer offboarding

JavaScript

async function offboardCustomer(customerId) {
  // 1. Disable sync
  await db.upsert('sync_configs', {
    customer_id: customerId,
    enabled: false,
    offboarded_at: new Date(),
  });

  // 2. Stop scheduled jobs
  await jobScheduler.removeAllForCustomer(customerId);

  // 3. Delete credentials immediately
  await credentials.deleteForCustomer(customerId);

  // 4. Keep mapping table 30 days for re-onboarding, then delete
  await jobScheduler.scheduleOneTime({
    delayDays: 30,
    type: 'permanent_offboard',
    customerId,
  });
}

Customers sometimes re-enable shortly after disabling. The 30-day retention window allows re-onboarding without re-running the full backfill.

Per-customer rate-limit budgets

JavaScript

class PerCustomerRateBudget {
  constructor() {
    this.usageBucket = new Map();
  }

  async canCustomerProceed(customerId, requestCount = 1) {
    const minuteKey = `${customerId}:${Math.floor(Date.now() / 60000)}`;
    const current = this.usageBucket.get(minuteKey) ?? 0;

    if (current + requestCount > 100) return false; // 100 req/min per customer

    this.usageBucket.set(minuteKey, current + requestCount);
    return true;
  }
}

Fair-share rate limiting prevents one customer’s heavy use from starving others.

Things to watch for

A few subtleties that surface in production:

The participation gap

This recipe syncs users but not their participations (since participations don’t advance updated_at). For integrations that need participation data, see Detecting User Changes: The participation caveat and add a separate participation-polling layer.

Large customers stress the integration

A 100,000-user customer can produce 5,000 changes per day during active periods. The polling worker handles this through pagination, but downstream processing (writes to external system) is often the bottleneck. Monitor per-customer processing rate and scale workers as needed.

Backfill scheduling

For very large customers, the initial backfill can take hours. Schedule it for low-traffic hours and communicate timing to the customer. Prevent multiple workers from picking up the same backfill (use a lock or unique-claim pattern).

Mapping table growth

Per-customer mapping tables grow unbounded over time. For multi-tenant integrations:

Periodic archival of long-inactive customers’ mapping tables
Partitioning by customer for query performance
Indexes on (customer_id, vomo_user_id) and (customer_id, email)

Dead-letter queue management

The DLQ catches failures but isn’t self-cleaning. Build:

Automatic retry of recoverable failures (timeouts, transient 5xx)
Manual review queue for unrecoverable failures (data validation, permanent destination errors)
Aging policies (alert on DLQ entries older than N days)

What you’ve built

After this recipe:

✅ Onboarding flow that captures and validates the Bearer token
✅ Initial backfill that mirrors all VOMO users to the external system
✅ Polling worker that catches incremental changes every 15-30 minutes
✅ Daily reconciliation catching gaps
✅ Weekly deletion detection
✅ Per-customer dashboards showing sync health
✅ Operational handling for token failures, offboarding, and rate budgets

This is the foundational integration shape for most VOMO partner work. Other recipes (Groups, hours reporting) build on this base.

Where to go next

Build a Group from a Query

Build Groups dynamically from query results.

Report on Volunteer Hours

The reporting recipe using participation data.

Reconciliation Patterns

The reconciliation patterns this recipe references.

Sync Architecture Patterns

The broader architectural patterns for sync designs.

​What you’ll build

​When this recipe fits

​Architecture

​Step 1: customer onboarding

​Why initialize the checkpoint to epoch

​Step 2: initial backfill

​What processUserForBackfill does

​Backfill duration estimates

​Step 3: steady-state polling

​Why use the mapping table to detect new vs. update

​Step 4: daily reconciliation

​Weekly deletion detection

​Step 5: customer-facing visibility

​Bidirectional sync

​Operational concerns

​Token expiration

​Customer offboarding

​Per-customer rate-limit budgets

​Things to watch for

​The participation gap

​Large customers stress the integration

​Backfill scheduling

​Mapping table growth

​Dead-letter queue management

​What you’ve built

​Where to go next

Build a Group from a Query

Report on Volunteer Hours

Reconciliation Patterns

Sync Architecture Patterns

What you’ll build

When this recipe fits

Architecture

Step 1: customer onboarding

Why initialize the checkpoint to epoch

Step 2: initial backfill

What `processUserForBackfill` does

Backfill duration estimates

Step 3: steady-state polling

Why use the mapping table to detect new vs. update

Step 4: daily reconciliation

Weekly deletion detection

Step 5: customer-facing visibility

Bidirectional sync

Operational concerns

Token expiration

Customer offboarding

Per-customer rate-limit budgets

Things to watch for

The participation gap

Large customers stress the integration

Backfill scheduling

Mapping table growth

Dead-letter queue management

What you’ve built

Where to go next