> ## Documentation Index
> Fetch the complete documentation index at: https://docs.virtuous.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Reconciliation Patterns

> The slow-scan patterns that catch what polling misses — daily reconciliation against polling gaps, deletion detection without webhooks, three-way state reconciliation, and the per-customer orchestration that keeps integrations production-grade reliable.

Polling catches most changes — but not all. Some events slip through the gaps:

* **Deletions** don't appear in `updated_after` queries (the record is just gone)
* **Silent processing failures** advance the checkpoint despite the change not being applied
* **Field changes that don't advance `updated_at`** (rare but possible)
* **Worker crashes mid-batch** that fail to track which records were processed
* **External-system writes that fail asynchronously** — the integration thought it succeeded but didn't

Reconciliation is the answer to all of these. It's a separate, slower-cadence process that audits actual state against expected state and catches the gaps. Combined with polling, it produces production-grade reliability that polling alone cannot.

This page covers the reconciliation patterns: daily and weekly cadences, deletion detection, three-way state checks, and the per-customer orchestration that makes reconciliation work at scale.

If you haven't yet, skim the [Polling Overview](/volunteer/polling-and-sync/overview) for the polling foundation that reconciliation complements.

## What reconciliation does

A reconciliation pass:

<Steps>
  <Step title="Reads the actual current state from VOMO">
    Typically a full or windowed list of a resource.
  </Step>

  <Step title="Reads the expected state from your integration's external store">
    What you believe VOMO contains based on prior sync activity.
  </Step>

  <Step title="Computes the diff">
    Records in VOMO but not in your store (missed creates / updates), records in your store but not in VOMO (missed deletions), records with mismatched fields (drift).
  </Step>

  <Step title="Applies corrective actions">
    Re-process missed records, mark deletions in downstream systems, alert on unexplained drift.
  </Step>

  <Step title="Records the reconciliation result">
    For monitoring, audit trail, and capacity planning.
  </Step>
</Steps>

The pattern is the same regardless of resource (Users, Projects, Groups, etc.). The specifics differ — what fields to compare, what corrective actions are appropriate — but the structure stays consistent.

***

## The minimum reconciliation: daily catch-up

The simplest useful reconciliation: every night, re-read yesterday's changes and verify they were processed.

```javascript JavaScript theme={null}
async function dailyReconcileUsers(customerId) {
  const yesterday = new Date();
  yesterday.setDate(yesterday.getDate() - 1);
  yesterday.setHours(0, 0, 0, 0);

  const today = new Date(yesterday);
  today.setDate(today.getDate() + 1);

  // 1. Re-query for yesterday's changes
  const params = new URLSearchParams({
    updated_after: yesterday.toISOString(),
    updated_before: today.toISOString(),
  });

  let url = `https://api.vomo.org/v1/users?${params}`;
  const expectedUsers = [];

  while (url) {
    const response = await fetch(url, {
      headers: { Authorization: `Bearer ${process.env.VOMO_API_TOKEN}` },
    });
    const page = await response.json();
    expectedUsers.push(...page.data);
    url = page.links.next;
  }

  // 2. Check which were actually processed
  const gaps = [];
  for (const user of expectedUsers) {
    const processed = await externalDb.userWasProcessedAt(
      customerId,
      user.id,
      new Date(user.updated_at)
    );
    if (!processed) {
      gaps.push(user);
    }
  }

  // 3. Re-process the gaps
  for (const user of gaps) {
    try {
      await processUserChange(customerId, user);
      await externalDb.recordUserProcessed(customerId, user.id, new Date());
    } catch (err) {
      await deadLetterQueue.publish({
        type: 'reconciliation_failure',
        customerId,
        userId: user.id,
        error: err.message,
      });
    }
  }

  // 4. Record result
  await externalDb.recordReconciliation(customerId, {
    type: 'user_daily',
    date: yesterday,
    expectedCount: expectedUsers.length,
    gapCount: gaps.length,
  });

  return { expectedCount: expectedUsers.length, gapCount: gaps.length };
}

// Schedule for 2am customer-local-time each day
```

### What this catches

| Issue                                                       | How reconciliation catches it                                                                      |
| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Polling worker crashed mid-batch                            | Records modified during the crash weren't in any successful poll; reconciliation re-discovers them |
| Per-record processing failure went to DLQ but was forgotten | Reconciliation finds the un-processed records and tries again                                      |
| Polling worker silently advanced checkpoint past records    | Reconciliation re-reads the time range, doesn't trust the checkpoint                               |
| Network error during a poll's pagination                    | Records on later pages were missed; reconciliation re-fetches them                                 |

### What this doesn't catch

| Issue                                    | Why                                                                      |
| ---------------------------------------- | ------------------------------------------------------------------------ |
| Deletions                                | Deleted records aren't in `updated_after` results — need a separate scan |
| Drift between polling and external state | Need a three-way reconciliation — see below                              |
| Changes that don't advance `updated_at`  | These are invisible to time-based reconciliation; need full-state scan   |

For most production needs, daily catch-up reconciliation closes 90%+ of the gaps that polling misses. The remaining gaps require more comprehensive patterns.

***

## Deletion detection

Deletions are the **hardest** thing to detect in a polling architecture. The pattern: periodically read the full current state, compare to what your external store thinks exists, and treat the difference as deletions.

### The full-state scan

```javascript JavaScript theme={null}
async function detectDeletedUsers(customerId) {
  // 1. Read all current Users from VOMO
  const currentUsers = await listAllUsers();
  const currentIds = new Set(currentUsers.map((u) => u.id));

  // 2. Read what your external store knows
  const knownIds = await externalDb.getAllKnownVomoUserIds(customerId);

  // 3. Find IDs known to external but not in VOMO
  const possiblyDeletedIds = [...knownIds].filter((id) => !currentIds.has(id));

  // 4. Confirm each as actually deleted (vs. inaccessible)
  const confirmedDeletions = [];
  for (const id of possiblyDeletedIds) {
    const detail = await getUserDetail(id);
    if (!detail) {
      // 404 — confirmed deleted
      confirmedDeletions.push(id);
    } else {
      // Still exists but didn't appear in list — log for investigation
      await alertOps({
        severity: 'low',
        type: 'user_missing_from_list',
        customerId,
        userId: id,
      });
    }
  }

  // 5. Propagate deletions
  for (const id of confirmedDeletions) {
    await externalSystem.markUserDeleted(id);
    await externalDb.removeUserKnown(customerId, id);
  }

  return { possiblyDeletedCount: possiblyDeletedIds.length, confirmedDeletions };
}
```

### Run cadence

| Resource     | Suggested deletion-scan cadence                                         |
| ------------ | ----------------------------------------------------------------------- |
| Users        | Weekly (deletions are rare; freshness less critical than other changes) |
| Projects     | Weekly                                                                  |
| Groups       | Daily (Groups can be intentionally deleted as part of admin cleanup)    |
| Campaigns    | Weekly                                                                  |
| Forms        | Monthly                                                                 |
| Certificates | Monthly                                                                 |

Deletion scans are full-dataset operations — expensive in API requests. Pace accordingly:

```text theme={null}
A scan for 10,000 users at 15-per-page = 667 requests
At 3 req/sec, that's ~3.7 minutes of solid polling
Across 10 customers, ~37 minutes/day on deletion scans alone
```

For larger customers, consider weekly cadences and off-peak scheduling.

### Distinguishing deletion from inaccessibility

A user "missing" from the list query could mean:

| Cause                                                | Action                                   |
| ---------------------------------------------------- | ---------------------------------------- |
| User was deleted in VOMO                             | Propagate deletion to external           |
| User moved to a different organization               | Don't propagate — handle as scope change |
| Token's permissions narrowed                         | Don't propagate — handle as scope change |
| User was banned/soft-deleted but record still exists | Don't propagate as full deletion         |

The detail fetch (`GET /users/{id}`) resolves the ambiguity:

* `200` response → user exists but didn't match list query (investigate filters)
* `404` response → user is truly gone

Without the detail-fetch step, you'd propagate "deletions" for users who actually just changed scope — confusing the external system.

***

## Three-way state reconciliation

The most rigorous pattern: compare three sources of truth.

```mermaid theme={null}
graph LR
  VOMO["VOMO<br/>(authoritative)"]
  Partner["Partner integration<br/>(state store)"]
  External["External system<br/>(destination)"]

  VOMO -.->|"reconcile"| Partner
  Partner -.->|"reconcile"| External
  VOMO -.->|"reconcile<br/>(through partner)"| External
```

Each pair has a possible disagreement:

| Comparison                   | Detects                                                               |
| ---------------------------- | --------------------------------------------------------------------- |
| VOMO ↔ Partner state         | Polling gaps (Partner missed a VOMO change)                           |
| Partner state ↔ External     | Push failures (Partner thought it pushed but External didn't receive) |
| VOMO ↔ External (transitive) | End-to-end correctness — what the customer ultimately experiences     |

### The end-to-end audit

```javascript JavaScript theme={null}
async function endToEndAudit(customerId, sampleSize = 100) {
  // 1. Sample a few users from VOMO
  const allUsers = await listAllUsers();
  const sample = randomSample(allUsers, sampleSize);

  const issues = [];

  for (const vomoUser of sample) {
    // 2. Verify partner state
    const partnerKnown = await externalDb.getUserState(customerId, vomoUser.id);
    if (!partnerKnown) {
      issues.push({
        userId: vomoUser.id,
        issue: 'unknown_to_partner',
        vomoLastUpdated: vomoUser.updated_at,
      });
      continue;
    }

    // 3. Verify external state
    const externalKnown = await externalSystem.getUser(vomoUser.id);
    if (!externalKnown) {
      issues.push({
        userId: vomoUser.id,
        issue: 'missing_from_external',
        vomoLastUpdated: vomoUser.updated_at,
        partnerLastSynced: partnerKnown.lastSyncedAt,
      });
      continue;
    }

    // 4. Verify field-level consistency
    if (vomoUser.email !== externalKnown.email) {
      issues.push({
        userId: vomoUser.id,
        issue: 'email_mismatch',
        vomoValue: vomoUser.email,
        externalValue: externalKnown.email,
      });
    }

    // ... other field checks ...
  }

  await externalDb.recordAuditResult(customerId, {
    timestamp: new Date(),
    sampleSize,
    issueCount: issues.length,
    issues,
  });

  if (issues.length > sampleSize * 0.05) {
    // >5% issue rate — significant problem
    await alertOps({
      severity: 'high',
      message: `Integration audit found ${issues.length}/${sampleSize} issues`,
      customerId,
    });
  }

  return { sampleSize, issueCount: issues.length, issues };
}
```

### When to run end-to-end audits

| Cadence                            | Purpose                                                        |
| ---------------------------------- | -------------------------------------------------------------- |
| Weekly random sample               | Detect drift early                                             |
| After any significant code change  | Verify the change doesn't regress sync correctness             |
| When a customer reports an issue   | Confirm whether issue is broader than the single reported case |
| Before quarterly reporting cutoffs | Confirm data integrity for downstream reports                  |

For partner integrations serving customers with strict data correctness needs (compliance reporting, audited financials), random sampling at 1% weekly catches most drift before it becomes a customer-facing issue.

***

## Incremental vs. full reconciliation

Two distinct cadences for different purposes:

### Incremental (daily) reconciliation

Re-reads a recent time window. Lower cost, catches recent gaps.

```javascript JavaScript theme={null}
async function dailyIncrementalReconcile(customerId, resourceType) {
  const yesterday = startOfDayUTC(daysAgo(1));
  const today = startOfDayUTC(daysAgo(0));

  const params = new URLSearchParams({
    updated_after: yesterday.toISOString(),
    updated_before: today.toISOString(),
  });

  const recentRecords = await paginate(
    `https://api.vomo.org/v1/${resourceType}?${params}`
  );

  // Check each was processed
  const gaps = [];
  for (const record of recentRecords) {
    const wasProcessed = await externalDb.wasRecordProcessed(
      customerId,
      resourceType,
      record.id,
      new Date(record.updated_at)
    );
    if (!wasProcessed) gaps.push(record);
  }

  // Re-process gaps
  await reprocessGaps(customerId, resourceType, gaps);

  return { recentRecords: recentRecords.length, gaps: gaps.length };
}
```

**Cost:** Bounded by the daily change volume. Typical: a few hundred records per day per customer. Manageable.

### Full (weekly) reconciliation

Re-reads everything. Higher cost, catches everything including deletions and drift.

```javascript JavaScript theme={null}
async function weeklyFullReconcile(customerId, resourceType) {
  // 1. Read all current records
  const current = await paginate(`https://api.vomo.org/v1/${resourceType}`);
  const currentIds = new Set(current.map((r) => r.id));

  // 2. Read what external store knows
  const known = await externalDb.getAllKnownRecordIds(customerId, resourceType);

  // 3. Detect categories of issue
  const missingFromPartner = current.filter((r) => !known.has(r.id));
  const possiblyDeleted = [...known].filter((id) => !currentIds.has(id));

  // 4. For each missing-from-partner, process
  for (const record of missingFromPartner) {
    await processChange(customerId, resourceType, record);
  }

  // 5. For each possibly-deleted, confirm
  for (const id of possiblyDeleted) {
    const detail = await getRecordDetail(resourceType, id);
    if (!detail) {
      await externalSystem.markDeleted(resourceType, id);
      await externalDb.removeKnown(customerId, resourceType, id);
    }
  }

  return {
    currentCount: current.length,
    addedCount: missingFromPartner.length,
    deletedCount: possiblyDeleted.length,
  };
}
```

**Cost:** Full dataset read each run. Expensive — schedule during off-peak hours.

### Combining the two

For most production integrations, both cadences run:

```mermaid theme={null}
graph TD
  Daily["Daily incremental<br/>(02:00 customer time)"]
  Weekly["Weekly full reconcile<br/>(Sundays, 03:00)"]
  Monthly["Monthly audit<br/>(1st of month, 04:00)"]

  Daily -.->|"Catches recent gaps"| Daily
  Weekly -.->|"Catches deletions, drift"| Weekly
  Monthly -.->|"Catches anything still missed"| Monthly
```

Different cadences catch different issue classes; the layered approach catches more than any single pass.

***

## Per-resource reconciliation patterns

Different resources have different reconciliation needs:

### Users

| Concern                          | Pattern                               |
| -------------------------------- | ------------------------------------- |
| Polling caught most changes      | Daily incremental — re-read yesterday |
| Detect deletions                 | Weekly full scan                      |
| Detect drift in email/name/phone | Monthly random-sample audit           |
| Detect missed participations     | Daily Project Date scan (see below)   |

### Projects

| Concern                                                   | Pattern                                      |
| --------------------------------------------------------- | -------------------------------------------- |
| Metadata change gaps                                      | Daily incremental                            |
| Schedule drift (Project Date count differs from snapshot) | Weekly full scan with `all_dates` comparison |
| Project deletion (rare but happens)                       | Weekly full scan                             |
| Project Date deletions                                    | Re-fetched as part of schedule diff          |

### Groups

| Concern                                      | Pattern                                        |
| -------------------------------------------- | ---------------------------------------------- |
| Membership drift                             | Daily — re-read members for active Groups      |
| Group deletion (more common — admin cleanup) | Daily full scan                                |
| Parent-child relationship changes            | Daily — re-check `parent_id` on changed Groups |

### Forms / Form Completions

| Concern                  | Pattern                           |
| ------------------------ | --------------------------------- |
| Form structure changes   | Weekly                            |
| Missed Form Completions  | Daily incremental per active Form |
| Completion modifications | Detected by `updated_after`       |

### Participation reconciliation (the hardest case)

Because participations aren't directly polled, reconciliation is the only path. Run daily:

```javascript JavaScript theme={null}
async function reconcileRecentParticipations(customerId) {
  // 1. Get Projects with active recent Dates
  const recent = await getProjectsWithRecentDates(customerId, 7); // last 7 days

  for (const project of recent) {
    for (const date of project.all_dates) {
      const isRecent =
        new Date(date.starts_at) > new Date(Date.now() - 14 * 24 * 60 * 60 * 1000);
      if (!isRecent) continue;

      const dateDetail = await getProjectDate(date.id);
      const currentParticipants = dateDetail.data.participants ?? [];
      const knownParticipantIds = new Set(
        await externalDb.getKnownParticipantIds(customerId, date.id)
      );

      // New participants
      for (const p of currentParticipants) {
        if (!knownParticipantIds.has(p.id)) {
          await processNewParticipation(customerId, project, date, p);
          await externalDb.recordParticipant(customerId, date.id, p.id);
        }
      }

      // Removed participants (cancellations)
      const currentIds = new Set(currentParticipants.map((p) => p.id));
      for (const knownId of knownParticipantIds) {
        if (!currentIds.has(knownId)) {
          await processRemovedParticipation(customerId, date.id, knownId);
          await externalDb.removeParticipant(customerId, date.id, knownId);
        }
      }
    }
  }
}
```

This runs on a tighter cadence than other reconciliation (every few hours for active customers) because participation freshness matters more for downstream workflows.

***

## Per-customer orchestration

For partner integrations serving many customers, reconciliation across them needs orchestration:

```javascript JavaScript theme={null}
class ReconciliationOrchestrator {
  constructor({ customers, throttle }) {
    this.customers = customers;
    this.throttle = throttle;
  }

  async runDaily() {
    for (const customerId of this.customers) {
      try {
        await this._reconcileCustomer(customerId);
      } catch (err) {
        await alertOps({
          severity: 'medium',
          message: `Reconciliation failed for ${customerId}`,
          error: err.message,
        });
        // Continue to next customer
      }
    }
  }

  async _reconcileCustomer(customerId) {
    const tasks = [
      { name: 'users_incremental', fn: () => dailyReconcileUsers(customerId) },
      { name: 'projects_incremental', fn: () => dailyReconcileProjects(customerId) },
      { name: 'participations', fn: () => reconcileRecentParticipations(customerId) },
    ];

    for (const task of tasks) {
      const startTime = Date.now();
      const result = await task.fn();
      const durationMs = Date.now() - startTime;

      await externalDb.recordReconciliationRun({
        customerId,
        task: task.name,
        startedAt: new Date(startTime),
        durationMs,
        result,
      });
    }
  }
}
```

### Staggering

Avoid running reconciliation for all customers at the same minute — spike load and rate-limit hits will follow. Stagger:

```javascript JavaScript theme={null}
async function stagggeredDailyReconciliation(customers) {
  for (let i = 0; i < customers.length; i++) {
    const delay = i * 60 * 1000; // 1 minute between customer starts
    setTimeout(() => reconcileCustomer(customers[i]), delay);
  }
}
```

For 60 customers, this spreads reconciliation across an hour. The total elapsed time is the same, but no minute has all 60 customers' reconciliation running simultaneously.

### Per-customer cadence

Not every customer needs the same cadence. Small customers (a few hundred users) might be fine with weekly full reconciliation; large customers (50,000+ users) need daily.

```javascript JavaScript theme={null}
async function chooseReconciliationCadence(customerId) {
  const stats = await getCustomerStats(customerId);

  if (stats.userCount > 10000 || stats.changesPerDay > 500) {
    return 'daily_full';
  }
  if (stats.userCount > 1000) {
    return 'daily_incremental_weekly_full';
  }
  return 'weekly_full';
}
```

The cadence becomes per-customer configuration, tuned to actual activity.

***

## Reconciliation as documentation

A useful side-effect of reconciliation: it produces an **audit trail**. Each reconciliation run records:

* When it ran
* What it checked
* What gaps it found
* What corrective actions it took
* Whether the integration is operating correctly

For customer-facing transparency, this audit trail is valuable:

```javascript JavaScript theme={null}
async function getCustomerSyncHealth(customerId) {
  const recent = await externalDb.getRecentReconciliations(customerId, 30);

  const dailyResults = recent.filter((r) => r.task === 'users_incremental');
  const gapsPerDay = dailyResults.map((r) => ({
    date: r.startedAt,
    expectedCount: r.result.expectedCount,
    gapCount: r.result.gapCount,
    rate: r.result.gapCount / r.result.expectedCount,
  }));

  const avgGapRate = average(gapsPerDay.map((d) => d.rate));

  return {
    averageGapRate: avgGapRate,
    healthy: avgGapRate < 0.01, // <1% gaps is healthy
    days: gapsPerDay,
  };
}
```

Customers asking "is our sync working?" get a definitive answer backed by audit data.

***

## When reconciliation finds too much

A successful reconciliation finds zero gaps. A few gaps per day is normal. A sudden spike in gaps means something is wrong with polling:

| Pattern                                         | Likely cause                                                                                                                      |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Daily incremental finds 50%+ of records as gaps | Polling is broken — checkpoint not advancing, workers stalled, or polling cadence is too long                                     |
| Weekly full finds 100s of "deleted" records     | Either a real mass-deletion event, or polling missed creates (records exist in VOMO and externally but not in your partner state) |
| Sudden field-level drift on a specific field    | Mapping logic is broken — recent code change may have regressed                                                                   |
| One customer has high gap rate; others normal   | Customer-specific issue — token, rate-limiting, or org-specific data problem                                                      |

Treat reconciliation results as a **signal for investigation**, not just automatic correction. If gaps grow, pause auto-correction until the root cause is understood.

***

## Production checklist

For reconciliation:

* Daily incremental reconciliation per resource per customer
* Weekly full reconciliation including deletion detection
* Three-way audits (sample-based) at least weekly
* Reconciliation is throttled — doesn't compete with regular polling
* Per-customer cadence is configurable, not hardcoded
* Staggered scheduling avoids same-minute spikes
* Audit trail of every reconciliation run is preserved
* Per-customer "sync health" view exposes the data
* Alerts fire when gap rate exceeds threshold (e.g., >1% daily)
* When reconciliation finds too much, pause auto-correction and surface for review

These practices catch the residual failures that polling misses — turning a "mostly works" integration into one that's auditable, debuggable, and trustworthy.

***

## Where to go next

<CardGroup cols={2}>
  <Card title="Change Detection Best Practices" icon="list-check" href="/volunteer/polling-and-sync/change-detection-best-practices">
    The cross-cutting reliability patterns: checkpointing, idempotency, drift detection.
  </Card>

  <Card title="Sync Architecture Patterns" icon="route" href="/volunteer/best-practices/sync-architecture-patterns">
    The broader architectural picture for sync designs.
  </Card>

  <Card title="Detecting User Changes" icon="user-clock" href="/volunteer/polling-and-sync/detecting-user-changes">
    The polling pattern that reconciliation complements.
  </Card>

  <Card title="Detecting Project Changes" icon="briefcase-clock" href="/volunteer/polling-and-sync/detecting-project-changes">
    Same for Projects, including the schedule-diff pattern.
  </Card>
</CardGroup>