> ## Documentation Index
> Fetch the complete documentation index at: https://docs.virtuous.org/llms.txt
> Use this file to discover all available pages before exploring further.
# Reconcile Failed Syncs
> Detect, investigate, and resolve the sync failures that any production integration eventually encounters — stuck Transactions, needs-update gifts, missed webhooks, and out-of-sync records.
Every production sync integration eventually has records out of sync between the partner platform and Virtuous. Webhooks get dropped during outages. Transactions land in the needs-update bucket. Manual entries on either side create records the other doesn't know about. Reconciliation is the practice of finding these gaps and resolving them.
This workflow covers the four categories of sync failure, how to detect each, and the resolution patterns for each. It's the safety net that makes the rest of the [Sync External Donations](/crm/workflows/sync-external-donations) and [Build a Two-Way Sync](/crm/workflows/build-a-two-way-sync) architectures complete.
## Categories of sync failure
| Category | Symptom | Detection |
| ----------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
| **Stuck Transactions** | Submitted to Virtuous, accepted, but never produced a real Contact or Gift. | Records marked `submitted` in your queue with no matching webhook after a reasonable window. |
| **Needs-update bucket** | Gift Transactions that failed contact matching and require manual resolution in the Virtuous UI. | A `404` from `GET /api/Gift/{transactionSource}/{transactionId}` long after submission, combined with no `giftCreate` webhook. |
| **Missed webhook deliveries** | Records that exist in Virtuous but your platform was never notified of. | Records present in a Virtuous Query result but not in your platform's database. |
| **Direction-specific drift** | A record exists on both sides but with different field values. | Field-by-field comparison during reconciliation surfaces mismatches. |
Each has a different resolution path. The detection and resolution patterns below assume the sync architecture from [Sync External Donations into Virtuous](/crm/workflows/sync-external-donations).
***
## Category 1: stuck Transactions
A Transaction submitted to Virtuous is **stuck** if it was accepted (`200 OK` response from `POST /api/v2/Gift/Transaction` or `POST /api/Contact/Transaction`) but no `giftCreate`/`contactCreate` webhook has been delivered and no Gift/Contact can be looked up by the `transactionSource`/`transactionId` pair after a reasonable window.
The typical cause is webhook delivery loss during your endpoint's outage combined with a missed reconciliation pass. The Transaction may have actually been resolved successfully; your records just never reflect it.
### Detection
```javascript JavaScript theme={null}
async function findStuckTransactions(customerId) {
const token = await loadCustomerApiToken(customerId);
const cutoff = new Date(Date.now() - 24 * 60 * 60 * 1000); // 24 hours ago
// Find queue records still in 'submitted' state past the cutoff
const stuck = await db.virtuous_donation_queue.find({
where: {
customer_id: customerId,
status: 'submitted',
submitted_at: { lt: cutoff },
},
});
for (const record of stuck) {
const response = await fetch(
`https://api.virtuoussoftware.com/api/Gift/YourPlatform/${encodeURIComponent(record.platform_donation_id)}`,
{ headers: { Authorization: `Bearer ${token}` } }
);
if (response.ok) {
// The Gift exists in Virtuous — the webhook was lost.
const gift = await response.json();
console.log(`Recovered stuck submission ${record.platform_donation_id} → Gift ${gift.id}`);
await db.virtuous_donation_queue.update(
{ platform_donation_id: record.platform_donation_id },
{
status: 'confirmed',
confirmed_at: new Date(),
virtuous_gift_id: gift.id,
recovery_path: 'reconciliation_lookup',
}
);
}
// If response is 404, the Transaction may be in needs-update — see Category 2.
}
}
```
### Resolution
For each record that returns a Gift on the reconciliation lookup, mark it `confirmed` with a note that it was recovered through reconciliation rather than through the normal webhook path. The data is now consistent; the integration just had a delivery gap.
Run this reconciliation on a slow cadence (every few hours, with a 24-hour lookback window). It's a backstop, not the primary outcome-detection mechanism.
***
## Category 2: needs-update bucket
When the matching algorithm cannot confidently associate an incoming Gift Transaction with a Contact, the Transaction is moved to the **needs-update** bucket in Virtuous. The Gift doesn't become a real Gift record — it sits in the bucket waiting for a Virtuous administrator to manually pick a Contact, create a new one, or discard the Transaction.
From the partner integration's perspective, a needs-update Transaction looks the same as a stuck Transaction during the reconciliation lookup: `GET /api/Gift/{transactionSource}/{transactionId}` returns `404` long after submission. The difference is that the Transaction *will* eventually be resolved (when the admin acts on it), not that it's been silently lost.
### Detection
If your reconciliation lookup returns `404` and the submission is significantly past the nightly batch window (say, 72+ hours), the Transaction is most likely in needs-update:
```javascript JavaScript theme={null}
async function findNeedsUpdateTransactions(customerId) {
const cutoff = new Date(Date.now() - 72 * 60 * 60 * 1000); // 72 hours
const stale = await db.virtuous_donation_queue.find({
where: {
customer_id: customerId,
status: 'submitted',
submitted_at: { lt: cutoff },
},
});
const token = await loadCustomerApiToken(customerId);
for (const record of stale) {
const response = await fetch(
`https://api.virtuoussoftware.com/api/Gift/YourPlatform/${encodeURIComponent(record.platform_donation_id)}`,
{ headers: { Authorization: `Bearer ${token}` } }
);
if (response.status === 404) {
// Transaction not resolved to a real Gift after 72 hours.
// Highly likely it landed in needs-update for manual resolution.
await db.virtuous_donation_queue.update(
{ platform_donation_id: record.platform_donation_id },
{
status: 'needs_review',
last_error: 'Likely in needs-update bucket — manual resolution required in Virtuous',
}
);
}
}
}
```
The CRM+ API does not expose a direct endpoint to query the needs-update bucket. Detection is by indirect inference — a Transaction that should have been resolved by now but cannot be looked up by its external reference. If a future API exposes the bucket directly, this detection pattern will be more precise.
### Resolution
Surface needs-review records to your customer's team:
* In your integration's UI, present a "needs Virtuous review" queue listing each record with its donor details and the relevant Stripe/Eventbrite/etc. reference.
* Include a link to the Virtuous UI's Imports section where the customer can resolve the bucket.
* After the customer resolves the items, the resulting Gift `giftCreate` webhook will fire — your normal handler captures the resolution.
For partner integrations with frequent needs-review fallout, the customer's onboarding documentation should explain the bucket exists and that the customer's team needs to monitor it periodically. A growing needs-review queue is a sign that the matching algorithm is under-resolving — usually because the partner integration isn't providing enough matching signal in the Transaction's embedded contact data. See [Handle Duplicate Records](/crm/workflows/handle-duplicate-records#include-strong-matching-signals).
***
## Category 3: missed webhook deliveries
Webhook deliveries can be lost during your endpoint's outages, during prolonged unavailability, or after the retry budget exhausts. The lost events span every event type your integration subscribes to — `contactCreate`, `contactUpdate`, `giftCreate`, `giftUpdate`, and so on.
### Detection
Periodic Query-based reconciliation surfaces missed events:
```javascript JavaScript theme={null}
async function reconcileMissedGiftEvents(customerId, sinceTimestamp) {
const token = await loadCustomerApiToken(customerId);
// Query Virtuous for all Gifts modified since the last reconciliation
const virtuousGifts = await fetchAllPages(async (skip, take) => {
const response = await fetch(
'https://api.virtuoussoftware.com/api/Gift/Query',
{
method: 'POST',
headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
body: JSON.stringify({
groups: [
{
conditions: [
{
parameter: 'Last Modified Date',
operator: 'Is After',
value: sinceTimestamp,
},
],
},
],
sortBy: 'GiftDate',
descending: false,
skip,
take,
}),
}
);
return await response.json();
});
// For each Virtuous Gift, ensure our records reflect it
for (const virtuousGift of virtuousGifts) {
const localGift = await db.partner_gifts.findOne({
virtuous_gift_id: virtuousGift.id,
});
if (!localGift) {
// We don't have this Gift on our side — webhook was missed.
// Apply it through the normal inbound path.
await applyGiftToPartner(virtuousGift);
}
}
}
```
The pattern uses `Last Modified Date` to catch both new gifts (modification timestamp at creation) and updates to existing ones. Run on a slower cadence than the webhook receiver — every 4–12 hours catches most gaps without consuming excessive rate-limit budget.
For Contacts, run the same pattern against `POST /api/Contact/Query`.
### Resolution
Apply missed events through the same inbound-event handler your webhook receiver uses. The handler should be idempotent (see [Idempotency and Safe Reprocessing](/crm/webhooks/idempotency-and-safe-reprocessing)) so that applying the same change a second time — once via the eventually-arriving webhook and once via reconciliation — is a no-op.
If your handler isn't idempotent, the reconciliation pass can produce double-applies. Test this scenario explicitly: replay the same event through the handler twice and confirm the second application produces identical state.
***
## Category 4: direction-specific drift
The hardest category to detect: a record exists on both sides but with different field values. The two systems disagree about the truth.
Causes:
* A sync that wrote partial data (only some fields succeeded).
* A bidirectional sync where conflict resolution went the wrong way for a particular field.
* A manual edit on one side that wasn't captured by the other side's sync.
### Detection
Periodic field-by-field comparison between paired records:
```javascript JavaScript theme={null}
async function findDriftedContacts(customerId) {
const token = await loadCustomerApiToken(customerId);
const drifts = [];
// Walk each paired record on your side
const pairs = await db.partner_contacts.find({
where: { customer_id: customerId, sync_state: 'in_sync' },
});
for (const partnerRecord of pairs) {
const virtuousResponse = await fetch(
`https://api.virtuoussoftware.com/api/Contact/${partnerRecord.virtuous_contact_id}`,
{ headers: { Authorization: `Bearer ${token}` } }
);
if (!virtuousResponse.ok) continue;
const virtuousContact = await virtuousResponse.json();
const differences = compareCanonicalFields(partnerRecord, virtuousContact);
if (differences.length > 0) {
drifts.push({
partnerId: partnerRecord.id,
virtuousContactId: virtuousContact.id,
differences,
});
}
}
return drifts;
}
function compareCanonicalFields(partnerRecord, virtuousContact) {
const differences = [];
if (partnerRecord.email !== extractPrimaryEmail(virtuousContact)) {
differences.push({ field: 'email', partner: partnerRecord.email, virtuous: extractPrimaryEmail(virtuousContact) });
}
if (partnerRecord.name !== virtuousContact.name) {
differences.push({ field: 'name', partner: partnerRecord.name, virtuous: virtuousContact.name });
}
// ... compare any other canonical fields
return differences;
}
```
Drift detection is expensive — one API call per paired record. Run it infrequently (weekly or monthly) and on a subset (e.g., contacts modified in the last 30 days) rather than the entire database.
### Resolution
Drift resolution depends on the source-of-truth model from [Build a Two-Way Sync](/crm/workflows/build-a-two-way-sync#source-of-truth-resolution):
* **Most-recent-wins**: apply the side with the later `modifiedDateTimeUtc`.
* **Per-field ownership**: re-apply the owning side's value to the non-owning side.
* **Virtuous-as-canonical**: pull the Virtuous value into your platform.
For records where the resolution is ambiguous (timestamps inconclusive, no clear ownership), mark them as `conflict` and surface to the customer for manual decision.
***
## Building a reconciliation report
Run a daily reconciliation report that summarizes the state of the sync. A reasonable shape:
| Metric | Healthy value | Investigate when |
| ------------------------------------------- | ------------------------------------------ | ---------------------------------------------------- |
| Total records in `pending` | 0 — being drained continuously | Stays > 0 for hours |
| Total records in `submitted` for > 24 hours | 0 — should be confirmed by then | Persistent >0 count indicates webhook or batch issue |
| Total records in `needs_review` | Near 0 with steady-state customer workflow | Growing count indicates matching-quality issue |
| Total records in `failed` | 0 — each should be investigated | Any non-zero count |
| Total in `conflict` (two-way only) | Near 0 with good ownership rules | Growing count indicates conflict-resolution issue |
| Latency: submitted → confirmed | A few hours typical (batch + webhook) | Sustained increases |
| Reconciliation-recovered count | Small fraction (\< 1%) of total syncs | Spikes indicate webhook delivery issues |
Surface this report to the customer's team and to your own ops dashboard. Most sync regressions show up as a metric anomaly first.
***
## Operational practices
### Daily reconciliation cadence
Run a daily reconciliation per customer:
1. Check for stuck Transactions older than 24 hours (Category 1).
2. Mark long-pending submissions as needs-review (Category 2).
3. Run missed-webhook reconciliation against modification-date queries (Category 3).
4. Generate the reconciliation report.
A more thorough run (drift detection) runs weekly.
### Escalation paths
Define escalation thresholds:
* **Auto-resolved.** A stuck Transaction recovered through reconciliation — log but no alert.
* **Customer-resolvable.** A needs-review item — surface in the customer's queue, no engineering alert.
* **Engineering investigation.** A spike in `failed` records, a sustained growth in `needs_review`, or drift across many records — alert your team for investigation.
Most reconciliation work should be operational rather than engineering — the daily report keeps the customer's team aware, and engineering involvement is rare.
### Audit trail
Persist every reconciliation outcome:
```sql theme={null}
CREATE TABLE virtuous_reconciliation_log (
id BIGSERIAL PRIMARY KEY,
customer_id TEXT NOT NULL,
run_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
category TEXT NOT NULL, -- 'stuck' | 'needs_update' | 'missed_webhook' | 'drift'
platform_record_id TEXT,
virtuous_record_id INTEGER,
resolution TEXT, -- 'auto_recovered' | 'marked_needs_review' | 'applied_inbound' | etc.
details JSONB
);
```
This log is invaluable for diagnosing regressions ("the missed-webhook count spiked on the 14th — what changed?") and for compliance ("show me every reconciliation action taken on this customer's data in Q4").
***
## End-to-end reconciliation runbook
When a customer reports a missing record:
Find your platform's record by the platform identifier. Capture its current `sync_state`, `last_sync_at`, and the `transactionSource`/`transactionId` (for Gifts) or `referenceSource`/`referenceId` (for Contacts).
`GET /api/Contact/{referenceSource}/{referenceId}` or `GET /api/Gift/{transactionSource}/{transactionId}`. If found, the data is in Virtuous — the issue is on your side or a missed webhook.
Look at your queue record. If `status: submitted` and submitted within the last 48 hours, the Transaction may still be in the nightly batch queue — wait. If older, suspect needs-update.
Notify the customer's team to check the Virtuous Imports / needs-update bucket. Once they resolve it, the resulting webhook captures the resolution.
Apply the Virtuous data through your inbound handler. Update the reconciliation log with the recovery action.
Apply the source-of-truth resolution rule. If ambiguous, mark `conflict` and surface to the customer.
This runbook handles the vast majority of "where's my record?" support questions. Keep it linked from your integration's support documentation.
***
## Where to go next
The one-way push architecture that this reconciliation supports.
The two-way architecture that this reconciliation also supports — including the drift category.
The companion workflow for handling matching failures that produce duplicates rather than needs-update.
The Transaction lifecycle and the needs-update bucket where unresolved Transactions land.