Categories of sync failure
| Category | Symptom | Detection |
|---|---|---|
| Stuck Transactions | Submitted to Virtuous, accepted, but never produced a real Contact or Gift. | Records marked submitted in your queue with no matching webhook after a reasonable window. |
| Needs-update bucket | Gift Transactions that failed contact matching and require manual resolution in the Virtuous UI. | A 404 from GET /api/Gift/{transactionSource}/{transactionId} long after submission, combined with no giftCreate webhook. |
| Missed webhook deliveries | Records that exist in Virtuous but your platform was never notified of. | Records present in a Virtuous Query result but not in your platform’s database. |
| Direction-specific drift | A record exists on both sides but with different field values. | Field-by-field comparison during reconciliation surfaces mismatches. |
Category 1: stuck Transactions
A Transaction submitted to Virtuous is stuck if it was accepted (200 OK response from POST /api/v2/Gift/Transaction or POST /api/Contact/Transaction) but no giftCreate/contactCreate webhook has been delivered and no Gift/Contact can be looked up by the transactionSource/transactionId pair after a reasonable window.
The typical cause is webhook delivery loss during your endpoint’s outage combined with a missed reconciliation pass. The Transaction may have actually been resolved successfully; your records just never reflect it.
Detection
JavaScript
Resolution
For each record that returns a Gift on the reconciliation lookup, mark itconfirmed with a note that it was recovered through reconciliation rather than through the normal webhook path. The data is now consistent; the integration just had a delivery gap.
Run this reconciliation on a slow cadence (every few hours, with a 24-hour lookback window). It’s a backstop, not the primary outcome-detection mechanism.
Category 2: needs-update bucket
When the matching algorithm cannot confidently associate an incoming Gift Transaction with a Contact, the Transaction is moved to the needs-update bucket in Virtuous. The Gift doesn’t become a real Gift record — it sits in the bucket waiting for a Virtuous administrator to manually pick a Contact, create a new one, or discard the Transaction. From the partner integration’s perspective, a needs-update Transaction looks the same as a stuck Transaction during the reconciliation lookup:GET /api/Gift/{transactionSource}/{transactionId} returns 404 long after submission. The difference is that the Transaction will eventually be resolved (when the admin acts on it), not that it’s been silently lost.
Detection
If your reconciliation lookup returns404 and the submission is significantly past the nightly batch window (say, 72+ hours), the Transaction is most likely in needs-update:
JavaScript
Resolution
Surface needs-review records to your customer’s team:- In your integration’s UI, present a “needs Virtuous review” queue listing each record with its donor details and the relevant Stripe/Eventbrite/etc. reference.
- Include a link to the Virtuous UI’s Imports section where the customer can resolve the bucket.
- After the customer resolves the items, the resulting Gift
giftCreatewebhook will fire — your normal handler captures the resolution.
Category 3: missed webhook deliveries
Webhook deliveries can be lost during your endpoint’s outages, during prolonged unavailability, or after the retry budget exhausts. The lost events span every event type your integration subscribes to —contactCreate, contactUpdate, giftCreate, giftUpdate, and so on.
Detection
Periodic Query-based reconciliation surfaces missed events:JavaScript
Last Modified Date to catch both new gifts (modification timestamp at creation) and updates to existing ones. Run on a slower cadence than the webhook receiver — every 4–12 hours catches most gaps without consuming excessive rate-limit budget.
For Contacts, run the same pattern against POST /api/Contact/Query.
Resolution
Apply missed events through the same inbound-event handler your webhook receiver uses. The handler should be idempotent (see Idempotency and Safe Reprocessing) so that applying the same change a second time — once via the eventually-arriving webhook and once via reconciliation — is a no-op. If your handler isn’t idempotent, the reconciliation pass can produce double-applies. Test this scenario explicitly: replay the same event through the handler twice and confirm the second application produces identical state.Category 4: direction-specific drift
The hardest category to detect: a record exists on both sides but with different field values. The two systems disagree about the truth. Causes:- A sync that wrote partial data (only some fields succeeded).
- A bidirectional sync where conflict resolution went the wrong way for a particular field.
- A manual edit on one side that wasn’t captured by the other side’s sync.
Detection
Periodic field-by-field comparison between paired records:JavaScript
Resolution
Drift resolution depends on the source-of-truth model from Build a Two-Way Sync:- Most-recent-wins: apply the side with the later
modifiedDateTimeUtc. - Per-field ownership: re-apply the owning side’s value to the non-owning side.
- Virtuous-as-canonical: pull the Virtuous value into your platform.
conflict and surface to the customer for manual decision.
Building a reconciliation report
Run a daily reconciliation report that summarizes the state of the sync. A reasonable shape:| Metric | Healthy value | Investigate when |
|---|---|---|
Total records in pending | 0 — being drained continuously | Stays > 0 for hours |
Total records in submitted for > 24 hours | 0 — should be confirmed by then | Persistent >0 count indicates webhook or batch issue |
Total records in needs_review | Near 0 with steady-state customer workflow | Growing count indicates matching-quality issue |
Total records in failed | 0 — each should be investigated | Any non-zero count |
Total in conflict (two-way only) | Near 0 with good ownership rules | Growing count indicates conflict-resolution issue |
| Latency: submitted → confirmed | A few hours typical (batch + webhook) | Sustained increases |
| Reconciliation-recovered count | Small fraction (< 1%) of total syncs | Spikes indicate webhook delivery issues |
Operational practices
Daily reconciliation cadence
Run a daily reconciliation per customer:- Check for stuck Transactions older than 24 hours (Category 1).
- Mark long-pending submissions as needs-review (Category 2).
- Run missed-webhook reconciliation against modification-date queries (Category 3).
- Generate the reconciliation report.
Escalation paths
Define escalation thresholds:- Auto-resolved. A stuck Transaction recovered through reconciliation — log but no alert.
- Customer-resolvable. A needs-review item — surface in the customer’s queue, no engineering alert.
- Engineering investigation. A spike in
failedrecords, a sustained growth inneeds_review, or drift across many records — alert your team for investigation.
Audit trail
Persist every reconciliation outcome:End-to-end reconciliation runbook
When a customer reports a missing record:Confirm the record's intended state on your platform side
Find your platform’s record by the platform identifier. Capture its current
sync_state, last_sync_at, and the transactionSource/transactionId (for Gifts) or referenceSource/referenceId (for Contacts).Look up the record in Virtuous by external reference
GET /api/Contact/{referenceSource}/{referenceId} or GET /api/Gift/{transactionSource}/{transactionId}. If found, the data is in Virtuous — the issue is on your side or a missed webhook.If 404, check the submission state
Look at your queue record. If
status: submitted and submitted within the last 48 hours, the Transaction may still be in the nightly batch queue — wait. If older, suspect needs-update.If suspected needs-update, surface to the customer
Notify the customer’s team to check the Virtuous Imports / needs-update bucket. Once they resolve it, the resulting webhook captures the resolution.
If the record exists in Virtuous but is missing on your side
Apply the Virtuous data through your inbound handler. Update the reconciliation log with the recovery action.
Where to go next
Sync External Donations into Virtuous
The one-way push architecture that this reconciliation supports.
Build a Two-Way Sync
The two-way architecture that this reconciliation also supports — including the drift category.
Handle Duplicate Records
The companion workflow for handling matching failures that produce duplicates rather than needs-update.
Transactions
The Transaction lifecycle and the needs-update bucket where unresolved Transactions land.