> ## Documentation Index
> Fetch the complete documentation index at: https://docs.virtuous.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Polling Overview

> Why Volunteer integrations rely on polling — the lack of webhooks, the core pattern of scheduled queries with checkpoints and time filters, the trade-offs versus webhook-driven sync, and the architectural shapes for production polling.

The Volunteer API has no webhook surface. There's no `/webhooks` endpoint, no event subscription mechanism, no push notifications when records change in VOMO. For partner integrations that need to react to changes — sync new participations to a CRM, detect waiver completions, mirror Project schedules to external calendars — **polling is the only path**.

This page covers why that's the design, what polling looks like in production, and the architectural shapes that work well. The next pages in this group walk through the specific patterns for each resource family.

## Why polling

CRM+ and Raise both have rich webhook surfaces — subscribe to events, receive push notifications when they fire, react in near-real-time. The Volunteer API does not. Partner integrations need an alternative for change detection.

### What polling means

Polling is the pattern of **periodically querying the API for changes since the last query**. Instead of being told "this record changed," the integration asks "what has changed?" on a schedule:

```mermaid theme={null}
graph LR
  Cron["Scheduled job<br/>(every N minutes)"]
  Checkpoint["Last sync time<br/>(checkpoint)"]
  Query["GET /users?updated_after=&#123;checkpoint&#125;"]
  Process["Process changes"]
  Advance["Advance checkpoint"]
 
  Cron --> Query
  Checkpoint --> Query
  Query --> Process
  Process --> Advance
  Advance --> Checkpoint
```

The fundamental pattern: **schedule + filter + checkpoint**. A scheduled job runs at intervals, queries with a time filter for changes since the last checkpoint, processes the changes, and advances the checkpoint to the latest record seen.

### Trade-offs vs. webhooks

Polling and webhooks each have strengths:

| **Aspect** | **Webhooks (CRM+/Raise)** | **Polling (Volunteer)** |
| :- | :- | :- |
| **Latency** | Seconds | Minutes (depends on poll interval) |
| **Infrastructure** | Need public-facing endpoint, signature verification | Just outbound requests; no inbound infra |
| **Reliability** | Can miss events; need backstop | Detect events at next poll automatically |
| **API rate consumption** | Near-zero for detection | Each poll consumes budget |
| **Catching deletions** | Webhook fires on delete | Hard — deleted records don't appear in queries |
| **Setup complexity** | Subscribe, secure, route | Schedule the cron job |
| **Operating cost** | Hosting endpoint, queue, processing | Polling worker, throttling |

Polling is **simpler to set up** but **continuously costs more in API requests**. For most Volunteer integrations, the trade-off is acceptable — the data doesn't change second-to-second, and partner-side infrastructure can run polling cheaply.

***

## The core polling pattern

A minimal polling worker has four parts:

**1. Scheduler**

A cron job, scheduled task, or interval timer that fires the polling worker periodically.

**2. Checkpoint storage**

Per-customer persistent storage of "the last `updated_at` we successfully processed."

**3. Query**

`GET /users?updated_after={checkpoint}` (or the equivalent for whatever resource).

**4. Process and advance**

Process each result; advance the checkpoint to the latest `updated_at` actually seen.

```javascript JavaScript theme={null}
async function pollUserChanges(customerId) {
  // 1. Read the checkpoint
  const lastSync = await getCheckpoint(customerId, 'user_sync');

  // 2. Query for changes
  const params = new URLSearchParams({
    updated_after: lastSync.toISOString(),
  });

  let url = `https://api.vomo.org/v1/users?${params}`;
  let latestSeen = lastSync;

  // 3. Process pages
  while (url) {
    const response = await fetch(url, {
      headers: { Authorization: `Bearer ${process.env.VOMO_API_TOKEN}` },
    });
    if (!response.ok) throw new Error(`Poll failed: ${response.status}`);

    const page = await response.json();

    for (const user of page.data) {
      await processUserChange(customerId, user);

      const userUpdated = new Date(user.updated_at);
      if (userUpdated > latestSeen) latestSeen = userUpdated;
    }

    url = page.links.next;
  }

  // 4. Advance the checkpoint
  await setCheckpoint(customerId, 'user_sync', latestSeen);
}

// Run on a schedule
setInterval(() => pollUserChanges(customerId).catch(console.error), 15 * 60 * 1000);
```

This is the foundational shape. The pages that follow specialize it for specific resources and add layers (error handling, reconciliation, drift detection).

***

## Which resources support change detection

Most list endpoints accept `updated_after` (and the related time filters), making them suitable for polling:

| Resource | Supports `updated_after` | Notes |
| - | - | - |
| `GET /users` | ✓ | Primary User change-detection path |
| `GET /projects` | ✓ | Reflects Project metadata changes; schedule changes too |
| `GET /groups` | ✓ | Reflects metadata + member roster changes |
| `GET /campaigns` | ✓ | Campaigns change infrequently |
| `GET /certificates` | ✓ | Certificates change rarely |
| `GET /forms` | ✓ | Form structure changes |
| `GET /forms/{id}/completions` | ✓ | New Completions submitted |
| `GET /organizations` | ✗ | No time filter — Organizations change rarely; full re-read when needed |
| `GET /groups/{id}/members` | ✗ | No time filter — but the Group's `updated_at` reflects membership changes |
| `GET /projects/today` | ✗ | Daily snapshot, not a sync pattern |
| `GET /projects/date/{id}` | ✗ | Single resource — fetch when you have the ID |

The seven endpoints with full time filters cover the resources that change frequently and need detection. For the others, alternative strategies apply (full re-read with deduplication, transitive detection through the parent resource, etc.).

***

## What `updated_at` captures — and what it misses

The `updated_at` filter captures changes to the record itself. What changes count varies by resource:

| Resource | Changes that advance `updated_at` |
| - | - |
| User | Profile updates, membership status changes, possibly form completions |
| Project | Metadata changes, schedule changes (likely), policy changes |
| Group | Metadata changes, member roster changes |
| Form | Structure changes (fields added/removed), metadata |
| Form Completion | Initial submission; possibly modifications |

What `updated_at` typically does **not** capture:

| Change | Why it's missed |
| - | - |
| A new Participation is recorded | The Participation is created against a Project Date, not the User directly |
| A volunteer is checked in to a shift | Same — affects the Participation, not the User record |
| A Certificate is awarded to a user | The Certificate record is created separately |
| A Project Date is added to a Project | May or may not advance the Project's `updated_at` — confirm against live behavior |

These gaps matter for designing the polling architecture. If your integration needs to detect new participations, polling `/users` with `updated_after` is **not** sufficient — you'd need to poll Project Dates separately, or rely on the User's last-known participation list growing.

See [Detecting User Changes](https://docs.virtuous.org/volunteer/polling-and-sync/detecting-user-changes.md) for the User-specific patterns and [Reconciliation Patterns](https://docs.virtuous.org/volunteer/polling-and-sync/reconciliation-patterns.md) for what to do about the gaps.

***

## Poll cadence — how often to poll

The right interval depends on the business need and the rate budget:

| Cadence | Use case | Tradeoff |
| - | - | - |
| **Every 5 minutes** | Near-real-time workflows (e.g., live check-in displays) | High request volume; reserve for genuinely time-sensitive needs |
| **Every 15 minutes** | Standard sync (most external CRM/BI integrations) | Good default — fresh enough for most needs |
| **Every hour** | Reporting integrations, lower-stakes data flows | Lower request cost; acceptable lag for most reports |
| **Every 6-12 hours** | Bulk transfers, end-of-day reports | Minimal cost; same-day freshness |
| **Daily** | Backup, reconciliation, archival | Lowest cost; suitable for archival/audit |
| **More than once per day per Form for completion polling** | Probably wasteful unless the Form is high-volume | Most Forms see \<1 completion/day |

### Cadence by resource

Different resources benefit from different cadences. A reasonable per-resource breakdown:

| Resource | Suggested cadence |
| - | - |
| Users | Every 15-60 minutes |
| Projects | Hourly (schedules change infrequently) |
| Groups | Hourly to daily |
| Campaigns | Daily |
| Certificates | Daily or weekly |
| Forms (structure) | Daily |
| Form Completions | Every 15-60 minutes (per active Form) |

The cadence may also vary by customer — a customer with active daily operations needs faster polling than one with a once-a-week volunteer event.

### Why not just poll constantly?

Each poll consumes API request budget. The math:

* A 5-minute poll cadence = 288 polls/day per resource per customer
* Across 7 resources, that's \~2,000 polls/day per customer
* Add pagination (with 15-records-per-page) and the actual request count multiplies

For a partner integration serving many customers, polling every few minutes becomes the dominant API cost. Right-size the cadence to actual needs.

See [Rate Limits](https://docs.virtuous.org/volunteer/rate-limits.md) for the throttling pattern and [Change Detection Best Practices](https://docs.virtuous.org/volunteer/polling-and-sync/change-detection-best-practices.md) for the broader cost discussion.

***

## Architectural shapes

A few common patterns for organizing polling work:

### Shape 1: per-resource workers

One worker per resource, each running at its own cadence:

```mermaid theme={null}
graph TD
  Sched["Scheduler"]
  UserWorker["User poll worker<br/>(every 15 min)"]
  ProjectWorker["Project poll worker<br/>(every hour)"]
  GroupWorker["Group poll worker<br/>(every hour)"]
  FormWorker["Form Completion worker<br/>(every 30 min, per Form)"]
 
  Sched --> UserWorker
  Sched --> ProjectWorker
  Sched --> GroupWorker
  Sched --> FormWorker
```

Clean separation, easy to reason about, easy to scale individually. The downside: more cron schedules to manage, more checkpoint state per customer.

### Shape 2: unified poll cycle

One worker that polls multiple resources in sequence:

```mermaid theme={null}
graph LR
  Cron["Cron (every 15 min)"]
  Worker["Polling worker"]
  Users["Poll /users"]
  Projects["Poll /projects"]
  Groups["Poll /groups"]
  Process["Process changes"]
 
  Cron --> Worker
  Worker --> Users
  Users --> Projects
  Projects --> Groups
  Groups --> Process
```

Simpler scheduling, easier to monitor as a single workflow. The downside: every resource polls at the same cadence even when faster or slower would be more efficient.

For most partner integrations, **Shape 2 (unified cycle) at 15-30 minute intervals** is a reasonable starting design. Move to per-resource workers if you find that one resource needs significantly different cadence than the others.

### Shape 3: event-driven within a workflow

For integrations where changes in one resource trigger queries in others:

```mermaid theme={null}
graph LR
  UserPoll["Poll /users<br/>for changed users"]
  PerUser["For each changed user..."]
  UserDetail["GET /users/&#123;id&#125;<br/>for participations"]
  Project["GET /projects/&#123;participation.project_id&#125;<br/>for project context"]
 
  UserPoll --> PerUser
  PerUser --> UserDetail
  UserDetail --> Project
```

The User-change poll triggers cascading fetches for related data. Useful when the partner integration needs to react to the full context of each change.

The downside: many more API requests. Use sparingly and cache aggressively.

***

## The role of reconciliation

Polling alone misses some cases:

| **Missed case** | **Example** |
| :- | :- |
| A record was deleted | Soft-deleted records don't appear in `updated_after` queries |
| A record was modified, polled, processed — but the processing failed silently | The checkpoint advanced; the record is "done" but the destination is missing it |
| A field changed in a way that doesn't advance `updated_at` (rare but possible) | Subtle data drift |
| The polling worker crashed mid-batch and missed some records | Possible if checkpoint advancement is naive |

The mitigation is **reconciliation** — a separate, less-frequent process that audits the actual state against expected state and catches gaps:

| **Reconciliation cadence** | **What it does** |
| :- | :- |
| Daily | Re-fetches all records updated yesterday; compares to what was processed; re-queues gaps |
| Weekly | Same, but for the past week — catches longer-tail issues |
| Per-customer-onboarding | Initial full-state read to seed the integration |

The combination of polling + daily reconciliation produces **most** integrations' production-grade reliability. See [Reconciliation Patterns](https://markdowntorichtext.com/volunteer/polling-and-sync/reconciliation-patterns) for the full pattern.

***

## Polling vs. one-time sync

Polling is for ongoing change detection. For one-time operations — initial onboarding, migration, full data dump — use a separate **full-state sync** pattern:

| **Pattern** | **When** |
| :- | :- |
| **Initial sync** (one-time, full dataset) | Customer onboards; integration starts |
| **Polling** (ongoing, incremental) | Steady-state operation |
| **Reconciliation** (periodic, full-or-partial dataset) | Verifying polling integrity |

The three patterns layer:

```mermaid theme={null}
graph LR
  Onboard["Customer<br/>onboarding"]
  Initial["Initial full sync"]
  Polling["Steady-state polling"]
  Reconcile["Periodic<br/>reconciliation"]
 
  Onboard --> Initial
  Initial --> Polling
  Polling --> Reconcile
  Reconcile --> Polling
```

See [Sync Architecture Patterns](https://markdowntorichtext.com/volunteer/best-practices/sync-architecture-patterns) for the broader picture.

***

## Common pitfalls

A few mistakes that surface in production polling integrations:

### Pitfall 1: advancing the checkpoint to "now" instead of to the latest record

```javascript theme={null}
// ❌ Anti-pattern
const startTime = new Date();
await pollUsers();
await setCheckpoint(startTime); // Skips any records that arrive during the poll
```

The window between "start of poll" and "end of poll" can be minutes. Records modified in that window have `updated_at` between `startTime` and `endTime` — but the next poll uses `startTime` as the lower bound, missing them.

The right pattern is to advance to **the latest** `updated_at` **actually observed in the response**:

```javascript theme={null}
// ✅ Correct
let latestSeen = lastCheckpoint;
for (const user of users) {
  const u = new Date(user.updated_at);
  if (u > latestSeen) latestSeen = u;
  await processUser(user);
}
await setCheckpoint(latestSeen);
```

This guarantees no records are missed, even if the poll spans a long window or processes many updates.

### Pitfall 2: processing failures silently advance the checkpoint

If your worker processes records, encounters an error mid-batch, but still advances the checkpoint, the missed records are gone:

```javascript theme={null}
// ❌ Anti-pattern
for (const user of users) {
  try {
    await processUser(user);
  } catch (err) {
    console.error('Failed:', err); // Logged but not handled
  }
}
await setCheckpoint(latestSeen); // Advances even though some failed
```

Either fail the whole batch (so the next run retries everything) or queue failures for later retry while still tracking them:

```javascript theme={null}
// ✅ Correct
const failures = [];
for (const user of users) {
  try {
    await processUser(user);
  } catch (err) {
    failures.push({ user, error: err });
  }
}
 
if (failures.length > 0) {
  await deadLetterQueue.publish(failures);
  await alertOps({ failureCount: failures.length });
}
 
await setCheckpoint(latestSeen); // Safe — failures are tracked
```

### Pitfall 3: not throttling polling against the rate budget

Polling that runs concurrently with other integration work can exhaust the rate budget. Use a shared throttled client:

```javascript theme={null}
const client = new ThrottledVomoClient({ token, requestsPerSecond: 5 });
 
// All polling workers share this client — rate limit is enforced globally
```

See [Rate Limits](https://markdowntorichtext.com/volunteer/rate-limits).

### Pitfall 4: polling too frequently for the use case

A 1-minute poll cadence consumes 60× the rate budget of an hourly one. Audit your polling cadences against actual freshness requirements — often "near-real-time" is a comfort goal, not a hard need.

***

## Production checklist

Before deploying polling-based change detection:

* Each resource has a defined polling cadence aligned with business needs
* Checkpoints are persisted per-customer in durable storage (not just in-memory)
* Checkpoint advancement uses the latest `updated_at` actually seen, not wall-clock time
* Failures during processing don't silently advance the checkpoint
* Failed records go to a dead-letter queue for retry / investigation
* Rate-limit-aware throttling is in place (shared across polling workers)
* A daily reconciliation pass catches missed records
* Per-customer monitoring tracks: poll success rate, processed-records counts, lag time (max `updated_at` to "now"), and dead-letter queue depth
* Alerts fire on: sustained `5xx`/`429` errors, stalled checkpoints, growing DLQ
* Customer onboarding includes an initial full sync, not just turning polling on

***

## Where to go next

> [card] [Detecting User Changes](https://docs.virtuous.org/volunteer/polling-and-sync/detecting-user-changes.md)
>
> The User-specific polling pattern with checkpoint advancement and the participation-detection caveat.

> [card] [Detecting Project Changes](https://docs.virtuous.org/volunteer/polling-and-sync/detecting-project-changes.md)
>
> The Project-specific polling pattern with schedule-change considerations.

> [card] [Reconciliation Patterns](https://docs.virtuous.org/volunteer/polling-and-sync/reconciliation-patterns.md)
>
> The companion pattern that catches what polling misses.

> [card] [Change Detection Best Practices](https://docs.virtuous.org/volunteer/polling-and-sync/change-detection-best-practices.md)
>
> The cross-cutting practices: checkpointing, idempotency, drift detection.


## Related topics

- [Detecting User Changes](https://docs.virtuous.org/volunteer/polling-and-sync/detecting-user-changes.md)
- [Detecting Project Changes](https://docs.virtuous.org/volunteer/polling-and-sync/detecting-project-changes.md)
- [Reconciliation Patterns](https://docs.virtuous.org/volunteer/polling-and-sync/reconciliation-patterns.md)
- [Overview](https://docs.virtuous.org/volunteer/overview.md)
