Lists, cursors and the change feed

<!-- order: 3 | Paging through lists, and keeping a copy of the catalog current without re-pulling it. -->

Pages

Lists return a page and a cursor:

{ "data": [ … ], "next_cursor": "01a18995-a4f3-755a-9e8f-383bc25aa232" }

Pass next_cursor as cursor for the next page; when it's null, you have everything. limit sets the page size (up to 200). Cursors don't expire, and pages don't skip or repeat items when the data changes between calls.

let cursor: string | null = null
do {
  const url = new URL("https://api.foodintegrityproject.org/v2/catalog/products")
  url.searchParams.set("limit", "200")
  if (cursor) url.searchParams.set("cursor", cursor)
  const page = await (await fetch(url, { headers: { Authorization: `Bearer ${process.env.FIP_TOKEN}` } })).json()
  for (const product of page.data) save(product)
  cursor = page.next_cursor
} while (cursor)

Only what changed

Every list takes updated_since (an ISO 8601 time) and returns only what changed since then.

The change feed

/v2/catalog/changes lists every listing change, recorded every 15 minutes: listed (newly listed), changed (its certificate, dates, status or name changed) and delisted. Each has a cursor.

1. Export the catalog once.
2. Call /v2/catalog/changes?since=0 and keep the next_cursor it returns.
3. From then on, call /v2/catalog/changes?since=<the cursor you kept> and apply what comes back. A delisted change has no product: remove it from your copy.