# Webhook delivery and retries

## Answering

Answer with any `2xx` status within 10 seconds. We ignore the body of your answer, but it must not be larger than 64 KB. Everything else counts as a failed attempt: another status, a redirect (we don't follow redirects), a timeout, a connection or TLS error, or a larger answer.

Answer first and do the work afterwards, for example by putting the event on a queue.

## When events are sent

We look for new events every minute, so an event usually arrives within a minute of the change. An event is sent until your endpoint accepts it or the retries run out (so you may receive an event more than once). Every endpoint subscribed to the event gets its own deliveries.

## Retries

| Attempt | When | After the first attempt |
|---|---|---|
| 1 | Usually within a minute of the change | — |
| 2 | 1 minute after attempt 1 failed | about 1 minute |
| 3 | 5 minutes after attempt 2 failed | about 6 minutes |
| 4 | 25 minutes after attempt 3 failed | about 31 minutes |
| 5 | 125 minutes after attempt 4 failed | about 2 h 36 min |
| 6 | 625 minutes after attempt 5 failed | about 13 h 1 min |

There are at most 6 attempts. When the last one fails, the delivery is marked failed. `X-Webhook-Attempt` says which attempt a request is.

## Automatic turn-off

After 50 failed attempts in a row (attempts, not events; test events don't count), we turn the endpoint off, mark its open deliveries as failed and notify you in the portal. Fix your receiver, then click **Turn on**. Events from while the endpoint was off are not sent again: catch up with the API, for example with [Incremental sync](https://docs.rephut.com/developers/incremental-sync).

## When API access is off

No new events are recorded. Deliveries already waiting are paused and continue when API access is back on; anything still undelivered 24 hours after the event is marked expired.

## Order and duplicates

- Events arrive in no particular order. A later change can arrive before an earlier one, and a retry arrives after newer events.
- An event can arrive more than once, for example when your answer got lost. Every attempt and every endpoint gets the same event `id`, also in `X-Webhook-Id`. Store the ids you processed and skip repeats.
- Treat an event as a hint: fetch the resource with `GET` and act on what it says now, not on the event's `status`.

## Test events

**Send test event** on an endpoint's page sends a `webhook.test` event to that endpoint at once and shows the result. Test events are not retried and don't count towards turning the endpoint off. You can send one every 30 seconds per endpoint, also to an endpoint that is turned off.

## Delivery log

The endpoint's page shows the last 50 deliveries with the time, the event, the status (Queued, Retrying, Delivered or Failed), the number of attempts and your endpoint's answer. We keep events and deliveries for 30 days. Failure reasons:

| Shown in the delivery log | What it means |
|---|---|
| Your endpoint answered HTTP 500. | Your endpoint answered with a status that isn't 2xx. |
| Your endpoint didn't answer within 10 seconds. | No answer in time. Answer first, then do the work. |
| Your endpoint answered with a redirect. Redirects aren't followed. | We don't follow redirects. Register the final URL. |
| Your endpoint's response was larger than 64 KB. | Send a short body, or none. |
| We couldn't find this host. | The host name doesn't resolve. |
| We couldn't connect to your endpoint. | Connection refused, reset or TLS failed. |
| The URL resolves to a private or local address, so we didn't send it. | The host resolves to a private, local or link-local address. Endpoints must be public. |
| Not delivered within 24 hours. | The event couldn't be delivered within the delivery window. |
| The endpoint was turned off. | The endpoint was turned off before the event was delivered. |
| We couldn't send the request. | Any other failure. |

## Firewalls

The addresses we send from can change, so check the signature rather than allow-listing IP addresses. Requests carry `User-Agent: Fulfillbot-Webhooks/1.0`.