Webhook retries and dead-letter queues for agent email

Use webhook retries for temporary delivery failures and a dead-letter queue for events that exhaust a finite retry budget. Both mechanisms depend on the same consumer rule: commit each event once, keyed by its stable event ID, before returning a success response. Retries recover from outages. Dead letters make persistent failures inspectable and recoverable without turning an outage into data loss.

The distinction matters for an email agent. One inbound message may start an extraction, update a ticket, or wake an agent that can send a reply. A duplicate notification must not repeat those effects. A missing notification must not leave the message stranded forever.

The four failure points to design for#

A webhook request crosses more than one reliability boundary. Naming them makes the retry policy easier to reason about.

Failure point	Example	Expected result
Before the event is persisted	The sender cannot write its delivery record	Do not attempt delivery yet
Before the consumer accepts it	DNS, TLS, timeout, or a non-success HTTP response	Retry after backoff
After the consumer commits but before the sender sees the response	The connection drops after the database transaction	Retry the same event ID; consumer recognizes the duplicate
Every time the consumer handles it	A schema migration is missing or a permanent code bug rejects the payload	Stop after a finite budget and mark the event dead

The third row is why a 200 OK cannot prove exactly-once delivery. The consumer may have committed its work even though the sender did not receive the response. Retrying is the safe choice only when the consumer can identify the repeat.

The fourth row is why retries need an end. A poison event that can never succeed otherwise consumes workers, fills logs, and competes with healthy traffic.

Start with a durable outbox#

The sender should persist an event before the first network call. This is the transactional outbox pattern in its simplest form: the delivery record is the source of truth, while the immediate HTTP attempt is only a low-latency optimization.

Postfleet creates an outbox row for each message.received event, then tries the configured endpoint. If the inline attempt fails, a worker can recover the persisted row later. A process crash after persistence does not erase the notification.

Each event has an event_id that remains stable across automatic retries and manual redrive. Delivery timestamps and signatures change on each attempt, but the event's identity does not. That gives consumers one durable value to use for deduplication.

Postfleet currently uses this retry schedule after the first attempt:

Retry	Approximate delay
1	1 minute
2	5 minutes
3	30 minutes
4	2 hours
5	12 hours

Each delay includes up to 20 percent jitter, so endpoints recovering at the same time do not receive every queued event at once. A valid Retry-After value in delta seconds can replace the next scheduled delay, up to 24 hours. After six total attempts over roughly 15 hours, the event becomes dead.

Any 2xx response acknowledges the event. Network errors, timeouts, and every non-2xx response remain eligible for retry. The complete contract, headers, and payload are in the webhook documentation.

Make the consumer transaction idempotent#

Signature verification answers, "Did Postfleet send these bytes?" Deduplication answers a different question: "Have we already committed the effect represented by this event?" A production handler needs both.

Create a table with a unique key on the provider's event ID:

create table processed_webhook_events (
  event_id text primary key,
  processed_at timestamptz not null default now()
);

Then claim the event and write the business effect in one database transaction. The code below uses an intentionally generic database interface so the transaction boundary stays visible:

type MessageReceived = {
  event_id: string;
  type: "message.received";
  data: {
    message: {
      id: string;
      mailbox_id: string;
      subject: string;
      body: string;
    };
  };
};

async function acceptMessageReceived(event: MessageReceived) {
  return db.transaction(async (tx) => {
    const claimed = await tx.query(
      `insert into processed_webhook_events (event_id)
       values ($1)
       on conflict do nothing
       returning event_id`,
      [event.event_id],
    );

    if (claimed.rowCount === 0) {
      return { duplicate: true };
    }

    await tx.query(
      `insert into agent_jobs (source_event_id, message_id, mailbox_id)
       values ($1, $2, $3)`,
      [
        event.event_id,
        event.data.message.id,
        event.data.message.mailbox_id,
      ],
    );

    return { duplicate: false };
  });
}

The HTTP handler should read the exact raw body, verify the HMAC signature and timestamp, parse the JSON, run this transaction, and return a 2xx response only after the transaction commits. Postfleet signs ${timestamp}.${rawBody} with HMAC SHA-256. Re-serializing parsed JSON changes the bytes and invalidates the signature.

Use a short freshness window and compare timestamps in both directions. A timestamp far in the future is no fresher than one far in the past. Also use a constant-time signature comparison. The webhook verification example shows the required headers and signing format.

If the same request arrives after the transaction committed, the insert returns no row. The handler can acknowledge the duplicate without inserting a second job. If the transaction rolls back, the event ID is not left behind, so the next attempt can still do the work.

Acknowledge durable acceptance, not completed agent work#

Do not keep the webhook request open while an agent calls models, reads other systems, or waits for human approval. Verify the request, store the job durably, and acknowledge it. A worker can process the job separately.

This split avoids two common problems:

• Slow model calls do not exceed the webhook request timeout.
• An agent failure uses the job queue's retry policy instead of asking the webhook sender to deliver the same event again.

The webhook and job retries protect different hops. Postfleet owns delivery to your endpoint. Your queue owns work after acceptance. Keep separate attempt counts and failure states for each.

There is also a security reason to acknowledge after safe storage. A message with skipped_injection_risk is not a broken webhook. It is a delivered event whose comprehension status tells your application not to treat the body as cleared for extraction. Persist the event and route it according to policy, then return 2xx. Repeated transport delivery does not change the trust verdict.

The same rule applies to partial: durable acceptance succeeded even though a downstream comprehension stage did not fully complete. Email to JSON with AI explains why those failure states should remain visible to the workflow.

What belongs in the dead-letter queue#

A dead-letter queue is a holding area for events that used their retry budget. It is not a second inbox, and it should not become permanent storage that nobody checks.

For each dead event, an operator needs enough context to answer:

• Which endpoint and event ID failed?
• What was the last HTTP status or network error?
• How many attempts ran, and when did the last one finish?
• Did several events fail after the same deployment?
• Is the payload safe to replay after the consumer is repaired?

Postfleet exposes dead webhook deliveries in the dashboard. Manual redrive resets the delivery attempt budget and preserves the original event_id. That last property is important. If an earlier attempt actually committed but its response was lost, the repaired consumer still treats the redrive as a duplicate.

Redrive one event first. Confirm that the consumer commits it and that the duplicate guard works. Then increase the batch gradually. A large redrive can interleave with live traffic, so consumers must not depend on webhook arrival order.

AWS gives similar operational guidance for dead-letter queues and redrive: retain failed messages long enough to investigate them, start redrive conservatively, and expect replayed work to mix with new work. Postfleet's event identity behavior is its own contract, not an SQS behavior.

Retry mistakes that create duplicate agent work#

Returning success before durable storage is the most dangerous shortcut. A crash in the gap loses the event because the sender has already been told to stop.

Recording the event ID before the business write is almost as bad when they are separate transactions. If the write fails, the next attempt sees the event as processed and silently skips it.

Other failure patterns are easier to spot:

• Generating a new internal job without a unique source_event_id.
• Treating webhook order as message order.
• Doing model work inside the request and timing out after it succeeds.
• Returning an error for a valid event that policy intentionally quarantined.
• Redriving the entire dead set before testing one repaired event.
• Logging a failure without alerting anyone that dead events are accumulating.

Stripe's webhook guidance makes the same central point for consumers: duplicate events can arrive, handlers should log processed event IDs, and asynchronous processing keeps request handling reliable.

Monitor the recovery path before you need it#

A healthy endpoint can hide a broken recovery path for months. Track at least these signals:

• Delivery success rate and response latency by endpoint.
• Retry count and oldest pending event age.
• Dead event count and age.
• Duplicate claim rate at the consumer.
• Job queue lag after webhook acceptance.
• Redrive outcomes and repeated deaths for the same event.

Test the awkward cases deliberately. Drop the connection after the consumer commits. Return 429 with a reasonable Retry-After. Hold an endpoint offline through several retries. Send the same signed event twice. Repair a poison event and redrive it. The expected result is one durable business effect, even when the transport makes several attempts.

Current Postfleet limits#

Postfleet currently emits message.received webhooks. Delivery, bounce, complaint, and draft lifecycle events are not available yet. The retry worker uses a finite schedule rather than an infinite queue, and manual redrive is a dashboard operation.

Those limits make the consumer contract simpler, but they do not remove the need for deduplication. At-least-once delivery means a valid event can arrive more than once by design. A stable event ID, an atomic claim, and an inspectable dead state turn that ambiguity into a routine recovery path.

Sources#

• Postfleet webhook documentation
• Stripe webhook guidance
• AWS: using dead-letter queues
• AWS: dead-letter queue redrive
• Email to JSON with AI
• Email prompt injection