Field notes — the blog

Notes from inside the system.

Specific write-ups on decisions that moved the needle: architecture trade-offs, delivery pipelines, and leading engineers who are better than you at their part of it.

              {{ totalPosts }} notes
              {{ totalMinutes }} min total
              Monthly
            

/

                Latest
                {{ featured.date }}
                {{ featured.cat }}
                {{ featured.read }}
              

{{ featured.title }}

{{ featured.excerpt }}

Read the note →

// before → after
{{ row.text }}

{{ listHeading }}

{{ resultCount }}

                {{ post.cat }}
                {{ post.date }}
              

{{ post.title }}

{{ post.excerpt }}

{{ post.read }}

Nothing matches that.

Try a different term, or clear the filters.

The architect's dispatch

One note a month. Nothing else.

A single deep write-up on a system I worked on: the trade-off, the numbers, what I'd do differently.

            {{ article.cat }}
            {{ article.date }}
            {{ article.read }}
          

{{ article.title }}

{{ article.excerpt }}

{{ part.text }}{{ part.text }}{{ part.text }}

{{ b.alt }} — {{ b.caption }}

{{ b.label }}

{{ h.text }}

{{ cell.text }}

{{ b.label }}

{{ link.label }} {{ link.note }} {{ link.host }}

{{ b.caption }}

{{ b.text }}

{{ b.text }}

{{ b.label }}

{{ line.text }}

{{ item.no }}

{{ part.text }}

{{ part.text }}

{{ part.text }}

Takeaways

▖ {{ t }}

Tags

{{ tag }}

← Newer

{{ prevPost.title }}

Older →

{{ nextPost.title }}

Related notes

{{ post.cat }}

{{ post.title }}

{{ post.read }}

RabbitMQ retries: the three patterns that actually work

Architecture · Apr 2026 · 9 min read

Naive requeue is an infinite loop with extra steps. Delay exchanges, capped backoff, and a poison lane cover nearly everything you will hit in production.

The first retry strategy everyone writes is nack with requeue set to true. It looks correct and it is a denial of service against your own broker: a permanently broken payload will spin as fast as the consumer can pick it up, burning CPU and pushing every other message behind it.

Pattern one: TTL delay exchanges

Make the wait explicit. On failure, publish the message to a delay queue with a per-message TTL and a dead-letter route back to the work queue. The broker holds it for you; no consumer sleeps, no thread is blocked, and the delay is visible in the topology rather than buried in application code.

work.queue        <-- consumers
  on failure: publish to retry.5s / retry.30s / retry.5m
retry.30s   ttl=30000  dlx=work.exchange
  after ttl expires -> back to work.queue

x-attempt: 3   # header, incremented by the consumer
attempt > 5 -> publish to poison.queue, ack original

Pattern two: cap the backoff, then park

Count attempts in a message header and cap them. Unbounded exponential backoff sounds prudent but eventually means a message retrying once a day forever, which nobody is watching. After the cap, move the message to a poison queue and acknowledge the original. Parking is not dropping.

A poison queue with a small dashboard beats a work queue you eventually purge in frustration. The parked message body is the best bug report you will get.

Pattern three: idempotent consumers

None of the above is safe unless a message can be processed twice without harm. At-least-once delivery means duplicates are a certainty, not an edge case: a consumer can finish its work and die before the ack lands. Key every side effect on something derived from the message, and make the second attempt a no-op.

Upserts keyed on a stable business identifier, not an auto-increment id.
An idempotency table for anything that leaves your system — emails, webhooks, payments.
Ack after the side effect, never before.

Three patterns, roughly two hundred lines of shared infrastructure code, and the retry conversation stops recurring in every code review.

Takeaways

Never requeue blindly; a TTL delay exchange gives you real backoff for free.
Count attempts in a header, cap them, then park in a poison queue you actually monitor.
Idempotency is the precondition that makes every other retry pattern safe.

All notes · Shehzad Aslam