Field notes — the blog

Notes from inside the system.

Specific write-ups on decisions that moved the needle: architecture trade-offs, delivery pipelines, and leading engineers who are better than you at their part of it.

              {{ totalPosts }} notes
              {{ totalMinutes }} min total
              Monthly
            

/

                Latest
                {{ featured.date }}
                {{ featured.cat }}
                {{ featured.read }}
              

{{ featured.title }}

{{ featured.excerpt }}

Read the note →

// before → after
{{ row.text }}

{{ listHeading }}

{{ resultCount }}

                {{ post.cat }}
                {{ post.date }}
              

{{ post.title }}

{{ post.excerpt }}

{{ post.read }}

Nothing matches that.

Try a different term, or clear the filters.

The architect's dispatch

One note a month. Nothing else.

A single deep write-up on a system I worked on: the trade-off, the numbers, what I'd do differently.

            {{ article.cat }}
            {{ article.date }}
            {{ article.read }}
          

{{ article.title }}

{{ article.excerpt }}

{{ part.text }}{{ part.text }}{{ part.text }}

{{ b.alt }} — {{ b.caption }}

{{ b.label }}

{{ h.text }}

{{ cell.text }}

{{ b.label }}

{{ link.label }} {{ link.note }} {{ link.host }}

{{ b.caption }}

{{ b.text }}

{{ b.text }}

{{ b.label }}

{{ line.text }}

{{ item.no }}

{{ part.text }}

{{ part.text }}

{{ part.text }}

Takeaways

▖ {{ t }}

Tags

{{ tag }}

← Newer

{{ prevPost.title }}

Older →

{{ nextPost.title }}

Related notes

{{ post.cat }}

{{ post.title }}

{{ post.read }}

Build a RabbitMQ retry pipeline in Laravel, end to end

Architecture · Aug 2026 · 21 min read

A complete walkthrough: topology, publisher, consumer contract, delay-exchange backoff, a poison lane, idempotency, and the tests that prove it.

This is the full build, not the summary. By the end you will have a working consumer that retries with backoff, gives up in a way you can inspect on Monday morning, and survives the same message arriving twice. Everything here is code I have shipped, with the Laravel-specific parts marked so you can lift the pattern into another stack.

The prerequisite is a running RabbitMQ and a Laravel app. I am using php-amqplib directly rather than a queue driver, because the whole point is controlling the topology, and a driver that abstracts the topology away is the wrong tool for this job.

Publisher to app.events to the work queue and consumer. A rejected message dead-letters to app.retry, waits in a TTL delay queue, and returns to app.events. After eight attempts it routes through app.dead into a poison queue. — The whole topology. Everything below is one box in this picture.

What we are building

A topic exchange that carries facts, with consumers binding to the ones they care about.
A retry path built from a delay queue and a dead-letter route, so no consumer ever sleeps.
Capped attempts, after which the message lands in a poison queue that a human can read.
An idempotency guard, because at-least-once delivery means duplicates are a certainty, not a risk.
Enough instrumentation that a stuck consumer is a number on a dashboard rather than a support ticket.

Step 1: declare the topology in code, once

Declare everything at boot, from a single class, and make declaration idempotent. The failure mode you are avoiding is the one where staging and production drift because somebody created a queue by hand in the management UI eighteen months ago and nobody remembers the arguments they used.

// three exchanges, and every queue is created from this one place
$ch->exchange_declare('app.events', 'topic', false, true, false);
$ch->exchange_declare('app.retry',  'topic', false, true, false);
$ch->exchange_declare('app.dead',   'topic', false, true, false);

// work queue: failures are routed out to app.retry, not requeued
$ch->queue_declare('search-indexer', false, true, false, false, false,
  new AMQPTable([
    'x-dead-letter-exchange'    => 'app.retry',
    'x-dead-letter-routing-key' => 'search-indexer',
  ]));
$ch->queue_bind('search-indexer', 'app.events', 'catalogue.#');

// delay queue: holds a message for its TTL, then dead-letters it
// straight back onto the work queue. the broker does the waiting.
$ch->queue_declare('search-indexer.delay', false, true, false, false, false,
  new AMQPTable([
    'x-dead-letter-exchange'    => 'app.events',
    'x-dead-letter-routing-key' => 'catalogue.replay',
  ]));
$ch->queue_bind('search-indexer.delay', 'app.retry', 'search-indexer');

// poison lane: nothing is bound onwards. it sits here until read.
$ch->queue_declare('search-indexer.poison', false, true, false, false);
$ch->queue_bind('search-indexer.poison', 'app.dead', 'search-indexer');

Note what is not there: no consumer calls sleep, and no application code holds a timer. The delay is a property of a queue, which means it is visible in the management UI, survives a deploy, and costs you no worker threads.

Step 2: publish facts, in the past tense

One rule settles most design arguments before they start: a producer publishes a statement about something that has already happened, and it does not know who consumes it. If you find yourself writing a routing key like rebuild.index, you have published a command, and you have coupled two services that did not need to be coupled.

public function publish(string $key, array $payload): void
{
    $msg = new AMQPMessage(json_encode([
        'id'          => (string) Str::uuid(),  // the idempotency key
        'occurred_at' => now()->toIso8601String(),
        'payload'     => $payload,
    ]), [
        'content_type'  => 'application/json',
        'delivery_mode' => AMQPMessage::DELIVERY_MODE_PERSISTENT,
    ]);

    $this->channel->basic_publish($msg, 'app.events', $key);
}

// publish('catalogue.import.completed', ['catalogue_id' => 42])
// NOT publish('rebuild.index', ...) — that is a command wearing a hat

The id is generated by the producer, once, and it never changes across retries. That single field is what makes step 6 possible, and adding it later means backfilling every producer under time pressure. Put it in on day one.

Step 3: give the consumer one job and a clear contract

A consumer either succeeds, fails in a way that is worth retrying, or fails in a way that never will be. Most bugs in message handling come from treating the second and third cases as the same thing. A malformed payload will be exactly as malformed on the fortieth attempt.

$ch->basic_qos(null, 10, null);   // prefetch. do not skip this line.

$ch->basic_consume('search-indexer', '', false, false, false, false,
  function (AMQPMessage $m) {
    try {
        $this->handle(json_decode($m->body, true, 512, JSON_THROW_ON_ERROR));
        $m->ack();
    } catch (PermanentFailure|JsonException $e) {
        // never going to work. straight to the poison lane.
        $this->poison($m, $e);
        $m->ack();
    } catch (Throwable $e) {
        // transient: reject WITHOUT requeue so the dead-letter
        // rule sends it to app.retry. requeue=true is the trap.
        $m->reject(false);
    }
  });

Two lines deserve a comment each. The prefetch cap stops one worker from claiming the entire queue and leaving its colleagues idle. And reject with requeue set to false is the whole retry mechanism: the message is not thrown away, it is dead-lettered onto app.retry, which is where the delay queue is waiting for it.

Naive nack-with-requeue is a denial of service you wrote against your own broker. A permanently broken message will spin as fast as your consumer can pick it up.

Step 4: back off, without anybody sleeping

A fixed retry delay is better than none, but it synchronises your failures: a downstream service that falls over comes back to the entire backlog arriving at once. Growing the delay per attempt spreads the recovery out, and a per-message TTL lets you do that with one delay queue rather than one queue per step.

$attempt = $this->attempts($m);            // from x-death, see below
$delayMs = min(60000, (2 ** $attempt) * 1000);   // 1s, 2s, 4s ... 60s

$copy = new AMQPMessage($m->body, $m->get_properties());
$copy->set('expiration', (string) $delayMs);
$ch->basic_publish($copy, 'app.retry', 'search-indexer');
$m->ack();   // the original is done; the copy carries the work

// attempts() reads the x-death header rabbit maintains for you:
//   $m->get('application_headers')['x-death'][0]['count']
// no counter of your own to keep in sync, and it survives restarts.

There is one sharp edge worth knowing about. A per-message TTL only expires a message when it reaches the head of the queue, because RabbitMQ inspects queues from the front. Mixed TTLs in one delay queue therefore expire in publish order, not in delay order. If you need strictly ordered wakeups, use one delay queue per tier — 1s, 10s, 60s — and route by attempt count. For a backoff ladder, publish order is fine and one queue is simpler.

Step 5: give up properly

Infinite retries are how a queue quietly fills a disk. Cap the attempts, and when you hit the cap, move the message somewhere a person can look at it with the failure attached. The goal is that Monday morning starts with a list rather than an investigation.

if ($attempt >= 8) {                      // ~2 minutes of ladder
    $headers = new AMQPTable([
        'x-failure'      => substr($e->getMessage(), 0, 500),
        'x-failed-at'    => now()->toIso8601String(),
        'x-attempts'     => $attempt,
        'x-consumer'     => 'search-indexer',
    ]);
    $dead = new AMQPMessage($m->body, $m->get_properties());
    $dead->set('application_headers', $headers);
    $ch->basic_publish($dead, 'app.dead', 'search-indexer');
    $m->ack();
    return;
}

# replaying after a fix is a shovel, not a rewrite:
$ php artisan messaging:replay search-indexer.poison --limit=500

Write the replay command on the same day you write the poison queue. A dead-letter queue you cannot drain is just a slower way to lose messages, and the command is about thirty lines: consume from the poison queue, republish to app.events with the original routing key, stop at the limit.

Step 6: make the handler idempotent, because it will run twice

At-least-once delivery is not a caveat in the documentation, it is a description of Tuesday. A network blip between your ack and the broker means the message is redelivered, and your handler runs again on work it already did. The cheapest reliable guard is a table with a unique constraint and a database that enforces it.

Schema::create('processed_messages', function (Blueprint $t) {
    $t->uuid('message_id');
    $t->string('consumer');
    $t->timestamp('processed_at');
    $t->unique(['message_id', 'consumer']);   // the entire mechanism
});

DB::transaction(function () use ($msg) {
    // insert first: if this throws, we have seen it, so stop.
    ProcessedMessage::create([
        'message_id' => $msg['id'], 'consumer' => 'search-indexer',
        'processed_at' => now(),
    ]);
    $this->reindex($msg['payload']);   // same transaction. this matters.
});

// catch QueryException with a duplicate-key code -> ack and move on.

Both statements must be in the same transaction. If the insert commits and the work then fails, you have permanently marked a message as done that never was — and the retry you carefully built will now skip it. I have shipped that bug, and it is invisible until the day it matters.

Prune the table on a schedule. Message ids older than your longest possible retry window plus a margin can go; seven days is a comfortable default and keeps the unique index small.

delivery guarantees, and what each one costs you
Guarantee	What you must build	Cost
At most once	nothing — ack before you work	silent data loss on any crash
At least once	idempotent handlers (step 6)	duplicate work, guarded
Exactly once	not available across a network	a claim, not a guarantee

Step 7: what to put on a dashboard

The reason to build this rather than use scheduled jobs is that failure becomes observable. That only pays off if somebody is actually looking, so wire these four before you call it done.

Depth of each work queue. A number that climbs and does not come down is the earliest honest signal you get.
Depth of each poison queue, alerting on any value above zero. This should be rare enough to be worth waking up for.
Consumer count per queue. Zero consumers on a queue with depth is a deploy that half-happened.
Redelivery rate. A rise here usually means a downstream dependency is wobbling before it fully breaks.

Step 8: prove it, with tests that would actually fail

Test the retry path against a real broker. Mocking the broker tests your understanding of RabbitMQ, which is precisely the thing in doubt. A container in CI is cheap and catches the topology mistakes that unit tests cannot see.

public function test_a_transient_failure_is_retried_then_succeeds(): void
{
    $this->failNextHandles(2);         // blow up twice, then work
    $this->publish('catalogue.import.completed', ['id' => 42]);

    $this->runConsumerFor(seconds: 8);

    $this->assertQueueEmpty('search-indexer');
    $this->assertQueueEmpty('search-indexer.poison');
    $this->assertHandledTimes(3);
}

// the three that have caught real bugs for me:
//   permanent failure goes to poison on the FIRST attempt
//   the same message id delivered twice does the work once
//   attempt 9 lands in poison with x-failure populated

Keep the ladder short in tests. An 8-attempt exponential backoff takes over two minutes of real time, so make the base delay configurable and set it to 50ms under test. The behaviour you are asserting is the routing, not the arithmetic.

What this cost, and what it bought

On the migration I keep referring back to, the pipeline itself was about two days of work. Making the consumers idempotent was three weeks, and that is the honest number nobody puts in the tutorial. It was also work that needed doing regardless — the bus just made it non-optional, which is the useful kind of forcing function.

What we got: median catalogue freshness from 4h 20m to 11m, six ordering-related incidents in the prior quarter down to zero in the two since, and recovery from a failed import going from a full re-run to a single replay command. None of that came from the queue being fast. It came from failure being something you can see and re-do.

Takeaways

Declare the whole topology in one class, and let the broker own retry delays instead of your workers.
Separate transient from permanent failure in the consumer; only the first kind belongs on the retry path.
Give every message a producer-generated id and guard the handler with a unique constraint, in the same transaction as the work.
Cap attempts, route to a poison queue with the error attached, and write the replay command the same day.

All notes · Shehzad Aslam