Field notes — the blog

Notes from inside the system.

Specific write-ups on decisions that moved the needle: architecture trade-offs, delivery pipelines, and leading engineers who are better than you at their part of it.

              {{ totalPosts }} notes
              {{ totalMinutes }} min total
              Monthly
            

/

                Latest
                {{ featured.date }}
                {{ featured.cat }}
                {{ featured.read }}
              

{{ featured.title }}

{{ featured.excerpt }}

Read the note →

// before → after
{{ row.text }}

{{ listHeading }}

{{ resultCount }}

                {{ post.cat }}
                {{ post.date }}
              

{{ post.title }}

{{ post.excerpt }}

{{ post.read }}

Nothing matches that.

Try a different term, or clear the filters.

The architect's dispatch

One note a month. Nothing else.

A single deep write-up on a system I worked on: the trade-off, the numbers, what I'd do differently.

            {{ article.cat }}
            {{ article.date }}
            {{ article.read }}
          

{{ article.title }}

{{ article.excerpt }}

{{ part.text }}{{ part.text }}{{ part.text }}

{{ b.alt }} — {{ b.caption }}

{{ b.label }}

{{ h.text }}

{{ cell.text }}

{{ b.label }}

{{ link.label }} {{ link.note }} {{ link.host }}

{{ b.caption }}

{{ b.text }}

{{ b.text }}

{{ b.label }}

{{ line.text }}

{{ item.no }}

{{ part.text }}

{{ part.text }}

{{ part.text }}

Takeaways

▖ {{ t }}

Tags

{{ tag }}

← Newer

{{ prevPost.title }}

Older →

{{ nextPost.title }}

Related notes

{{ post.cat }}

{{ post.title }}

{{ post.read }}

Why we replaced 14 cron jobs with a single event bus

Architecture · Jul 2026 · 8 min read

Scheduled jobs are the cheapest way to build a distributed system and the most expensive way to run one. Here is the migration that closed our nightly failure window.

Every cron job is an implicit contract: this thing will have finished by the time that thing starts. We had fourteen of those contracts spread across three services, and none of them were written down anywhere except in the crontab's whitespace.

It worked for two years. Then a client tripled their catalogue, the nightly import went from nine minutes to fifty, and the index rebuild started reading a half-written table. We found out from the support queue at 9am, which is the worst monitoring system money can buy.

What the schedule was actually encoding

The first useful exercise was writing down what each job assumed. Not what it did — what it needed to already be true. Of the fourteen jobs, four had a real ordering dependency. The other ten were scheduled at staggered times purely so they would not collide on CPU.

Ten of the fourteen jobs did not need a schedule at all. They needed to know when something had happened.

One rule: publish facts, not commands

We moved to RabbitMQ with a single design rule that settled most arguments before they started. A producer publishes a fact about something that already happened, in the past tense, and it does not know or care who consumes it. No job ever tells another job to run.

# 14 crons across 3 services
0 2 * * *  php artisan import:feed
0 3 * * *  php artisan index:rebuild   # hopes import is done
0 4 * * *  php artisan notify:digest   # hopes index is done

# one bus
publish  catalogue.import.completed
  |- search-indexer     (retry x3, backoff)
  |- digest-builder     (retry x3, backoff)
  '- audit-log          (dlq on failure)

The four real orderings stayed explicit: the indexer publishes its own completion event, and the digest builder waits for that rather than for a clock. Everything else became parallel by default, which turned out to be the throughput win we were not looking for.

The part that actually mattered

The payoff was not speed. It was that failure became a thing you can see. A stuck consumer is a queue with depth on a dashboard, and the message is still sitting there, replayable once you ship the fix. A failed cron job is silence until someone complains.

Median end-to-end catalogue freshness went from 4h 20m to 11m.
Nightly incidents attributable to job ordering: 6 in the prior quarter, 0 in the two since.
Time to recover a failed import step: from a full re-run to a single message replay.

The migration took three weeks, most of it spent making consumers idempotent. That work would have been necessary eventually anyway — the bus just made it non-optional, which is the useful kind of forcing function.

Takeaways

Write down what each scheduled job assumes is already true. Most of those assumptions are not real dependencies.
Publish facts in the past tense; let consumers decide what to do about them.
Dead-letter queues convert overnight silence into a replayable Monday morning.

All notes · Shehzad Aslam