No Kafka in V1: the complexity we refused to buy

We run a modular monolith. Our async work goes through a Postgres table, polled with row-level locking, rather than a message broker. There is no service mesh, no Kafka, and no distributed tracing across services that do not exist.

This is not an argument that microservices are wrong. It is an argument that most complexity in modern software is not solving a problem the company has — it is solving a problem the company has read about.

The opinion is the least interesting part, so here are the mechanics.

The table

The whole queue is one table and one index. This is the migration, unedited:

create table job_queue (
    id            uuid primary key default gen_random_uuid(),
    queue_name    text not null,
    payload       jsonb not null,
    status        text not null default 'PENDING'
                    check (status in ('PENDING','RUNNING','SUCCESS',
                                      'FAILED','RETRYING','CANCELLED')),
    attempts      int not null default 0,
    max_attempts  int not null default 5,
    available_at  timestamptz not null default now(),
    locked_at     timestamptz,
    locked_by     text,
    error_message text,
    created_at    timestamptz not null default now(),
    updated_at    timestamptz not null default now()
);

create index idx_job_queue_claim
    on job_queue (queue_name, status, available_at);

Two details worth pausing on. available_at instead of a separate scheduled-jobs mechanism means delayed work and retried work are the same thing — a retry is a job that becomes available later, and there is one code path for both. And the index column order matches the claim query’s predicate order, which is what keeps the claim from degrading into a sequential scan as the table grows.

The claim

The part people expect to be hard is giving exactly one worker each job. Postgres does it in one statement:

update job_queue
set status = 'RUNNING',
    locked_at = now(),
    locked_by = ?,
    attempts = attempts + 1,
    updated_at = now()
where id = (
    select id from job_queue
    where queue_name = ?
      and status in ('PENDING', 'RETRYING')
      and available_at <= now()
    order by available_at
    for update skip locked
    limit 1
)
returning id, queue_name, payload, status, attempts,
          max_attempts, available_at, correlation_id;

for update skip locked is the load-bearing clause, and it is the single most useful Postgres feature most application engineers have never used. It says: take a row lock, and if a row is already locked by another transaction, do not wait — skip it and take the next one.

Without skip locked, ten workers polling the same queue serialise. Each waits behind the first for the same hot row and your effective concurrency is one. With it, ten workers claim ten different rows in parallel with no coordination service, no lease protocol, and no consensus algorithm. The returning clause makes claim-and-read a single round trip, so there is no window between selecting a job and marking it running.

The property that matters most, though, is the one you get for free: the enqueue is a plain insert into the same database as your business data. It participates in the same transaction.

Teams with a broker solve dual-write with an outbox table, a relay process, and deduplication. We solve it by not having the problem.

That removes an entire category of bug — the one where the database write commits and the corresponding message does not, so your data says the order exists and your queue says nothing needs to happen to it. If you have ever debugged an order that is real but unprocessed, this is the shape of the bug you were chasing.

Retries, backoff, and the poison job

A queue without a considered failure path is a queue that loses work. Failure moves the job to RETRYING with exponential backoff until attempts reach the limit, at which point it is FAILED and waits for a human:

set status = case when attempts >= max_attempts
                  then 'FAILED' else 'RETRYING' end,
    available_at = case when attempts >= max_attempts
                        then available_at
                        else now() + (least(power(2, attempts), 300)
                                      * interval '1 second') end

Three decisions are encoded there. The backoff doubles, so a struggling dependency is not hammered. It caps at five minutes, because uncapped exponential backoff schedules a retry days out, which reads as data loss to everyone except the person who wrote it. And FAILED is terminal by design — a job that has failed five times will not succeed on the sixth, and retrying it forever converts one broken job into permanent background load. Recovering it is an explicit requeue that resets attempts, so trying again is someone’s deliberate act rather than the system’s default optimism.

The genuinely hard failure is the one everybody hits eventually: a worker claims a job, marks it RUNNING, and then dies. The row is held by nobody, owned by a process that no longer exists, and it will sit there forever. That is why locked_at and locked_by exist, and why a reaper sweeps rows running longer than any job plausibly should and returns them to the queue.

Every at-least-once queue needs a way to reclaim work from a dead worker. The brokers that hide this have not removed the problem — they have chosen a timeout on your behalf.

Ours is visible in a column we can query, which means the answer to “what is stuck?” is a select rather than a support ticket.

What a broker would have cost us

A message broker is not one decision. It is a cluster to operate, a delivery semantics question to answer, a schema-evolution policy, a dead-letter strategy, a replay story, a monitoring surface, and a new class of incident where the system is up but eleven minutes behind. Every one of those is tractable. None is free, and at our scale none buys anything the table above does not.

Notice how much of that list we still had to answer ourselves. Backoff, dead-lettering, reaping, and observability are queue problems, not broker problems. A broker would have given us implementations, not exemptions. What it adds on top is the cluster, the schema registry, and a second system of record that can disagree with the first.

Where this design actually breaks

Honest limits, because “Postgres is enough” without limits is marketing.

The part that makes this safe

The reason this is not technical debt is that the boundaries are drawn now, even though the split is deferred. The modules have real seams. The queue sits behind the same kind of owned interface as every other provider category, so swapping Postgres for a broker later is a configuration change and an adapter, not a rewrite of the calling code.

Deferred complexity and absent structure look identical from outside and behave completely differently under growth.

The difference is entirely in whether the boundaries exist before you need them.

What would change our minds

Specific, measured conditions — not a headcount and not a funding round:

1. Throughput where polling contention is measurably the bottleneck, after pruning, vacuum tuning, batching, and notify-based wake-ups are all done.
2. A genuine need to replay a stream, or several independent consumers of the same event with different retention. This is a shape problem and no tuning addresses it.
3. Deployment coupling demonstrably slowing releases — teams blocked on each other, not merely sharing a repository.

Until one of those is true and measured, adding a broker would be paying a permanent operational tax to look like a larger company.

Simple until proven otherwise

The word doing the work in that sentence is proven. It is not a commitment to stay small or a rejection of distributed systems. It is a requirement that the evidence arrives before the complexity does.

Own every layer.