githublinkedinemail
← /articles

The "exactly once" myth

Exactly-once doesn't exist on the network. What exists is at-least-once plus idempotency. Stop chasing a marketing myth — design for reality and sleep at night.

4 mindistributed

The "exactly once" myth

Queue marketing loves one phrase: "exactly-once delivery".

Every time you read it, someone is lying, omitting, or using terms with an asterisk the size of a paragraph.

Spoiler: on the network, exactly-once does not exist.

The sentence that sold a lot of SaaS

"Our queue guarantees exactly-once delivery."

Shows up on Kafka, SQS, RabbitMQ, Pulsar landing pages, and on every alternative to them.

In small print, there's always a caveat. Usually the size of the product itself.

The uncomfortable truth: exactly-once delivery is mathematically impossible in distributed systems.

It's not an implementation limit. It's a physical one.

The two generals problem, in plain words

Two generals need to attack a city at the same time. They can only win if they attack together.

They communicate via messengers that can be captured.

General A sends: "attack at 6am".

Did the message reach B? A doesn't know. Waits for confirmation.

B replies: "ok, attack at 6am".

Did the confirmation reach A? B doesn't know. Waits for confirmation of the confirmation.

And so on, forever.

Formal conclusion: no protocol can guarantee mutual certainty about a message over an unreliable channel.

Now swap "general" for "system" and "messenger" for "TCP network".

Every queue, every broker, every HTTP call lives inside this problem.

What can go wrong between A and B

Producer (A)              Broker
   |                         |
   |---- msg ---------------->|
   |                         |  (writes)
   |<--- ack ----------------|
   |                         |

Four things can fail at any point:

  1. Msg never reached the broker → producer got no ack → resends (dup if it had arrived and only the ack was lost).
  2. Msg arrived, broker wrote it, ack got lost → producer resends → duplicate.
  3. Msg arrived, broker crashed before writing → loss.
  4. Msg was written, consumed, but consumer's ack got lost → will be redelivered.

You have two honest choices:

  • at-most-once: never duplicates, may lose
  • at-least-once: never loses, may duplicate

There is no third option at the network level.

What "exactly-once" means in marketing

When someone says exactly-once, they mean one of three things:

1. Exactly-once processing

Message may arrive twice, but the side effect happens once.

Solved in your application, with idempotency. Not in the broker.

2. Exactly-once within a closed boundary

Kafka does this. Within Kafka, a message produced inside a transaction shows up exactly once to consumers reading from the same transaction.

It does not cross the boundary: if you consume from Kafka and write to Postgres, exactly-once is gone.

3. Pure lie

Happens. A lot.

Kafka "exactly-once" — the caveats

Kafka has enable.idempotence=true and transactions. Works, but only under strict rules:

# producer
enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5
transactional.id=my-producer-1

producer.beginTransaction();
producer.send(record1);
producer.send(record2);
producer.sendOffsetsToTransaction(offsets, "my-group");
producer.commitTransaction();

Works if:

  • producer, broker and consumer are all in Kafka
  • the consumer runs isolation.level=read_committed
  • you don't write to an external system in between
  • you don't cross clusters

Crossed the Kafka boundary? Guarantee is gone.

And "gone" is subtle: you still receive the message once at the consumer, but if your consumer fails between processing and committing the offset, the next poll brings the same message.

You're back to at-least-once.

Every system you build is at-least-once

Take your webhook handler, your Sidekiq queue, your Kafka consumer writing to Postgres, your SQS lambda.

They all have the same shape:

1. read message
2. process (apply side effect)
3. acknowledge (ack, commit offset, delete from queue)

If step 3 fails after step 2, the message comes back.

You will process it again.

You can't avoid this without doing step 2 and step 3 in the same atomic transaction — and that's only possible if the side effect is in the same system as the ack.

Spoiler: it almost never is.

The only honest pattern: at-least-once + idempotency

Grown-up solution:

  1. Accept that messages can arrive more than once.
  2. Make your processing idempotent.

Idempotent = applying twice has the same effect as applying once.

Pattern 1: idempotency key

class PaymentProcessor
  def charge(payment_id, idempotency_key)
    return if Charge.exists?(idempotency_key: idempotency_key)

    ActiveRecord::Base.transaction do
      Charge.create!(
        idempotency_key: idempotency_key,
        payment_id: payment_id,
        amount: 100
      )
      Stripe::Charge.create(
        amount: 100,
        idempotency_key: idempotency_key
      )
    end
  end
end

Message arrived twice? Second time hits exists? and exits.

The key has to come from the producer, not the consumer. If the consumer generates it, two deliveries generate two different keys.

Pattern 2: deterministic upsert

INSERT INTO orders (id, user_id, amount)
VALUES ($1, $2, $3)
ON CONFLICT (id) DO NOTHING;

Same message, same row, same result.

Pattern 3: transactional outbox

You want to publish an event + write to the database atomically?

Don't publish directly. Write to an outbox table in the same transaction:

BEGIN;
INSERT INTO orders (...) VALUES (...);
INSERT INTO outbox (event_type, payload) VALUES ('order.created', '{...}');
COMMIT;

A separate worker reads outbox and publishes to the broker. If it publishes a duplicate after a failure, the consumer handles it through idempotency.

You just turned "distributed exactly-once" (impossible) into "at-least-once + idempotency" (possible).

Senior vs junior

Junior:

"Kafka guarantees exactly-once, so I don't worry about duplicates."

Senior:

"Kafka can promise whatever it wants. I have an idempotency_key column with a unique index and ON CONFLICT DO NOTHING. A duplicate message is harmless."

One trusts the broker. The other knows trust at that level is negligence.

The honest architecture

Producer
   ↓
[generates deterministic idempotency_key]
   ↓
Broker (at-least-once)
   ↓
Consumer
   ↓
[checks idempotency_key]
   ↓ unseen           ↓ already seen
process               ignore
   ↓
write result + mark key
(same transaction)
   ↓
ack

Failure anywhere:

  • before ack: message comes back, idempotency catches it
  • after ack: nothing happens, already processed

You just built "effectively-once". Which is what marketing calls exactly-once, but honestly.

Numbers that matter

System with no idempotency, queue at-least-once:

  • ~0.01% of messages duplicate under normal conditions
  • ~5–15% duplicate during an incident (failover, deploy, retry storm)

At 10 million events/day:

  • normal: 1,000 duplicates/day
  • incident: up to 1.5 million duplicates

Without idempotency, that's 1.5 million double charges, repeated emails, cloned orders.

With idempotency, that's 1.5 million SELECTs on a unique index. Cost: irrelevant.

Conclusion

Exactly-once is a fantasy.

It exists in marketing. It exists inside very specific product boundaries.

It does not exist in your system, which crosses network, database, broker, external API.

Your choice is not between at-least-once and exactly-once.

It's between at-least-once with idempotency and at-least-once without idempotency, hoping for the best.

Stop chasing exactly-once.

Design for at-least-once and sleep at night.

The "exactly once" myth
The "exactly once" myth