Quick answer
Use a durable queue when work must continue after the initiating request ends. Return acceptance only after the task is durably recorded, then let a worker validate the message, perform an idempotent effect, and acknowledge successful completion. Design for a job arriving twice and for a worker stopping between its effect and its acknowledgement.
A document-processing product such as PDF.ai is a useful domain example: accepting a document, analysing it, and exposing a result have different completion states. This does not establish which queue the product uses. An asynchronous API should give the caller a stable job identifier and an authorised status endpoint rather than claiming the analysis finished when it was only queued.
Practical implementation steps
Give the job a versioned type, a stable business identifier, and the minimum data required to locate its authorised input. Prefer an object reference over embedding confidential document bytes in every queue message. Recheck ownership when processing and when returning the result; queue membership is not an authorisation grant.
Define retries by failure class. A temporary service outage may justify a delayed retry; invalid input needs correction or terminal failure. Bound attempts and elapsed processing time, retain a diagnostic reason without sensitive payloads, and provide a deliberate replay path for dead-letter jobs. Monitor oldest-job age as well as queue length.
Implementation example
This worker boundary makes the ordering explicit. The adapter must implement the idempotent effect using durable storage; the interface is not an in-memory deduplication guarantee. A failure leaves the message unacknowledged so the queue can redeliver it under its retry policy.
type Job = { id: string; inputKey: string; receipt: string };
type WorkerAdapters = {
applyIdempotently: (job: Job) => Promise<void>;
acknowledge: (receipt: string) => Promise<void>;
};
export async function handleJob(job: Job, adapters: WorkerAdapters) {
await adapters.applyIdempotently(job);
await adapters.acknowledge(job.receipt);
}
For a database-only effect, enforce a unique operation key and commit the deduplication record with the business change in one transaction. If acknowledgement fails after commit, the repeated job observes the recorded result. For an external payment or email, the database cannot atomically commit the other service’s effect; use a provider idempotency key where available and retain enough state to reconcile uncertain outcomes.
Tradeoffs and alternatives
A database job table can simplify small workloads when ownership and polling are clear. A managed queue supplies delivery and visibility mechanisms but still requires idempotent consumers. When a database write and queue publication must stay consistent, an outbox recorded in the same transaction avoids the gap where one succeeds and the other fails.
Common mistakes
Avoid acknowledging before the effect commits, treating a visibility lease as a permanent lock, retrying malformed jobs forever, and using a process-local Set as the sole duplicate record. A timeout can indicate an unknown result rather than a failed effect; blindly repeating such work can charge or notify twice.
Production checklist
- Acceptance follows durable job persistence.
- Duplicate deliveries reuse a stable operation key.
- Acknowledgement follows successful completion.
- Visibility timeouts and worker heartbeats fit job duration.
- Dead-letter diagnosis, authorised replay, and backlog alerts are defined.
FAQ
Does a visibility timeout prevent every duplicate?
No. Use it to coordinate processing and recovery, while keeping the effect safe under repeated delivery.
Should the API wait for the worker?
Only if the user contract requires a bounded synchronous result. Otherwise return a persisted job ID, expose honest status, and make completion notification a separate concern.
Next step
Stop a worker after its effect commits but before acknowledgement, then redeliver the same job. The effect should occur once and its completion remain observable.
Primary sources
- Amazon SQS: At-least-once delivery
- Amazon SQS: Visibility timeout
- PostgreSQL: Constraints
- PostgreSQL: INSERT and ON CONFLICT
Related guides and examples
Need this implemented in a real product?
One connected implementation across the interface, application logic, APIs, and data layer.