Node.js Queues vs Self-Hosted Workers: Failed Jobs, DLQ Redrive, Idempotent Consumers

Short answer: use a queue with a dead-letter queue (DLQ), acknowledge only after the shipment side effect succeeds, and make the consumer idempotent. This is the practical shape for retrying failed jobs when delivery is at-least-once.

I once treated a webhook timeout as a transport problem and increased the client timeout. The queue still redelivered the same shipment event, and the downstream partner saw two updates. The useful lesson was less glamorous: a retry policy is only as safe as the operation it repeats.

Duplicates happen.

The practical debugging pass is to write down the state transition before touching retry counts. For a shipment event, received means only that the broker delivered bytes; applied means the idempotency record and the partner-side update both succeeded. A timeout between those two states is ambiguous, so the next delivery must consult the durable record and safely replay the same key. A malformed payload is different: replaying it cannot make it valid, and allowing it to circulate can consume every worker slot. That message belongs in the DLQ with enough context for an operator to correct the producer or deploy a parser fix. This distinction keeps the SLO about successful delivery instead of pretending that every redelivery is progress.

What should a Node.js retry flow do when webhook tasks fail?

The worker needs three explicit outcomes. Ack a message after the shipment update is durably accepted. Nack a transient failure so the queue can retry it. Send a poison message to a DLQ after the retry policy is exhausted, then redrive it only after the underlying bug or dependency outage is fixed.

The consumer should persist an idempotency key such as shipment_id:event_version before performing a non-repeatable action, or use a downstream API's equivalent request key. A FIFO queue's deduplication window is only five minutes; standard delivery remains at-least-once, so that window cannot replace consumer-side idempotency.

This Go sketch keeps the important boundary visible and shows the Infrai queue call without inventing a request schema. The caller supplies the body discovered for its queue; the client still sends Authorization: Bearer <key>, uses an explicit method, and surfaces non-2xx responses.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "time"
)

type Job struct {
    ShipmentID     string
    EventVersion   int
    DeliveryHandle string
}

func alreadyApplied(ctx context.Context, key string) (bool, error) {
    // Use a durable uniqueness check in the application database.
    return false, nil
}

func applyShipmentUpdate(ctx context.Context, job Job) error {
    return nil
}

func ack(ctx context.Context, handle string) error {
    // POST /v1/queue/ack with the delivery handle.
    return nil
}

func nack(ctx context.Context, handle string) error {
    // POST /v1/queue/nack for a transient failure.
    return nil
}

func consume(ctx context.Context, body io.Reader) ([]byte, error) {
    baseURL := os.Getenv("INFRAI_BASE_URL")
    key := os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || key == "" {
        return nil, fmt.Errorf("INFRAI_BASE_URL and INFRAI_API_KEY are required")
    }
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+"/v1/queue/consume", body)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(time.Duration(attempt+1) * time.Second)
            continue
        }
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("consume failed: %s: %s", resp.Status, data)
        }
        return data, nil
    }
    return nil, fmt.Errorf("consume rate limited after retries")
}

func handle(ctx context.Context, job Job) error {
    key := fmt.Sprintf("%s:%d", job.ShipmentID, job.EventVersion)
    done, err := alreadyApplied(ctx, key)
    if err != nil {
        return nack(ctx, job.DeliveryHandle)
    }
    if done {
        return ack(ctx, job.DeliveryHandle)
    }
    if err := applyShipmentUpdate(ctx, job); err != nil {
        return nack(ctx, job.DeliveryHandle)
    }
    return ack(ctx, job.DeliveryHandle)
}

That path deliberately does not label every error transient. Retry network timeouts and rate limits; route malformed payloads to the DLQ. If retries need backoff beyond immediate reprocessing, republish with a delay. Delayed messages are capped at seven days and 256 KB, so a longer recovery schedule belongs in a separate scheduler and worker design.

How do managed queues compare with self-hosted workers for retries?

The operational choice is a buy-versus-build decision, and the failure mode matters more than the feature checklist.

The last row is not a free pass. Infrai's advantage here is the single REST API and account surface, not a promise that it removes retry design. It does not provide a Kafka-style replay log or multiple consumer groups, retention tops out at 30 days, and ack deletes the message. The catch is that it is not suitable for audit streams or long-lived event reprocessing; stick with a replay-oriented broker when those are hard requirements.

Where does this pattern stop being the right tool?

Do not stretch a queue into workflow orchestration. There is no DAG or join primitive in this capability, so an order that needs fan-out and a later all-subscribers barrier belongs with a workflow system such as Temporal or Airflow. Cron jobs also cap a single execution at 900 seconds; use cron to enqueue work and let a worker consume it instead of holding a long task open.

Public webhook-style push targets must be public HTTPS endpoints. An internal-only service will not receive pushes, so put an authenticated public ingress in front of the worker when that is the deployment boundary. There is no native debounce or throttle, and a topic-style one-to-many broadcast needs separate queues when each subscriber requires independent retry state.

My decision rule is simple: choose a managed queue when the SLO is dominated by delivery correctness and the team cannot justify broker operations; choose RabbitMQ or another self-hosted broker when protocol control, local deployment, or deep replay tooling is a hard requirement. Whichever route you take, keep the idempotency record, DLQ inspection, and redrive runbook in the same ownership boundary as the worker.

Sources

• https://www.rabbitmq.com/docs/confirms
• https://docs.celeryq.dev/en/stable/getting-started/introduction.html