Designing a Python queue for 200K+ automations a day

How much queue do you actually need? For most products the honest answer is "whatever your framework already ships with". At Sendler.ai the automation workload came with enough specific rules that I designed an in-house Python queue system instead, and it scaled processing to 200K+ automations per day.

This post is about what that kind of queue has to get right, and how I think about building versus buying one. The code is simplified to show the ideas, not lifted from production.

The average is not the problem

200K automations a day averages out to a little over two per second. Any queue can handle that. What makes automation work hard is its shape:

• Every job belongs to a customer account, and accounts differ a lot in size.
• Jobs call external APIs that have their own limits, outages and quirks.
• Jobs have visible side effects. Sending the same outreach message twice is a bug a prospect sees.
• Load is bursty. A campaign launch can enqueue a large batch of work in seconds.

So the interesting requirements are idempotency, retries, visibility and fairness, not raw throughput.

Idempotency comes first

Any queue that survives worker crashes delivers at least once, which means some jobs will run twice. Design for that from the start instead of hoping it's rare.

Give each job a key derived from its business identity rather than a random UUID, so a re-enqueued job gets the same key as the original. Then record progress around the side effect:

import hashlib
 
def job_key(account_id: str, lead_id: str, step: str, campaign_rev: int) -> str:
    raw = f"{account_id}:{lead_id}:{step}:{campaign_rev}"
    return hashlib.sha256(raw.encode()).hexdigest()
 
def handle(job, store, gateway) -> None:
    if store.is_done(job.key):
        return  # a retry after the work already succeeded
    if store.was_started(job.key) and gateway.already_applied(job.key):
        store.mark_done(job.key)  # crashed after the side effect, before recording it
        return
    store.mark_started(job.key)
    result = gateway.perform(job.payload, idempotency_key=job.key)
    store.mark_done(job.key, external_id=result.id)

The gap between perform and mark_done is the dangerous part: a worker can die in between. If the external API accepts an idempotency key, pass yours and the gap closes. If it doesn't, the started marker tells the retry to check the external system's state before acting again. That reconciliation step is tedious to write, and it's the difference between "rare duplicates" and "no duplicates".

Retries need a classification, a backoff and an end

Not every failure deserves a retry. Split them first:

• Retryable: timeouts, connection errors, 429s and 5xx responses.
• Permanent: validation errors, revoked credentials, a lead that no longer exists.

Permanent failures go straight to a failed state with a reason the product can show. Retryable ones back off exponentially with jitter, so a hundred jobs that failed together don't all retry together. If the API sends Retry-After, it wins.

import random
 
RETRYABLE = {408, 429, 500, 502, 503, 504}
 
def next_delay(attempt: int, status: int | None, retry_after: float | None,
               base: float = 5.0, cap: float = 900.0, max_attempts: int = 8) -> float | None:
    """Seconds until the next attempt, or None to dead-letter the job."""
    if attempt >= max_attempts:
        return None
    if status is not None and status not in RETRYABLE:
        return None  # permanent: no point trying again
    if retry_after is not None:
        return retry_after
    return random.uniform(0, min(cap, base * 2 ** attempt))  # "full jitter"

Jobs that run out of attempts land in a dead-letter state, not in a log line. Someone needs to be able to list them, see why they failed, and re-run them after a fix.

Visibility, in both senses

Leases for crashed workers

A worker claims a job with a lease: "I own this until time T". If the worker dies, the lease expires and another worker picks the job up. Long jobs extend their lease with a heartbeat. On a relational database, SELECT ... FOR UPDATE SKIP LOCKED is a common way to let many workers claim rows without blocking each other.

Set the lease too short and healthy jobs get picked up twice, which is one more reason idempotency comes first.

Visibility for humans

The other kind of visibility is answering questions without reading logs. "Why hasn't this account's campaign sent anything since noon?" should be a query: jobs by account and status, next run time, last error, attempt count. Job state is product data, and support will need it as much as engineering does.

For alerting, the age of the oldest ready job is more useful than queue depth. Depth rises during every healthy burst. Age only climbs when work is stuck or capacity is short.

Per-account rate limits are a scheduling problem

A single global concurrency limit fails in two ways. One large account launching a campaign can starve everyone else, and each account has its own safe limits on the external platform that the product must respect to protect the customer's account.

The fix is to apply limits when choosing work, not while executing it:

• Keep a token bucket per account. Redis is a reasonable place for it.
• When a worker asks for work, rotate across accounts that have ready jobs, and only hand out a job if that account has capacity.
• If an account is at its limit, its jobs stay ready with a later run time. They are deferred, not failed, and they don't burn retry attempts.
• Time windows, such as only sending during an account's working hours, use the same mechanism: compute the next allowed time and schedule for it.

A worker that picks up a job only to sleep because the account is over its limit is holding a slot another account could use.

Build or buy

I'd still start most projects with an existing tool. A quick way to decide:

Build when the scheduling rules are part of the product, when job state is something users and support need to query, and when the core primitives (claim, lease, retry, dead-letter) are small enough for the whole team to understand. Buy when none of that is true. Owning a queue means owning its on-call, its metrics and its edge cases, and that cost doesn't show up in the first sprint.

Whether you build or buy, the parts that matter are the same: make every job idempotent by business key, classify failures before retrying them, alert on the age of the oldest job rather than the length of the queue, and enforce per-account limits at dequeue time. Get those four right and throughput is rarely the thing that breaks.