Operations and Monitoring

The durable queue mounts an operator surface under /queue. With the default max_attempts=1, a crashed run stays failed until an operator acts, so these endpoints list failed jobs, grant them another attempt, and report queue depth.

curl localhost:7777/queue/stats

{"counts": {"completed": 3, "cancelled": 1, "running": 1}, "oldest_queued_age_seconds": null}

Endpoints

All four answer 503 when the durable queue is not enabled on the replica that served the request.

Listing parameters

Job fields

Requeue

curl -X POST localhost:7777/queue/jobs/{job_id}/requeue

Requeue raises max_attempts to attempt + 1 and moves the job back to queued. Only failed and cancelled jobs qualify; anything else answers 400. An available matching worker can claim the requeued job on a subsequent poll.

Requeue does not bypass attempt fencing. If the presumed-dead worker finishes after all, the later attempt owns the job and the earlier one's writes are discarded.

Stats and alerting

Retention

Once an hour, the worker deletes terminal jobs (completed, failed, cancelled) older than retention_seconds (default 24 hours). Paused jobs are exempt: a paused run's job is what a later continue re-queues, and it must outlive human latency. Cancel a paused run to release its job. Run rows and sessions are not touched by queue retention.

Access control

Job rows expose payloads and user IDs across tenants, and requeue grants execution budget, so /queue is an operator surface. Any request carrying a JWT identity must hold the admin scope, or it gets 403, regardless of whether user isolation is on. Deployments without JWT enforcement (security key or open) pass through, matching how the run routes treat scope enforcement. See JWT middleware and Scopes.

Deployment affinity

deployment_id pins jobs to a subset of workers in a heterogeneous fleet:

queue=QueueConfig(durable=True, deployment_id="gpu-pool")

A mixed fleet needs no extra configuration. A continuation inherits the submitting job's deployment_id.

Fleet-wide settings

Every replica sharing one queue table must use the same values for lock_grace_seconds, stop_timeout_seconds, retention_seconds, max_attempts, and timeout_seconds. Each is applied by whichever replica performs the action. The replica that claims a job heartbeats on its own lock_grace_seconds, while the replica that sweeps judges staleness on its own value. max_attempts is written onto each job by the replica that accepted it. The smallest retention_seconds in the fleet wins the hourly cleanup. Divergent values, including transiently during a rolling deploy, can falsely sweep a healthy peer's runs or delete jobs early.

When changing lock_grace_seconds on a live fleet, only ever raise it, and roll the sweeping replicas first. A replica sweeping with a smaller grace than its peers heartbeat with judges their live leases stale.

Shutdown and restart

On graceful shutdown the worker stops claiming and gives in-flight runs stop_timeout_seconds (default 30, always below lock_grace_seconds) to finish. Heartbeats continue through the drain, so peers do not reclaim a run that is still finishing. Stragglers are cancelled once: with attempts remaining they return to queued, otherwise they fail with interrupted by worker shutdown. Queued jobs are untouched and execute after the restart, on whichever replica claims them.

Next Steps