Sooner or later a web app has work that shouldn’t hold up a page, like an email to send or a webhook to retry. That usually means running Redis or RabbitMQ beside the app. Ukue 0.1.0, released on 6 October, stores the whole queue in a single SQLite file instead. The file’s layout is documented, so programs in different languages can share one queue, and a copy of the file is a full backup.
What It Does
Failed jobs are retried with backoff, waiting 10 seconds at first and twice as long each time after, up to an hour, with 10 attempts unless you set otherwise. Jobs can be delayed or given a priority, and one that keeps failing is parked with the dead letters until someone looks at it. Workers take jobs on a lease and keep renewing it while they work, so when a worker crashes, its lease lapses and another worker picks the job up. Each change to a job commits as its own SQLite transaction and reaches the disk before the next step.
There are three ways in. Go programs import Ukue as a package. ukue work runs any script as a worker that reads the job on standard input, where exit code 0 means done and 65 sends the job to the dead letters. Everything else talks to ukue serve, which puts the queue behind an HTTP API, the way a Python worker on a rented GPU pulls transcription jobs. And with EnqueueTx, a Go app adds a job in the same transaction as its own data, so an order and its receipt email are saved together or not at all.
Tested by Killing Workers
The hardest test ran 8,000 jobs through four worker processes and killed one at random with SIGKILL 250 times in about 20 seconds, wherever it happened to be. Every job finished, none got stuck, and SQLite’s integrity check found nothing wrong, mid-run or at the end. 329 jobs ran more than once, each because a worker was killed partway through the job, before it could report it done. Those repeats are expected with at-least-once delivery, so handlers need to cope with running twice. HyperCrux went through the same kind of test with 200 killed writers.
Ukue is meant for a single app server or a handful of workers sharing one machine, at a few hundred jobs a second, and stays out of cluster-scale messaging. Version 0.1.0 is on the releases page for Linux, macOS and Windows.