sidekiq_solid_fetch: an open-source super_fetch for Sidekiq

Your Sidekiq worker just got OOM-killed. The job it was running is gone. Not "failed", not "waiting in the retry set" — gone, without a trace, and nothing in your monitoring will ever mention it again.

I ran into this in production, went looking for a fix, and ended up writing a gem: sidekiq_solid_fetch. This post explains the problem, why it's harder than it looks, and a couple of traps I fell into along the way.

The problem

When Sidekiq fetches a job, its default fetch strategy (BasicFetch) runs a Redis BRPOP against your queues. BRPOP is a destructive read: the job is removed from Redis and now exists in exactly one place — the memory of your worker process.

If the job succeeds, great. If it raises, Sidekiq catches the exception and schedules a retry. But there's a third case: the process dies without getting a chance to say anything. kill -9. The OOM killer. A Kubernetes pod evicted after its termination grace period. A spot instance reclaimed mid-job. A segfault in a native extension.

In that case there is no exception to catch and no retry to schedule. Sidekiq's retry mechanism helps with jobs that fail — it cannot help with jobs that vanish.

This is a known, documented property of Sidekiq OSS. The official answer is super_fetch, which is part of the commercial Sidekiq Pro and is excellent. There have been open-source attempts too — the best known is GitLab's reliable fetcher — but they target the pre-7.x internals, before Sidekiq's capsule rewrite. I wanted the same guarantee on modern Sidekiq (7 and 8), as a small open-source gem.

The core idea: fetch by moving, not popping

Instead of popping the job out of Redis, sidekiq_solid_fetch moves it — atomically, with LMOVE — from the queue into a processing list that is private to that worker:

queue:default  --LMOVE-->  sidekiq_solid_fetch:processing:{capsule}:{worker-identity}:queue:default

From there:

• Job succeeds → LREM removes it from the processing list. Done.

• Graceful shutdown → in-flight jobs are moved back to the front of their queues, so they run next after restart.

• Process dies mid-job → the payload is still sitting in the processing list, in Redis, waiting to be recovered.

The job payload never exists only in process memory. That's the whole trick. The rest of the gem is dealing with the consequences.

The hard part: who cleans up after the dead?

If a worker crashes, its processing lists survive it. Someone has to notice, and move those jobs back to their queues. This is where the design gets interesting, because the two obvious approaches are both wrong.

Obvious approach #1: encode everything in the key name and parse it back. The processing key contains the queue name and the worker identity, so recovery could scan for keys and split them apart. Except Sidekiq identities look like hostname:pid:nonce — they contain colons, queue names may too, and no regex splits that reliably. My first implementation did exactly this, and during review I discovered it would have "recovered" jobs from perfectly alive workers and moved them into queues like queue:default:MacBook-Pro.local:2486 — queues that no worker consumes. Silent job loss, in a gem whose entire purpose is preventing job loss.

So the gem doesn't parse anything. Each worker registers its processing queues in a Redis hash — the registry — mapping the key to structured data: the original queue, the worker identity, and a registration timestamp. Recovery walks the registry.

Obvious approach #2: a worker is dead if it's not in Sidekiq's processes set. Also wrong, in the other direction: entries for crashed workers linger in that set until Sidekiq's rate-limited cleanup gets around to pruning them. In the most common crash scenario — a supervisor restarts the worker within seconds — the new worker would look at the set, conclude its dead predecessor is still alive, and skip recovery. The jobs stay stranded.

Instead, liveness comes from Sidekiq's heartbeat key: every worker maintains a per-identity Redis key with a TTL of about a minute. Alive workers keep refreshing it; a crashed worker's key expires on its own. EXISTS on that key is a reliable answer to "is this worker still with us?"

Two more pieces complete the picture:

• A grace period. A freshly booted worker registers its queues before its first heartbeat lands. Without protection, another worker running recovery in that window would see "registered, but no heartbeat key" and steal its in-flight jobs. Registry entries younger than 60 seconds are never considered orphaned.

• Recovery runs continuously, not just at boot. Every worker checks at startup and then every 5 minutes (deduplicated across processes with a Redis lock). If your only worker crashes and nothing reboots for a while, the jobs are picked up as soon as anything does.

Small things that bite

Idle CPU. Sidekiq's processor loop calls retrieve_work in a tight loop with no delay of its own — blocking is the fetcher's job. BasicFetch blocks inside BRPOP for 2 seconds. A naive non-blocking LMOVE fetcher returns nil instantly and will happily hammer your Redis with thousands of no-op commands per second from every idle processor thread. sidekiq_solid_fetch pauses for 2 seconds when a full pass over the queues comes up empty, mirroring BasicFetch.

Requeue position. When Sidekiq requeues interrupted jobs, it puts them at the consuming end of the queue — they run next, not behind everything enqueued since. Recovered jobs are the oldest jobs you have; the gem preserves that semantic everywhere: graceful shutdown, orphan recovery, and per-job requeue.

At-least-once, not exactly-once. If a worker is partitioned from Redis for longer than its heartbeat TTL while still running a job, recovery will requeue that job and it will execute twice. This is inherent to every TTL-based liveness design — super_fetch carries the same caveat. Your jobs should be idempotent. (They should be anyway.)

The sidekiq-throttled trap

This one deserves its own warning sign. If you use sidekiq-throttled, you might be tempted to wire its fetch patches into a custom fetcher yourself:

# DO NOT do this with a reliable fetcher
SidekiqSolidFetch::Fetcher.prepend(Sidekiq::Throttled::Patches::BasicFetch)

Here's why. When sidekiq-throttled decides a fetched job is over its limit, it requeues it. For Sidekiq Pro's SuperFetch it has a dedicated path that cleans up properly — selected by a hardcoded class-name check. Every other fetcher gets the BasicFetch treatment: LPUSH a copy of the job back onto the queue. With a reliable fetcher, the original payload is still sitting in the processing list. Every throttle cycle leaks one duplicate.

I measured it: a concurrency-1 throttled job running, one more waiting — 127 duplicate payloads accumulated in the processing list within 30 seconds. They're invisible in normal operation. Then the worker shuts down, bulk_requeue faithfully returns everything in the processing list to the queue, and your throttled job runs a hundred-plus times.

The gem ships an integration that closes this:

require "sidekiq_solid_fetch/throttled"

Sidekiq.configure_server do |config|
  SidekiqSolidFetch.enable!(config)
  SidekiqSolidFetch::Throttled.setup!  # requires sidekiq-throttled >= 2.0
end

setup! applies the throttled fetch patches and replaces the copy with an atomic move (LREM + LPUSH in one MULTI) to the back of the target queue. Same test, with the integration: exactly one payload in the processing list, the throttled job waiting in its queue, and every job executing exactly once.

Kill it yourself

The scenario that started all of this, reproduced in a real Rails app (with sidekiq-cron, sidekiq-unique-jobs, sidekiq-status, and sidekiq-throttled all loaded):

1. A worker picks up a long-running job. The payload moves to its processing list.

2. kill -9 the worker mid-job. No goodbye, no bulk_requeue — exit code 137.

3. The payload is still in Redis. Nothing was lost.

4. The dead worker's heartbeat key expires. A new worker boots and logs:
   SidekiqSolidFetch: Re-queued 1 orphaned jobs

5. The same jid runs to completion, exactly once.

And on a graceful SIGTERM, in-flight jobs go back to the front of their queues and the registry is cleaned up — nothing left behind.

Getting started

# Gemfile
gem "sidekiq_solid_fetch"

# config/initializers/sidekiq.rb
require "sidekiq_solid_fetch"

Sidekiq.configure_server do |config|
  SidekiqSolidFetch.enable!(config)
end

That's the whole integration. Requirements: Ruby >= 3.3, Sidekiq >= 7.0 (tested against 7 and 8 in CI).

The code lives at github.com/k0va1/sidekiq_solid_fetch. If you run Sidekiq anywhere that processes can die ungracefully — which is to say, anywhere — I'd love for you to try it, and bug reports and PRs are very welcome.ve