Active Job Continuations: The end of lost jobs

Rails 8.1 brings a feature that's going to change how we handle long-running jobs: Active Job Continuations. Finally, we can interrupt and resume jobs without losing progress.

The problem to solve

Imagine you have a job importing 100,000 records. It's processing record 67,543 when you deploy an update that causes the container to restart. Or perhaps the container running the job queue simply restarts due to reaching memory limits.

The result? You're back to square one. Even worse, if the job wasn't designed with idempotency in mind, you could end up with inconsistent data.

With Active Job Continuations, this is no longer an issue. Jobs can pause at any moment and seamlessly resume exactly where they left off after a restart.

How it works

To enable continuations in your job, include the ActiveJob::Continuable module and use the step method to define stages:

class ProcessImportJob < ApplicationJob
  include ActiveJob::Continuable

  def perform(import_id)
    # This always runs, even if the job is resumed
    @import = Import.find(import_id)
    
    step :validate do
      @import.validate!
    end
    
    step :process_records do |step|
      @import.records.find_each(start: step.cursor) do |record|
        record.process
        step.advance!(from: record.id)
      end
    end
    
    step :reprocess_records
    step :finalize
  end

  def reprocess_records(step)
    @import.records.find_each(start: step.cursor) do |record|
      record.reprocess
      step.advance!(from: record.id)
    end
  end

  def finalize
    @import.finalize!
  end
end

If the job gets interrupted during process_records, when it resumes:

• It'll skip already completed steps (validate)
• Continue process_records from the last saved cursor
• Execute remaining steps (reprocess_records, finalize)

Cursors: your GPS through progress

Cursors are the key to everything. They let you mark exactly where you are within a step. Some examples of how to implement cursors:

Basic cursor with counter

step :iterate_items do |step|
  items[step.cursor..].each do |item|
    process(item)
    step.set!(step.cursor + 1)
  end
end

With initial value

step :iterate_items, start: 0 do |step|
  items[step.cursor..].each do |item|
    process(item)
    step.advance!
  end
end

For non-contiguous IDs

step :process_records do |step|
  records.find_each(start: step.cursor) do |record|
    record.process
    step.advance!(from: record.id)  # Uses the real ID, not sequential
  end
end

Nested cursors with arrays

For more complex cases, you can use arrays as cursors:

step :process_nested_records, start: [0, 0] do |step|
  Account.find_each(start: step.cursor[0]) do |account|
    account.records.find_each(start: step.cursor[1]) do |record|
      record.process
      step.set!([account.id, record.id + 1])
    end
    step.set!([account.id + 1, 0])
  end
end

Checkpoints: safe stopping points

A checkpoint is where a job can be safely interrupted. They're created automatically:

• At the end of each step
• When you call set!, advance!, or checkpoint!

step :destroy_records do |step|
  records.find_each do |record|
    record.destroy!
    step.checkpoint!  # Allow interruptions here
  end
end

When it hits a checkpoint, the job asks the queue adapter if it should stop (stopping?). If so, it saves progress and requeues itself to continue later.

Smart error handling

If a job fails after making progress (completed a step or advanced the cursor), it automatically retries while keeping the progress. This prevents losing work when something fails temporarily.

Queue adapter support

To work, your queue adapter must implement the stopping? method. It's already available in:

• Test adapter
• Sidekiq adapter

Other adapters like Solid Queue, Delayed Job, and Resque will need to implement it for full support.

Perfect use cases

Active Job Continuations shine for:

• Mass imports: CSVs with millions of records
• File processing: Resizing thousands of images
• Data migrations: Moving information between systems
• Report generation: Heavy calculations with lots of data
• Bulk cleanup: Maintenance jobs touching many records

The difference with other solutions

Unlike gems like job-iteration, continuations:

• Are natively integrated into Rails
• Allow multi-step flows
• Give you full control over when and how to checkpoint
• Don't intercept the perform method

Getting started

To use Active Job Continuations:

1. Wait for Rails 8.1 (or try with the main branch)
2. Include ActiveJob::Continuable in your long-running jobs
3. Define your steps with step
4. Use cursors to mark progress
5. Your queue adapter must support stopping?

It's a powerful tool but requires careful thought. As the docs say: "Continuations are a sharp knife" - you need to manually handle checkpoints and cursors, but in return you get full flexibility.