A First Look at Solid Queue and Mission Control Jobs

5th Feb 2024

#all #tech #rails

Background Jobs

Background jobs are something most web developers need to start reaching for whenever there’s a long running task, or something outside the scope of a typical web request. One classic example, and the one we’ll use as an example today, is sending email. Of course Rails makes it easy to send emails with ActionMailer, and you can send an email asynchronously to a user in response to them clicking a ‘forgot email’ link with deliver_later.

As nicely defined in the Rails Guides Active Job…

is a framework for declaring jobs and making them run on a variety of queuing backends. These jobs can be everything from regularly scheduled clean-ups, to billing charges, to mailings. Anything that can be chopped up into small units of work and run in parallel.

ActionMailer uses the ActiveJob :async adapter by default, which is the simplest implementation via an in-process thread pool to offload execution of jobs. The down side is if the server restarts or there are other issues, then any pending jobs will be lost. Similarly this strategy won’t work with rake tasks. So not really viable for most production use cases.

Over the last ~20 years, Rails has had numerous excellent implementations for ActiveJob, from DelayedJob by Tobi, to redis backed solutions like Resque from Chris Wanstrath and Sidekiq from Mike Perham. Each has it’s own requirements, benefits and trade-offs.

Solid Queue

Announced at Rails World 2023, 37signals - and specifically long time friend Rosa Gutiérrez - had been working on Solid Queue and its release was subsequently announced in December.

Solid Queue is a database backed queueing backend for ActiveJob that will be the default in Rails 8, and provide Rails developers with a highly scalable and performant solution out of the box that supports the major databases.

Introducing Solid Queue to 1500Cals

Installation

First lets add the Solid Queue gem. From within your root directory:

bundle add solid_queue

Then run the installer that will copy the migration files, update production.rb and create a starter config in config/solid_queue.yml.

bin/rails generate solid_queue:install

Lastly lets run the migrations to create our solid_queue tables

rails db:migrate

Note, if you want to try this locally as I did, you may need to add the following to config/environments/development.rb as the installer only seems to add this to production.rb

config.active_job.queue_adapter = :solid_queue

and then we can start up Solid Queue in its own process. Thats it!!

bundle exec rake solid_queue:start

Queueing Jobs - Mailer Example

Of course, Solid Queue isn’t very exciting without any enqueued jobs. In 1500cals, when a user signs up or signs in, we send a 6-digit 2FA code via email. This is implemented in a controller simply with AuthMailer.auth_code(user).deliver_later, which will enqueue a job in the configured config.active_job.queue_adapter. In this case Solid Queue!

Lets try this in the console, so type bin/rails c to start irb and then in my case I invoke my AuthMailer

AuthMailer.auth_code(User.first).deliver_later

Firstly we can see a SolidQueue::Job being inserted into the solid_queue_jobs table

SolidQueue::Job Create (1.0ms) INSERT INTO "solid_queue_jobs"
("queue_name", "class_name", "arguments", "priority", "active_job_id",
"scheduled_at", "finished_at", "concurrency_key", "created_at",
"updated_at")
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) RETURNING "id"

followed by a SolidQueue::ReadyExecution

SolidQueue::ReadyExecution Create (0.9ms)  INSERT INTO "solid_queue_ready_executions"
("job_id", "queue_name", "priority", "created_at") VALUES ($1, $2, $3, $4) RETURNING "id"  [["job_id", 5], ["queue_name", "default"], ["priority", 0], ["created_at", "2024-02-03 15:23:17.928214"]]

So there we have it, our first job is enqueued!

Enqueued ActionMailer::MailDeliveryJob (Job ID: 1818db52-88ce-4183-bea0-51276ff90d00) to SolidQueue(default) with arguments: "AuthMailer", "auth_code", "deliver_now", {:args=>[#<GlobalID:0x0000000106fb9198 @uri=#<URI::GID gid://fifteen-hundred-cals/User/1>>]}
↳ (irb):1:in `<main>'
=>
#<ActionMailer::MailDeliveryJob:0x0000000106ebdf00
 @_halted_callback_hook_called=nil,
 @_scheduled_at_time=Sat, 03 Feb 2024 15:23:17.904595000 UTC +00:00,
 @arguments=
  ["AuthMailer",
   "auth_code",
   "deliver_now",
   {:args=>
     [#<User:0x0000000106bd1f08
       id: 1,
       email: "miles.woodroffe@gmail.com",
       username: "tapster",
       auth_secret: "[FILTERED]",
       created_at: Thu, 21 Apr 2022 07:56:20.889225000 UTC +00:00,
       updated_at: Wed, 27 Dec 2023 13:33:13.967755000 UTC +00:00,
       admin: true,
       time_zone: "London",
       premium: nil,
       daily_target: 1500>]}],
 @exception_executions={},
 @executions=0,
 @job_id="1818db52-88ce-4183-bea0-51276ff90d00",
 @priority=nil,
 @provider_job_id=5,
 @queue_name="default",
 @scheduled_at=Sat, 03 Feb 2024 15:23:17.904595000 UTC +00:00,
 @successfully_enqueued=true,
 @timezone="UTC">

Executing Jobs

As soon as we start Solid Queue, our pending jobs are claimed, executed, and assuming all went well, are deleted from the “solid_queue_ready_executions” table, and the finished_at column is populated in the “solid_queue_jobs” table.

bundle exec rake solid_queue:start

[SolidQueue] Starting Dispatcher(pid=62663, hostname=miles-mba.local, metadata={:polling_interval=>1, :batch_size=>500})
[SolidQueue] Starting Worker(pid=62664, hostname=miles-mba.local, metadata={:polling_interval=>0.1, :queues=>"*", :thread_pool_size=>5})
[SolidQueue] Claimed 2 jobs

Puma Plugin

As mentioned above, you can start and run the Solid Queue supervisor in its own process with bundle exec rake solid_queue:start which is pretty standard. However, there’s a really cool way to have Puma handle this at startup with the provided plugin. Simply open up config/puma.rb and add the following line at the end.

plugin :solid_queue

This is another way to reduce operational complexity in simple apps, especially when deploying to heroku or fly.io for example. Love this.

Summing up

There’s a lot more to Solid Queue that makes this a very robust, database backed solution, and what I hope you’re seeing - especially if you’re more accustomed to Redis backed queues - is how refreshingly simple and accessible the jobs are when they’re in the database.

Mission Control - Jobs

Last week, 37signals open sourced Mission Control - Jobs as announced by Rosa again. This dashboard for background jobs like Solid Queue is perfect for managing jobs and the state of your queues. It can be used with Resque it seems but here we’ll add to 1500cals again so we can get some insight on our usage.

Installation

First lets install the gem

bundle add mission_control-jobs

Then we need to mount the gem in our routes file so we have a path to access it. Open up routes.rb and add the following in your Rails.application.routes.draw do block:

Rails.application.routes.draw do
  mount MissionControl::Jobs::Engine, at: "/jobs"
  # ... existing routes
end

Next.. wait, there is no next! Thats it :D We can immediately jump in and inspect the job queues and jobs in our app at localhost:3000/jobs

Here we can see a nice simple interface where we can navigate workers, queues, jobs in each state like in progress, finished, blocked, failed etc

As you can see I set up two failed jobs and inspecting that tab we can see the details of what happened, click a “Discard” or “Retry” button to delete or reschedule the job, and also drill down to see a stack trace

Security!

A quick word of warning that Mission Control - Jobs will be open to all out of the box! Most likely you don’t want your users being able access this so lets fix that by adding some authentication. If you’re using Devise you can wrap the engine route from the previous section with an authenticated :user do / end block.

In my case I’ve rolled my own auth, but it was still simple to add using the configuration Mission Control provides. First in my case I needed to add an AdminController. Since I already had this admin concept with a simple boolean on the User model I did it as follows, but adapt for your own needs as you see fit in the require_admin method

class AdminController < ApplicationController
  before_action :require_admin

  private

  def require_admin
    raise ActiveRecord::RecordNotFound unless Current.user.admin?
  end
end

Then we need to tell Mission Control we want to use this new base controller, since the default is ApplicationController. So, we simply add the following to config/environments/development.rb and config/environments/production.rb and we’re good to go. Our /jobs path is now only available to users with the admin role.

MissionControl::Jobs.base_controller_class = "AdminController"

Wrap Up

Thats it. The post took way longer to write than it did to get Solid Queue and Mission Control - Jobs installed, but I wanted to document my experience and hopefully help a few folks out along the way.

As always, love to hear your comments, thoughts and questions via the usual channels: twitter