No workers after I migrated from 1.0.0

[diegocharles]

Hey everyone!

I’m facing an issue that hasn’t been resolved since upgrading from version 1.0.0. The jobs are getting stuck, and when I manage them through MissionControl, I don’t see any workers.

Here’s my stack:

• Rails 8
• Kamal 2
• Hertzer:
  • Web server
  • Job server
  • Database server

The solid_queue tables are managed by PostgreSQL, and this is how my queue.yml looks:

  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_maintenance_interval: 300

  workers:
    - queues: “*”
      threads: 3
      polling_interval: 2
    - queues: [ real_time, background ]
      threads: 5
      polling_interval: 0.1
      processes: 3

Thanks in advance

[rosa]

Hey @diegocharles, how are you starting Solid Queue?

[diegocharles]

Hey Rosa!

I believe Kamal 2 starts it, because I don’t have any explicit calls for production like I do for development in the Procfile.dev.

jobs: bin/rails solid_queue:start

Thanks for your help.

[rosa]

@diegocharles that sounds like you're using the Puma plugin to run Solid Queue supervised by Puma, wherever you're running your web app. Do you have logs for when your production server starts?

[diegocharles]

@rosa Well, I may be wrong, but I think I am not using SQ supervised by Puma, because I've never enabled that setting:

env:
  clear:
    # Run the Solid Queue Supervisor inside the web server's Puma process to do jobs.
    # When you start using multiple servers, you should split out job processing to a dedicated machine.
    # SOLID_QUEUE_IN_PUMA: true

puma.rb

# Run the Solid Queue supervisor inside of Puma for single-server deployments
plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"]

Here's the production's log when booting the app

2025-02-03T11:10:47.098794791Z [29] Puma starting in cluster mode...
2025-02-03T11:10:47.098874380Z [29] * Puma version: 6.6.0 ("Return to Forever")
2025-02-03T11:10:47.098879861Z [29] * Ruby version: ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +YJIT +PRISM [x86_64-linux]
2025-02-03T11:10:47.098883328Z [29] *  Min threads: 4
2025-02-03T11:10:47.098885982Z [29] *  Max threads: 4
2025-02-03T11:10:47.098888507Z [29] *  Environment: production
2025-02-03T11:10:47.098891152Z [29] *   Master PID: 29
2025-02-03T11:10:47.098893646Z [29] *      Workers: 5
2025-02-03T11:10:47.098896192Z [29] *     Restarts: (✔) hot (✖) phased
2025-02-03T11:10:47.098899407Z [29] * Preloading application
2025-02-03T11:10:47.099321429Z [29] * Listening on http://0.0.0.0:3000
2025-02-03T11:10:47.099346656Z [29] Use Ctrl-C to stop
2025-02-03T11:10:47.127818650Z [29] - Worker 0 (PID: 44) booted in 0.02s, phase: 0
2025-02-03T11:10:47.128063890Z [29] - Worker 1 (PID: 56) booted in 0.02s, phase: 0
2025-02-03T11:10:47.128596869Z [29] - Worker 2 (PID: 68) booted in 0.01s, phase: 0
2025-02-03T11:10:47.129143684Z [29] - Worker 3 (PID: 80) booted in 0.01s, phase: 0
2025-02-03T11:10:47.129794293Z [29] - Worker 4 (PID: 92) booted in 0.0s, phase: 0
2025-02-03T11:10:47.405475959Z {"time":"2025-02-03T11:10:47.40529482Z","level":"INFO","msg":"Request","path":"/up","status":200,"dur":69,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":73,"resp_content_type":"text/html; charset=utf-8","remote_addr":"172.18.0.2:51996","user_agent":"kamal-proxy","cache":"miss","query":""}
2025-02-03T11:10:57.467365713Z D, [2025-02-03T11:10:57.467046 #92] DEBUG -- [Bugsnag]: Request to https://sessions.bugsnag.com completed, status: 202
2025-02-03T11:11:03.005291577Z I, [2025-02-03T11:11:03.005132 #92]  INFO -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce] Started GET "/backoffice/jobs/applications/maybeclubbackend/workers?server_id=solid_queue" for 162.158.120.251 at 2025-02-03 11:11:03 +0000
2025-02-03T11:11:03.011631029Z I, [2025-02-03T11:11:03.011482 #92]  INFO -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce] Processing by MissionControl::Jobs::WorkersController#index as HTML
2025-02-03T11:11:03.011652309Z I, [2025-02-03T11:11:03.011548 #92]  INFO -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   Parameters: {"server_id" => "solid_queue", "application_id" => "maybeclubbackend"}
2025-02-03T11:11:03.084772797Z D, [2025-02-03T11:11:03.084442 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   User Load (2.2ms)  SELECT "users".* FROM "users" WHERE "users"."id" = $1 ORDER BY "users"."id" ASC LIMIT $2  [["id", 90], ["LIMIT", 1]]
2025-02-03T11:11:03.086160478Z D, [2025-02-03T11:11:03.086019 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   ↳ app/controllers/application_controller.rb:32:in 'ApplicationController#current_profile'
2025-02-03T11:11:03.135344684Z D, [2025-02-03T11:11:03.135049 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   Profile Load (2.8ms)  SELECT "profiles".* FROM "profiles" WHERE "profiles"."user_id" = $1 LIMIT $2  [["user_id", 90], ["LIMIT", 1]]
2025-02-03T11:11:03.136636265Z D, [2025-02-03T11:11:03.136464 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   ↳ app/controllers/application_controller.rb:34:in 'ApplicationController#current_profile'
2025-02-03T11:11:03.182427931Z D, [2025-02-03T11:11:03.182126 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   Session Load (1.7ms)  SELECT "sessions".* FROM "sessions" WHERE "sessions"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]
2025-02-03T11:11:03.187503364Z D, [2025-02-03T11:11:03.187252 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   ↳ app/controllers/concerns/authentication.rb:30:in 'Authentication#find_session_by_cookie'
2025-02-03T11:11:03.238758831Z D, [2025-02-03T11:11:03.238516 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   SolidQueue::Process Count (2.2ms)  SELECT COUNT(*) FROM (SELECT 1 AS one FROM "solid_queue_processes" WHERE "solid_queue_processes"."kind" = $1 LIMIT $2 OFFSET $3) subquery_for_count  [["kind", "Worker"], ["LIMIT", 100000000], ["OFFSET", 0]]
2025-02-03T11:11:03.248057169Z D, [2025-02-03T11:11:03.247858 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   Rendering layout /usr/local/bundle/ruby/3.4.0/gems/mission_control-jobs-1.0.1/app/views/layouts/mission_control/jobs/application.html.erb
2025-02-03T11:11:03.248086474Z D, [2025-02-03T11:11:03.247962 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   Rendering /usr/local/bundle/ruby/3.4.0/gems/mission_control-jobs-1.0.1/app/views/mission_control/jobs/workers/index.html.erb within layouts/mission_control/jobs/application
2025-02-03T11:11:03.254255026Z D, [2025-02-03T11:11:03.254110 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   SolidQueue::Process Count (1.2ms)  SELECT COUNT(*) FROM (SELECT 1 AS one FROM "solid_queue_processes" WHERE "solid_queue_processes"."kind" = $1 LIMIT $2 OFFSET $3) subquery_for_count  [["kind", "Worker"], ["LIMIT", 10], ["OFFSET", 0]]
2025-02-03T11:11:03.262811383Z D, [2025-02-03T11:11:03.262667 #92] DEBUG -- : [b6867dd8-dcde-4063-9b73-5f01b44a75ce]   SolidQueue::Process Load (1.0ms)  SELECT "solid_queue_processes".* FROM "solid_queue_processes" WHERE "solid_queue_processes"."kind" = $1 LIMIT $2 OFFSET $3  [["kind", "Worker"], ["LIMIT", 10], ["OFFSET", 0]]

[rosa]

Ah! Then you're not running Solid Queue at all? 🤔

[diegocharles]

@rosa, after the update from 1.0.0, do you think Solid Queue is no longer automatically starting up?

It still functions correctly if I lock the gem to 1.0.0.
gem “solid_queue”, “1.0.0”

Am I overlooking something in the Kamal 2 or Solid Queue documentation that indicates that I need to manually set SQ to start in production, similar to how I do in Procfile.dev for local development?

[rosa]

That's very strange 😕 Solid Queue doesn't have anything to automatically start, you had to start it via bin/jobs or with Puma. This hasn't changed and it's been always documented. There's something else going on here.