Building VTS

The Official VTS Engineering Blog

Dynamic Heroku Unicorn Worker Count

By Karl Baum

Every now and then some of our web dynos were suffering from what looked like the noisy neighbor problem. Symptoms included overall degraded performance on just one of our web dynos and, in some cases, numerous timeouts for our users. According to heroku support, the only known immediate cure for this ailment was issuing a restart for the problem dyno which was obviously not optimal. The other suggested remedy was isolating ourselves from our neighbors by moving to our own PX sized instance. Since a PX instance had 12 times the memory of a 1X instance, we could spawn many more unicorn workers for each of our web dynos. Of course, PX dynos are expensive and we would only need a dyno of that size within our production environments. Instead of explicity setting the number of unicorn workers within each environment using a config variable, we thought it would be easier to dynamically set the number of workers based on the dyno size. The only issue is that heroku does not make the dyno size available to the process at runtime. Luckily somemone had already put together a script solve this problem for the puma_auto_tune project. That script simply leverages ulimit to figure out our limits on memory. We borrowed that script and put the following within our unicorn.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#borrowed from the puma_auto_tune project
def default_ram
  result = `./script/heroku/heroku_ulimit_to_ram`
  default = if $?.success?
              Integer(result)
            else
              puts "Unable to determine memory limit.  Defaulting memory size to 512"
              512
            end
  puts "Default RAM set to #{default}"
  default
end


CONSERVATIVE_WEB_DYNO_SIZE = 250

def dynamic_number_of_worker_processes
  #dynamically calculate based on available memory
  @number_of_worker_processes ||= default_ram/CONSERVATIVE_WEB_DYNO_SIZE
end

def number_of_worker_processes
  #allow for setting the number if needed
  ENV.fetch('NUMBER_OF_UNICORN_WORKERS', dynamic_number_of_worker_processes)
end

puts "setting number of unicorn worker processes to #{number_of_worker_processes}"
worker_processes number_of_worker_processes

The default_ram method calculates the amount of memory in MB’s available within the current environment. We then divide that number by the estimated size of each unicorn worker to get the number of workers we want to spawn in this dyno. In the example case 8192/250 comes to 32 unicorn workers.

This allowed us to use PX web dynos in production and 1X or 2X within other environments like staging or dev. Since the PX dyno change, no more noisy neighbors causing any trouble within production.