Every now and then some of our web dynos were suffering from what looked like the noisy neighbor problem. Symptoms included overall degraded performance on just one of our web dynos and, in some cases, numerous timeouts for our users. According to heroku support, the only known immediate cure for this ailment was issuing a restart for the problem dyno which was obviously not optimal. The other suggested remedy was isolating ourselves from our neighbors by moving to our own PX sized instance. Since a PX instance had 12 times the memory of a 1X instance, we could spawn many more unicorn workers for each of our web dynos. Of course, PX dynos are expensive and we would only need a dyno of that size within our production environments. Instead of explicity setting the number of unicorn workers within each environment using a config variable, we thought it would be easier to dynamically set the number of workers based on the dyno size. The only issue is that heroku does not make the dyno size available to the process at runtime. Luckily somemone had already put together a script solve this problem for the puma_auto_tune project. That script simply leverages ulimit to figure out our limits on memory. We borrowed that script and put the following within our unicorn.rb:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
The default_ram method calculates the amount of memory in MB’s available within the current environment. We then divide that number by the estimated size of each unicorn worker to get the number of workers we want to spawn in this dyno. In the example case 8192/250 comes to 32 unicorn workers.
This allowed us to use PX web dynos in production and 1X or 2X within other environments like staging or dev. Since the PX dyno change, no more noisy neighbors causing any trouble within production.