How do I track down a memory leak in my Ruby code?

Question

Question

I'm debugging a memory leak in a rake task. I want to see a call stack of:

• Living objects
• What object or line originally allocated those objects

Is this possible with ruby-prof?

If not, what tool should I use?

Setup

Gems

• rails 3.2.16
• event_bus 1.0.0 (https://github.com/kevinrutherford/event_bus)
• activerecord-fast-import (https://github.com/jsuchal/activerecord-fast-import)

Rake task

• Imports a CSV file directly into a MySql database using DATA LOAD INFILE and Active Record objects.

What I've Tried

I've tried the modes

• RubyProf::ALLOCATIONS
• RubyProf::MEMORY

All it says in the documentation is:

RubyProf::ALLOCATIONS Object allocation reports show how many objects each method in a program allocates.

RubyProf::MEMORY Memory usage reports show how much memory each method in a program uses.

This implies that ruby-prof just reports on the total allocation of objects, not just the ones that are living.

I've tried Ruby-Mass and Bloat Check but neither seem to be able to do what I want. Ruby-Mass also crashes because it's finding FactoryGirl objects in memory for some reason...

Answer 1

I did not find ruby-prof very useful when it came to locating memory leaks, because you need a patched Ruby interpreter. Tracking object allocation has become easier in Ruby 2.1. Maybe it is the best choice to explore this yourself.

I recommend the blog post Ruby 2.1: objspace.so by tmml who is one of the Ruby core developers. Basically you can fetch a lot of information while debugging your application:

ObjectSpace.each_object{ |o| ... }
ObjectSpace.count_objects #=> {:TOTAL=>55298, :FREE=>10289, :T_OBJECT=>3371, ...}

require 'objspace'
ObjectSpace.memsize_of(o) #=> 0 /* additional bytes allocated by object */
ObjectSpace.count_tdata_objects #=> {Encoding=>100, Time=>87, RubyVM::Env=>17, ...}
ObjectSpace.count_nodes #=> {:NODE_SCOPE=>2, :NODE_BLOCK=>688, :NODE_IF=>9, ...}
ObjectSpace.reachable_objects_from(o) #=> [referenced, objects, ...]
ObjectSpace.reachable_objects_from_root #=> {"symbols"=>..., "global_tbl"=>...} /* in 2.1 */

With Ruby 2.1 you can even start to track allocation of new objects and gather metadata about every new object:

require 'objspace'
ObjectSpace.trace_object_allocations_start

class MyApp
  def perform
    "foobar"
  end
end

o = MyApp.new.perform
ObjectSpace.allocation_sourcefile(o) #=> "example.rb"
ObjectSpace.allocation_sourceline(o) #=> 6
ObjectSpace.allocation_generation(o) #=> 1
ObjectSpace.allocation_class_path(o) #=> "MyApp"
ObjectSpace.allocation_method_id(o)  #=> :perform

Use pry and pry-byebug and start exploring the memory heap where you think it will probably grow, respectively try different segments in your code. Before Ruby 2.1 I always relied on ObjectSpace.count_objects and calculated the result's difference, to see if one object type grows in particularly.

The garbage collection works properly when the number of objects growing are retested back to a much smaller amount during the iterations as opposed to keep growing. The garbage collector should run all the time anyway, you can reassure yourself by looking into the Garbage Collector statistics.

From my experience this is either String or Symbol (T_STRING). Symbols before ruby 2.2.0 were not garbage collected so make sure your CSV or parts of it is not converted into symbols on the way.

If you do not feel comfortable, try to run your code on the JVM with JRuby. At least the memory profiling is a lot better supported with tools like VisualVM.