Ruby on Rails 3: Streaming data through Rails to client

Question

I am working on a Ruby on Rails app that communicates with RackSpace cloudfiles (similar to Amazon S3 but lacking some features).

Due to the lack of the availability of per-object access permissions and query string authentication, downloads to users have to be mediated through an application.

In Rails 2.3, it looks like you can dynamically build a response as follows:

# Streams about 180 MB of generated data to the browser.
render :text => proc { |response, output|
  10_000_000.times do |i|
    output.write("This is line #{i}\n")
  end
}

(from http://api.rubyonrails.org/classes/ActionController/Base.html#M000464)

Instead of 10_000_000.times... I could dump my cloudfiles stream generation code in there.

Trouble is, this is the output I get when I attempt to use this technique in Rails 3.

#<Proc:0x000000010989a6e8@/Users/jderiksen/lt/lt-uber/site/app/controllers/prospect_uploads_controller.rb:75>

Looks like maybe the proc object's call method is not being called? Any other ideas?

Answer 1

Assign to response_body an object that responds to #each:

class Streamer
  def each
    10_000_000.times do |i|
      yield "This is line #{i}\n"
    end
  end
end

self.response_body = Streamer.new

If you are using 1.9.x or the Backports gem, you can write this more compactly using Enumerator.new:

self.response_body = Enumerator.new do |y|
  10_000_000.times do |i|
    y << "This is line #{i}\n"
  end
end

Note that when and if the data is flushed depends on the Rack handler and underlying server being used. I have confirmed that Mongrel, for instance, will stream the data, but other users have reported that WEBrick, for instance, buffers it until the response is closed. There is no way to force the response to flush.

In Rails 3.0.x, there are several additional gotchas:

• In development mode, doing things such as accessing model classes from within the enumeration can be problematic due to bad interactions with class reloading. This is an open bug in Rails 3.0.x.

• A bug in the interaction between Rack and Rails causes #each to be called twice for each request. This is another open bug. You can work around it with the following monkey patch:

  class Rack::Response
    def close
      @body.close if @body.respond_to?(:close)
    end
  end
  

Both problems are fixed in Rails 3.1, where HTTP streaming is a marquee feature.

Note that the other common suggestion, self.response_body = proc {|response, output| ...}, does work in Rails 3.0.x, but has been deprecated (and will no longer actually stream the data) in 3.1. Assigning an object that responds to #each works in all Rails 3 versions.