Testing concurrency in rails

[2013-05-15] Updated code example to work with postgresql. Many thanks to Carlos Alonso in the comments for finding both the problem and the solution!

Concurrency is hard to get right, and unfortunately it is hard to test as well. Let’s start with a simple controller action.

class CountersController < ApplicationController
  def increment
    counter = Counter.find(params[:id])
    counter.increment!(:value)
    render text: counter.value
  end
end

Looks good, right? Well, there is a race condition between the find and the increment!. Let’s see if we can add a failing test for this before fixing it. For this, we’ll use the gem fork_break which allows you to start subprocesses in which to execute your code, while synchronizing them from the parent process using breakpoints.

Now, for the following to work, you have to make sure that your tests are not being run in transactions.

Rspec.config do |config|
  ...
  # Make sure this is not set to true
  config.use_transactional_fixtures = false

  ...
end

Next, let’s add a failing test case.

require 'spec_helper'

describe CountersController do
  it "works for concurrent increments" do
    counter = Counter.create!
    
    # For postgresql we need to disconnect before forking
    # and for other databases, it won't hurt.
    ActiveRecord::Base.connection.disconnect!

    process1, process2 = 2.times.map do
      ForkBreak::Process.new do |breakpoints|
        # We need to reconnect after forking
        ActiveRecord::Base.establish_connection

        # Add a breakpoint after invoking find        
        original_find = Counter.method(:find)
        Counter.stub!(:find) do |*args|
          counter = original_find.call(*args)
          breakpoints << :after_find
          counter
        end

        get :increment, :id => counter.id
      end
    end
    process1.run_until(:after_find).wait
    process2.run_until(:after_find).wait 

    process1.finish.wait
    process2.finish.wait
    
    # The parent process also needs a new connection
    ActiveRecord::Base.establish_connection
    counter.reload.value.should == 2
  end
end
$ rspec spec/controllers/counters_controller_spec.rb

CountersController
  works for concurrent increments (FAILED - 1)

Excellent, a failing test case! Ok, so how do we fix this? For this example, we’ll use pessimistic locking.

class CountersController < ApplicationController
  def increment
    Counter.transaction do
      counter = Counter.find(params[:id], lock: true)
      counter.increment!(:value)
      render text: counter.value
    end
  end
end

However, if we run the spec it just hangs (eventually raising an exception after the database has timed out). The reason for this is that Counter.find is blocking in process2. To fix this, we’ll add have to modify the test somewhat.

require 'spec_helper'

describe CountersController do
  it "works for concurrent increments" do
    counter = Counter.create!

    # For postgresql we need to disconnect before forking
    # and for other databases, it won't hurt.
    ActiveRecord::Base.connection.disconnect!

    process1, process2 = 2.times.map do
      ForkBreak::Process.new do
        # We need to reconnect after forking
        ActiveRecord::Base.establish_connection

        # Add a breakpoint after invoking find        
        original_find = Counter.method(:find)
        Counter.stub!(:find) do |*args|
          breakpoints << :before_find
          counter = original_find.call(*args)
          breakpoints << :after_find
          counter
        end

        get :increment, :id => counter.id
      end
    end
    process1.run_until(:after_find).wait

    # Try to make sure that process2 is blocking in find
    process2.run_until(:before_find).wait
    process2.run_until(:after_find) && sleep(0.1)

    # Now finish process1 and wait for process2
    process1.finish.wait
    process2.finish.wait
    
    # The parent process also needs a new connection
    ActiveRecord::Base.establish_connection
    counter.reload.value.should == 2
  end
end
$ rspec spec/controllers/counters_controller_spec.rb

CountersController
  works for concurrent increments

Huzzah!! (of course, seeing how we changed the test, we need make sure that the original code fails)

Thanks to http://blog.ardes.com/2006/12/12/testing-concurrency-in-rails-using-fork for getting me started on this.

6 thoughts on “Testing concurrency in rails

  1. Hi Petter.

    Congratz for your gem, it’s really interesting, and useful I hope after solving my issue.

    It’s simple, I write a test case just like yours, but when I try to reconnect the parent before doing the final checks it fails with the following error.

    PG::Error: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.

    Can you please help me?

Comments are closed.