how do you offload asynchrous tasks out of your rails request cycle? we needed this ability in boomloop.com, where we write data into a statistics database.
frankly, i find backgroundrb a bit scary. rumours about instability persist, and it seems like a lot of weight for a little problem. what else is out there? topfunky has a few ideas here. after checking out a few of those options, i asked evan weaver of chow.com what he thought. he said:
…If you really need a queue, use Starling, which Blaine Cook of Twitter released, like, yesterday. Or SQS if you need really huge storage. If you just want to fire and forget a local process as you say, I think Spawn is pretty good ( http://rubyforge.org/projects/spawn ). I haven’t actually used it but seems like the best of the forking bunch. That should eliminate the startup overhead. On the other hand, you don’t get any message reliability or cross-machine scheduling. ..I agree that BDrb is shady (actually all of Drb is shady). ap4r is too
bloated. Thruqueue is promising if you make it past the crazy dependencies list. BackgroundFu is like a worse Spawn.”
result: i wrote a rails plugin called workling that integrates starling into your rails app. it also lets you swap starling for any other system you might want to offload work to, without needing to change your client code. along with starling, i’ve implemented a spawn runner and a local runner.
Client code
First, create analytics_worker.rb in app/workers:
class AnalyticsWorker < Workling::Base
def potential_invited(options)
Hit.create :potential_user_id => options[:potential_user_id], :action => "invited"
end def potential_converted(options)
Hit.create :potential_user_id => options[:potential_user_id], :action => "converted"
end
end
then, call it like this anywhere in your code:
AnalyticsWorker.asynch_potential_invited(:potential_user_id => 1234)
Starling Runner
This uses Twitter’s Starling to enable your asynch code to run on different VMs. Activate it like this in your environment:
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
The starling runner takes care of several things:
- mapping of queue names to worker code. this is done with Workling::ClassAndMethodRouting, but you can use your own by sublassing Workling::Routing. Some examples of Worker to queue routing: AnalyticsWorker class above routes the queues ‘analytics_worker:potential_invited’ and ‘analytics_worker:potential_converted’ to those methods on AnalyticsWorker. If you put your worker in a module, the queue will start with the_module_name:<worker_class>. don’t worry about any of this if you’re not dealing directly with the queues.
- there’s a client daemon that waits for messages and dispatches these to the responsible workers. if you intend to run this on a remote machine, then just check out your rails project there and start up the starling client.
Other runners
workling comes with a spawn based runner. this currently seems to be one of the better spawning/forking options. use this if you don’t want to run a separate starling server in your setup.
Getting started with starling and workling
start by installing starling and a memcached client. then install workling into your rails application. install the spawn plugin if you want the option of swapping this in instead of starling.
sudo gem install starling --include-dependencies
sudo gem install memcache-client --include-dependencies
./script/plugin install http://svn.playtype.net/plugins/workling/
./script/plugin install http://spawn.rubyforge.org/svn/spawn/
now you can start the little twitter birdie, followed by workling. and off you go!
mkdir /var/spool/starling
sudo starling -d
script/workling_starling_client start
the leading workling repository is here: it://github.com/purzelrakete/workling.git. go forth and fork!
