Feb
starling and asynchrous tasks in ruby on rails
how do you offload asynchrous tasks out of your rails request cycle? we needed this ability in boomloop.com, where we write data into a statistics database.
frankly, i find backgroundrb a bit scary. rumours about instability persist, and it seems like a lot of weight for a little problem. what else is out there? topfunky has a few ideas here. after checking out a few of those options, i asked evan weaver of chow.com what he thought. he said:
…If you really need a queue, use Starling, which Blaine Cook of Twitter released, like, yesterday. Or SQS if you need really huge storage. If you just want to fire and forget a local process as you say, I think Spawn is pretty good ( http://rubyforge.org/projects/spawn ). I haven’t actually used it but seems like the best of the forking bunch. That should eliminate the startup overhead. On the other hand, you don’t get any message reliability or cross-machine scheduling. ..I agree that BDrb is shady (actually all of Drb is shady). ap4r is too
bloated. Thruqueue is promising if you make it past the crazy dependencies list. BackgroundFu is like a worse Spawn.”
result: i wrote a rails plugin called workling that integrates starling into your rails app. it also lets you swap starling for any other system you might want to offload work to, without needing to change your client code. along with starling, i’ve implemented a spawn runner and a local runner.
Client code
First, create analytics_worker.rb in app/workers:
class AnalyticsWorker < Workling::Base
def potential_invited(options)
Hit.create :potential_user_id => options[:potential_user_id], :action => "invited"
end def potential_converted(options)
Hit.create :potential_user_id => options[:potential_user_id], :action => "converted"
end
end
then, call it like this anywhere in your code:
AnalyticsWorker.asynch_potential_invited(:potential_user_id => 1234)
Starling Runner
This uses Twitter’s Starling to enable your asynch code to run on different VMs. Activate it like this in your environment:
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
The starling runner takes care of several things:
- mapping of queue names to worker code. this is done with Workling::ClassAndMethodRouting, but you can use your own by sublassing Workling::Routing. Some examples of Worker to queue routing: AnalyticsWorker class above routes the queues ‘analytics_worker:potential_invited’ and ‘analytics_worker:potential_converted’ to those methods on AnalyticsWorker. If you put your worker in a module, the queue will start with the_module_name:<worker_class>. don’t worry about any of this if you’re not dealing directly with the queues.
- there’s a client daemon that waits for messages and dispatches these to the responsible workers. if you intend to run this on a remote machine, then just check out your rails project there and start up the starling client.
Other runners
workling comes with a spawn based runner. this currently seems to be one of the better spawning/forking options. use this if you don’t want to run a separate starling server in your setup.
Getting started with starling and workling
start by installing starling and the fiveruns memcached client. then install workling into your rails application. install the spawn plugin if you want the option of swapping this in instead of starling.
gem sources -a http://gems.github.com/
sudo gem install starling-starling
script/plugin install git@github.com:purzelrakete/workling.gitthe fiveruns-memcache-client is pulled in as a dependency to starling-starling. now you can start the little twitter birdie, followed by workling. and off you go!
mkdir /var/spool/starling
sudo starling -d
script/workling_client start
the leading workling repository is here: it://github.com/purzelrakete/workling.git. go forth and fork!
Comments
There are 72 Comments for this post. Write comment →
Looks nice and lightweight…Didn’t jump into the docs or the source – but is there any way to reference running worklings to manage things like progress bars?
I’d like to jump off BDRB asap if possible…Moving everything to one system would be dope.
Interesting alternative, thanks for the info.
Notice that I use backgroundrb with full success and after reading your article it appears to me that BDrb is no more difficult to use than your solution. On our side we are fully happy with BDrb which has been working with Rails to execute intensive images processing for more than 3 months and without any failure or instability. I agree that at the begining I had to dive half a day into the documentation and code.
;¬)
Oddly enough, backgroundrb doesn’t use drb anymore. it has been completely rewritten.
funny that you would use a library coming from twitter, a web app widely known for its instability and architectural problems, even though you previously said you were looking for a more stable solution
@seth: you can do this now over your database. we’re working on a status indicator for a workling in our app at the moment, and this works over a return queue in starling. i’m not sure where this fits into workling yet. i can show you some code if you like!
@fred: i dunno, call me superstitious ;). i just heard from too many people that want to switch away from it. that said, i’m sure it works great for some people.
@joe: really? i saw that it was recently completely rewritten. do you know what it’s using now?
@heri: time will tell how stable starling is in production use – i’ll be sure to post about my own experiences! one thing’s for sure: twitter built starling to address exactly those issues. it’s an open project which is in heavy production use – that gives me confidence. and it only does one thing, and does that well. you’ll see starling continuing to improve over time. already, chris wansrath has replaced the threading with eventmachine.
If I’m understanding correctly, the benefit is for long running Rails processes? If I have issues with concurrency due to process size in memory, this wouldn’t help or would it?
Thanks, Chirag
Well, new version of BackgrounDRb is based on Packet.
http://code.google.com/p/packet/
Packet is a network programming library in the spirit of EventMachine and yet it has nice functionality of letting you attach callbacks to workers running in separate process. It can even let you invoke callbacks running on worker in different machine and stuff like that. When I took over project it was based on DRb, but since then I have removed DRb and BackgrounDRb is 100% based on evented model of network programming.
I wish, people will check their facts before making any claims, I am kinda getting tired of fighting this FUD within community. There are few outstanding issues, but BackgrounDRb supports many features that other similar alternatives doesn’t offer. And I am working on it.
Evan Weaver might be a little wrong saying BackgroundFu is a worse spawn. I would say the opposite, but I might be biased ;) Better check it out for yourself. http://svn.trix.pl/public/background_fu/
It is far better than spawn. It gives you worker monitoring (progress monitoring) stopping and even restarting background jobs for free. It works reliably, uses db for messaging so it runs cross-network. Give it a try. Oh, and it proved reliability in quite a stressful environment where background jobs have to talk to bunch of webservices, etc.
If BackgroundFu doesn’t suit your needs, try BackgrounDRb which is a little more complicated but very reliable (yeah, all the FUD is outdated) and has more features.
This library looks like it could solve my problem (fire and forget notifications to another system). However, I’m having trouble figuring out where to put in the proper requires in my Rails app. I get:
/Library/Ruby/Gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:266:in `load_missing_constant’: uninitialized constant Workling (NameError)
I put in the Workling:Remote.dispatcher line in my Rails::Initializer.run block in environment.rb.
Can you please list the exact requires and locations you need to add in a rails app?
@andrew: configure your dispatcher anywhere after the initializer block. the default dispatcher is a NotRemoteRunner, so really you should only have to put a line in your production.rb for the starling dispatcher.
/r
VIM user??? Stray qs under your sidebar :-)
First off, awesome plugin. It’s so nice to have a simple way to integrate to this new message queue.
I am having some issues when trying to integrate it into my app’s testing environment. Here’s what I’ve come across.
1. I couldn’t find a built in way to run working_starling_client in test mode. I was able to tweak listen.rb (http://pastie.caboo.se/161313) to at least accept a—RAILS_ENV arguement and it seems to be loading the right Rails environment.
2. Even when Rails gets loaded in a test environment, there’s still issues with it interacting with the test database. The initial fixtures will load just fine, but it’s not finding any objects created in an individual test. This could be an issue with the build and teardown built into testing, but I thought I would at least mention it.
Thanks again for putting this out there!
As with anything, there are different tools for different jobs. Spawn is great for the running small slow bits of code that you don’t have to worry about, like sending emails or running slow non-critical database updates. If you want something quick and easy to use, try spawn.
If you need monitoring and/or messaging, then look at other alternatives like workling. These tools can be harder to use or setup but it may be just what you need.
Me likeum workling
workling > backgroundrb
Thank you for making this SIMPLE.
Thanks for the great plugin. I was using backgroundrb for my background processing. It worked, and it was very simple to setup. However, I found myself fighting with it too often, and I didn’t like all the processes it created. I switched over to Workling, and I’m not going back.
I did make one modification if you’re interested. In my app, I have many worker methods that execute very quickly, but others could take minutes to complete. To prevent a 20 minute task from stalling all the other tasks, I updated Poller to create a listener thread for each worker class. The only limitation I have is an assumption that routing has at least one thread per worker class.
Here is my updated poller.rb
module Workling
module Starling
class Poller
cattr_accessor :sleep_time
@@sleep_time = 1
end
ActiveRecord::Base.allow_concurrency = true
end
threads = []
Workling::Discovery.discovered.each do |clazz|
threads 0 # don’t sleep if methods were called
end
def dispatch!(clazz)
n = 0
for queue in @routing.keys
begin
next unless @routing[queue].class == clazz # Does a little extra work, but not a huge deal…
result = @connection.get(queue)
if result
@routing[queue].send(@routing.method_name(queue), result)
n += 1
end
rescue Exception => e
puts msg = “FAILED to process queue #{ queue }. #{ @routing[queue] } could not handle invocation of #{ @routing.method_name(queue) } with #{ result.inspect }: #{ e }.”
RAILS_DEFAULT_LOGGER.error(msg)
end
end
return n # return number of worker methods called
end
Hey there Dave,
glad you like Workling :). I think I’ll move it over to github so that people can work on it more easily.
Making the worklings concurrent with the StarlingRunner was next on the list, so just as well you brought it up ;).
There’s two ways I was considering. First, just start cluster of listeners. This is how they do it at twitter – they have a whole bunch of processes polling against starling. The other alternative would be to simply wrap the dispatch with spawn. The spawn plugin nicely handles concurrency, and you can easily switch between using threads or forking.
Having your pick of workling starling clients is probably the easiest thing to do, since everybody has different requirements.
thanks for the input, glad to see people like this :)
/r
I posted a patch with my changes at my blog. The nice thing about adding threads is that I can still go to a cluster of listeners for scale, but the workers will still stay independent. I didn’t go the thread route for scale. I went that way so a 20 minute call to one worker method wouldn’t hold up all the other workers. I didn’t want additional processes because them I’m back to the same problem I had with backgroundrb—too many processes sucking up all my memory.
One gotcha I found is that you have to make sure you keep your database connection alive. If your site has slow periods, ActiveRecord will drop your connections. Add the following to your listening loop:
unless ActiveRecord::Base.connection.active?
unless ActiveRecord::Base.connection.reconnect!
RAILS_DEFAULT_LOGGER.fatal(“FAILED – Database not available”)
break
end
end
Thanks again for the great plugin!
First off, thanks for a great plugin. We’re using it with great success. I’ll be forwarding you some capistrano and init.d scripts I used for our configuration.
Is there a mailing list for workling? I’d love to have somewhere to talk about workling techniques.
I have a specific issue I’m hoping can be solved. We’re using workling for sending SOAP notifications for specific events. The SOAP endpoint should be considered unreliable so I wanted to maintain a retry queue. I was hoping to use workling for that as well. I can fire off a specific Retry worker but ideally, the handler should be part of a periodic daemon that sweeps for retries and goes to sleep for intervals. Anyone have a pattern to accomplish this? Do I need to skip working and go directly to Starling/memcache?
@andrew: it’s pretty cool to see people starting to use this :). at the moment, you can’t really go anywhere other than this article to talk about workling. i plan to change this once i get boomloop.com out the door next week! i’ll move it onto github, put some rdoc up, and start a forum for it. in fact i’ll set up something automatic so that all of our projects get said support up and running.
i’d be quite interested to hear any solutions with regards to your retry problem. @dave also describes a common requirement, and me, i’d like to see something like child worklings which can be rejoined into the parent workling.
at the same time, i aim to keep the plugin small, grokkable and simple to use.
any suggestions highly appreciated!
/r
workling is up on github: git clone git://github.com/purzelrakete/workling.git. nice!
@rany Can’t wait to see it up on github. I planned to start with what @dave did and add my own ideas. Github should make it easier for us to work on it. Workling is quickly become very important to our pipeline.
Oh, I see in another comment that someone started a github project. Should we start using that as a jumping off point?
My basic solution:I’m going to use Ruby threads despite the possibility of hanging the whole ruby process since workling_client can be restarted separate from the Rails app.
Other changes I’ve done/working on:@andrew – really great to see workling is helping people :)
the person that started the github project is me ;) it://github.com/purzelrakete/workling.git is now the leading repository. i’ll commit back to svn though.
i know you’ve posted stuff here and in other blogs already, but the easiest thing would be for you to fork off github. that way, i can easily pull in your changes.
i’ll get a forum organized this week!
@rany Will do. I already did a simple fork/pull with Jamis Buck on a capistrano change. GitHub made it very easy. I’ll clean up my patches and get them up by end of week.
i used workling with starling and made asynchronous call.getting the following exception
could not handle invocation of my_queue with {:uid=>”uploadedsong_workers:my_queue:5b3f2b088432a7ac99593d7b59d99714”}: wrong number of arguments (1 for 0).
wrong number of arguments (1 for 0)** INT signal received.
i have a worker UploadedsongWorkers with a method my_queue().I am recieving this exception while calling UploadedsongWorkers.asynch_my_queue.
help me on this.If the call is synchronous i dont get this issue.
@Rajkumar: it looks like your worker method doesn’t take any arguments.
Workers need to specify one options = {} argument. this will automatically be populated with :uid, which is the uid of this job. You don’t have to worry about this, but it’s useful if you want to return results using a return store.
hope this helps!
/r
I’ve been trying to work the demo into my rails app but have this error:
FAILED to process queue my_workers:do_something_big. # could not handle invocation of do_something_big with nil: protected method `send’ called for #.
Any idea what I’m missing? In the meantime I’ll dig deeper into debugging…
Apparently my problem above is fixed by removing the ‘Ruby-MemCache’
Thanks rany.
But even after adding parameters i am getting the same exception.
I didnt understand how synchronous call works fine with the same code.
But i am trying to process starling queue asynchrounously.Please i need help on this.
class UploadedsongWorker < workling::Base
def my_queue( options = {})
end
end
And i am calling in the way,
UploadedsongWorker.asynch_my_queue()
in development.rb i have added the following for asynchronous
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
fwiw, this works nicely with sparrow as well as starling. I’ve had a few strange issues with starling, but it’s most likely operator error. :-)
Keep getting this from the poller:
could not handle invocation of do_something with nil: undefined class/module Xxxxx
where Xxxxx is an AR model in the Rails app and the asynch task is being called like:
Workling::Remote.run(:test_worker, :do_something, { :xxxxx => Xxxxx.find(3) })
Probably operator error.. what am I doing wrong?
Hello,
Great article and it helped quite a bit. One problem I have however is that I’m getting the NameError when I have
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new in “development.rb” or “production.rb”. When I moved this to “environment.rb” the error went away and it worked fine.
Can someone post the requires needed to configure this in the specific environment?
Thanks.
Hi there
Really like Workling, and I’m planning on using it in a high-traffic messaging app. I’m working on.
I’m using the github fork that includes Dave Dupre’s patches, and it seems to be working pretty well.
One question;
I’m planning to run multiple workling worker processes, so I’ve changed ‘multiple’ to ‘true’ in script/workling_starling_client. So, I can start multiple workers by invoking that script multiple times.
Is there a cleaner way to control the number of workers? Ideally, I’d like to be able to specify a number in a config file and have that many workers spawn. I know I can wrap a shell script around the workling_starling_client script, but that seems like a bit of a hack!
Thanks again for Workling.
David
This realy great!!
I find background jobber some month. And you giv me very nice sample!
Thanks!
This isn’t an informative comment, but it’s always funny when you randomly stumble across people using (or at least mentioning) your work.
I was checking out Workling and Starling/Sparrow – been having a helluva time with Bj and before that, BackgrounDrb – and I saw in the readme an example using our Blackbook gem. Radical!
Workling looks great and I’ll be moving to it soon.
I intermittently get this error:
I, [2008-06-26T06:23:00.076982 #29170] INFO—: Got exception while closing socket: closed stream
D, [2008-06-26T06:23:00.077248 #29170] DEBUG—: /Library/Ruby/Gems/1.8/gems/starling-0.9.3/lib/starling/server.rb:169:in `close’
Any one encountered this before?
Workling seems to poll memcache for starling messages every two seconds. This means a delay before a worker starts on a job.
Out of curiosity, I removed the sleep between polls and got a huge jump in throughput.
Is this sleep between polls necessary?
Does not having it..swamp memcache?
Any other recommendations for minimizing the delay in jobs starting?
Thanks Andy
default poll is 1 second.
Adding sleep_time to config/starling.yml can make it faster.
Making it faster drives the cpu load up.
development:
listens_on: localhost:22122
sleep_time: 0.001
Great plugin. Thanks. I’m trying to get it working with my functional tests, but can’t seem to get it to connect to Memcache in a test environment.
I’ve tried specifying memcache settings in test and using NotRemoteRunner explicitly in test, and still I get this error:
Workling::WorklingError: MemCache::MemCacheError – No connection to server
When I run any test that uses a workling call. Any suggestions?
Do you plan on doing a SQS runner?
Do you plan on making a SQS poller?
Are there any docs for all this stuff?
I would love to know how people are setting this up in development/staging/production.
Also, where does the starling.yml file come into play? What are the options, who reads it, etc.?
Arun, I also have this problem and I still didn’t find the reason. If anybody knows what’s the cause of the error, please let me know.
Did a forum ever get started? I’m about to prototype using workling/starling to handle backgrounded tasks for multiple instances of an app (each task needing to be run against a specific database). It seems like this comment section probably isn’t the most appropriate place to ask questions/get pointers.
I got this error
FAILED to connect with queue test_workers__potential_test: No connection to server }
Listener thread TestWorker failed to connect to memcache. Resetting connection.
This error is repeating
Can someone help me?
When I run this in the console…
>> MyWorker.asynch_do_something_amazing()
=> “my_workers:do_something_amazing:aa4073f2c3ef7f61ff59389e31f1044c”
..I’m getting the following appearing in my development.log file over and over again (it just keeps going):
Listener thread MyWorker processed 1 queue items
Calling MyWorker#do_something_amazing({:uid=>”my_workers:do_something_amazing:a42231a44b4565a1ac57bfc067f18553”})
Is this correct? It just calls that method infinitely!
After studying the many different solutions for asynchronous processing in Rails and going the wrong way with BackgroundDrb (too complex) and BackgroundFu (didn’t like the database queue), I started using Workling with Starling. It is relatively simple, it has low overhead, it queues jobs, it has low latency, it returns results and last but not least it just works.
However I think Workling deserves better documentation. This article is not the best introduction and the README has incomplete instructions and even contains errors.
RailsCastas make screencast with startling
look http://railscasts.com/episodes/128-starling-and-workling
I’ll second Maarten on the documentation issue.
I’m still finding essential information in blog posts and presentations written within the last few days (http://saizai.livejournal.com/895453.html).
Logging doesn’t seem to work properly for workling, and the only information I’ve been able to get about what’s going on under the hood has been from the status.rb file hidden in the workling plugin directory.
I know that starling/workling is not what one might call “a mature, stable library with years of use in production”, but overall the process has been irritatingly opaque, and while it works (spotty, but I’m chalking that up to my using 1.5.0 of memcache-client until I know otherwise), I still feel like I’m throwing background jobs into a black box.
Hopefully upgrading to fiveruns-memcache-client and starling-starling will sort me out. Time will tell.
you’re all right about the documentation. i’m consolidating and updating everything today.
@benjamin: workling+starling is has seen some pretty solid production use with millions of msgs per day. they’re even using it at getty images.
i guess what’s missing though is stable version tags. starling has been a bit of a moving target, and workling itself has evolved quite a bit over time, too. the most recent version went up just before railsconf berlin. i’ll document it and fix any problems, then give it a stable version tag.
you should be using fiveruns memcahced client together with the starling-starling gem on github. afaik the rubyforge version is pretty ancient.
as far as opaque – it would help if you listed the problems you had. i’ll look into them. simplicity is one of the main aims of workling, so we don’t want opaque anything ;)
Hi, cant read result of method.
My Ruby version:
ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0]
my environment.rb containg:
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
Workling::Return::Store.instance = Workling::Return::Store::MemoryReturnStore.new
test_worker.rb
class TestWorker > @uid = TestWorker.asynch_search()
=> “test_workers:search:93f344057afd6bc34d8af46fc17455fc”
>> Workling::Return::Store.get(@uid)
=> nil
The log file:
Calling TestWorker#search({:uid=>”test_workers:search:c257600aaf61a92eecfd0416719c5b3b”})
[4;35;1mAnswer Load (0.031258)[0m [0mSELECT * FROM `answers` [0m
Listener thread TestWorker processed 1 queue items
So the question. What i can do wrong?
Other issue. At some documentation seen that method get can be used with Workling::Return module. In my case it does not work.
>> Workling::Return.get(@uid)
NoMethodError: undefined method `get’ for Workling::Return:Module
from (irb):3
Is this correct?
Hi Valery,
the reason is that you’re using the memory store together with starling. This doesn’t work, because starling is running in a different process. So you have to use something that can go across processes with starling, which currently is only the starling return store.
Workling::Return.get is wrong – it should be Workling::Return::Store.get. I’m just getting the documentation into a spanking shiny accurate state ;).
hope this helps!
@von Valery
The documentation is wrong. Check out:
http://github.com/gaffneyc/workling/commit/184ce2edf080a1862733b54be9105d17f4ee8e5e
You will need to restart starling and its client after you change the worker.
Would be a nice thing … if it worked…
starling -P tmp/pids/starling.pid -q log/
script/workling_starling_client start -tStarting at 127.0.0.1:22122.
I, [2008-10-19T18:03:21.598329 #9581] INFO—: Starling STARTUP on 127.0.0.1:22122
=> Loading Rails…
client: ContactWorker.sync_contact_now(…
—> NOTHING (no errors, …) in 95% of all cases (the same call every time)
only the contact_workers__contact_now log file shows, that something should happen. In 5 % of all cases it workes like described.
Not very useful at all :(
@Anton I’ll contact you and help you to figure out what’s going wrong – i’ve got a free moment :)
Great peace of software. Installed, configured, programmed, success … in about 5 minutes :)
Btw.: On railscasts there is a screencast of Ryan Bates (http://railscasts.com/episodes/128-starling-and-workling) about starling and workling, that gives a good introduction.
Forget it, installed everything a second time on a second computer and it works perfect… (everything up and running in about 5 minutes)
Thanks for the superb support!!!
In your example what if your workers where setup for a sleep of 60 and you wanted to have the potential_invited(options) worker method call the potential_converted(options), how would you do this and have the 2nd worker still wait 60s. What I have done is have my worker call back a static method in my controller and that method calls my worker again, however the worker doesn’t us the sleep timer for the 2nd run.
any one have a Monit config for a workling deamon?
Hi I have started looking at adding support for a configurable pool of threads for each task type (primarily for IO-bound processes that can be done in parallel). I started a fork on github and am trying to figure out the best place to discuss a few questions with you in regards to what objects need to be thread safe. I’d rather not do it in github messages, its really info that is best kept in a public place, a mailing-list? the wiki? here? It looks like this project is picking up steam so a common place to talk might be helpful.
@samhandley http://groups.google.com/group/workling. no idea what took me so long ;)
Have you or anyone tried using starling (and workling) on Windows?
I have currently installed starling-starling gem (0.9.9) on my Windows XP machine.
I am able to start Starling. However, I get a Rubygems Load Error when I attempt a require ‘starling’.
Have you or anyone else encountered this problem with starling-starling gem on Windows?
Thanks,
Sam
Have you or anyone tried using starling (and workling) on Windows?
I have currently installed starling-starling gem (0.9.9) on my Windows XP machine.
I am able to start Starling. However, I get a Rubygems Load Error when I attempt a require ‘starling’.
Have you or anyone else encountered this problem with starling-starling gem on Windows?
Thanks,
Sam
I installed the gem for starling and plugin for workling. Then i added a file workling.yml to my config dir and added to it.
workstation:
listens_on: localhost:22122
sleep_time: 2
reset_time: 30
listeners:
Util:
sleep_time: 20
memcache_options:
namespace: myapp_development
and in my workstation.rb i have added
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
Then i created a directory for workers under app folder and added a worker class to it.
Now when i invoke that process asynchronously i get message as
Calling MusicWorker#create_recos({:uid=>”music_workers:create_recos:e3d01fc0a091f5f27c499996146bfdb5”, :test_id=>1})
Listener thread MusicWorker processed 1 queue items
Is this fine but i have added some print stmts to my asynchronous method and none of them are appearing as of now.
Is my starling working or am i missing something.
I was wanting to know if setting the sleep time is the time it waits between sending each message in the queue? I have a server that only allows 500 messages an hour and I want to limit certain messages that would go out the entire database.
Thank you,
Sean McGilvray
Having quite a few problems setting up a simple spawn runner. I initially had used starling, and when I swapped back to spawn it was working fine until I realised starling was still running. As soon as I killed it, workling wouldn’t load.
If I remove any listens_to values from the workling.yml (cos spawn doesn’t listen), it complains about invalid config.
If I put random port numbers in there even though there are no starlings running, it complains about not being able to connect in memcache_queue_client.rb:68. The odd thing here is that it always tries to connect to the test setting, regardless of my RAILS_ENV.
I have definitely set in my config/environment.rb
Workling::Remote.dispatcher = Workling::Remote::Runners::SpawnRunner.new
I don’t have starling or memcache-client installed anymore, so I dunno why it’s trying to use memcache_queue_client or trying to connect to it.
When i try and ./script/server i get:
WORKLING: couldn’t find a memcache client – you need one for the starling runner.
I added debugging to workling and see that it is using SpawnRunner, but still trying to connect to memcache-client. I don’t see a client suitable for spawn?
hi, if anyone try starling on windows???
I’m attempting to use RabbitMQ with Workling. On the outset, all the pieces seem to integrate correctly. My workling class gets invoked via the queue, but it’s options argument is a String and not a Hash. I had Workling working with Starling and didn’t have this issues and decided to try RabbitMQ. This indicates to me it’s either a problem with the amqp client or the workling amqp code. I’m trying to dig through the source but nothing is jumping out at me. Do you have any idea what might be wrong? I’ve verified my code works if I invoke the worker method directly and do not work via the queue. Something is getting lost in translation and I’m afraid it might be with amqp, but I have no idea yet. The doc seems to indicate it’s been or is being used with RabbitMQ. Anyone have any success with this?
Hi!
I faced with such kind of problem
$ ./script/workling_client start
$ ps ax| grep workling
14841 ? S 0:00 workling_monitor
14843 ? D 0:01 workling
14845 pts/44 S+ 0:00 grep workling
$ ./script/workling_client stop
$ ps ax| grep workling
14841 ? S 0:00 workling_monitor
14848 pts/44 S+ 0:00 grep workling
So, how should I stop workling in correct way?
P.S. I didn’t do any changes in config. I just installed the plugin and tried to start/stop it.
Since SQS was mentioned a couple times in the comments:
I have implemented a Workling SQS Client. Check out my blog post for further info:
http://www.digitalhobbit.com/2009/04/04/workling-and-amazon-sqs/
Workling in development environment successfully logs to development.log
However, when in production environment, nothing shows up in the production.log, but the work gets done because it sends the email fine.
I haven’t changed the default rails logger levels, and I’m outputting from the worker at info level. Also the workling is started as the same uid and gid as the production.log so has permission to access it.
Do you have any suggestions where to begin to track down the problem?
Thanks!
I’m having a similar problem to @von Danial .
I want to use the SpawnRunner.
I have Workling::Remote.dispatcher = Workling::Remote::Runners::SpawnRunner.new at the bottom of my environment.rb file.
When I try and start mongrel I get the error uninitialized constant MemCache (NameError).
Why is workling trying to load memcache?
I would rather not install gems that I don’t need …
is there a way to see if a process has finished (without using the return store)?
like:
MyWorker.asynch_do_something(id)
and then
MyWorker.active?(id)
or something like that?
Write a comment
Required in bold.