Oct
Workling Version 0.3 Released
Workling 0.3 is up on GitHub and svn.playtype.net. This release improves error logging and includes an updated README - finally ;). Here it is….
Workling
Workling gives your Rails App a simple API that you can use to make code run in the background, outside of the your request.
You can configure how the background code will be run. Currently, workling supports Starling, BackgroundJob and Spawn Runners. Workling is a bit like Actve* for background work: you can write your code once, then swap in any of the supported background Runners later. This keeps things flexible.
Installing Workling
The easiest way of getting started with workling is like this:
script/plugin install git://github.com/purzelrakete/workling.git
script/plugin install git://github.com/tra/spawn.git
If you’re on an older Rails version, there’s also a subversion mirror wor workling (I’ll do my best to keep it synched) at:
script/plugin install http://svn.playtype.net/plugins/workling/
Writing and calling Workers
This is pretty easy. Just put cow_worker.rb into into app/workers, and subclass Workling::Base:
# handle asynchronous mooing.
class CowWorker < Workling::Base
def moo(options)
cow = Cow.find(options[:id])
logger.info("about to moo.")
cow.moo
end
end
Make sure you have exactly one hash parameter in your methods, workling passes the job :uid into here. Btw, in case you want to follow along with the Mooing, grab ‘cows-not-kittens’ off github, it’s an example workling project. Look at the branches, there’s one for each Runner.
Next, you’ll want to call your workling in a controller. Your controller might looks like this:
class CowsController < ApplicationController
# milking has the side effect of causing
# the cow to moo. we don't want to
# wait for this while milking, though,
# it would be a terrible waste ouf our time.
def milk
@cow = Cow.find(params[:id])
CowWorker.asynch_moo(:id => @cow.id)
end
end
Notice the asynch_moo call to CowWorker. This will call the moo method on the CowWorker in the background, passing any parameters you like on. In fact, workling will call whatever comes after asynch_ as a method on the worker instance.
Worker Lifecycle
All worker classes must inherit from this class, and be saved in app/workers. The Worker is loaded once, at which point the instance method create is called.
Calling async_my_method on the worker class will trigger background work. This means that the loaded Worker instance will receive a call to the method my_method(:uid => "thisjobsuid2348732947923").
Exception handling in Workers
If an exception is raised in your Worker, it will not be propagated to the calling code by workling. This is because the code is called asynchronously, meaning that exceptions may be raised after the calling code has already returned. If you need your calling code to handle exceptional situations, you have to pass the error into the return store.
Workling does log all exceptions that propagate out of the worker methods.
Logging with Workling
RAILS_DEFAULT_LOGGER is available in all workers. Workers also have a logger method which returns the default logger, so you can log like this:
logger.info("about to moo.")
What should I know about the Spawn Runner?
Workling automatically detects and uses Spawn, if installed. Spawn basically forks Rails every time you invoke a workling. To see what sort of characteristics this has, go into script/console, and run this:
>> fork { sleep 100 }
=> 1060 (the pid is returned)
You’ll see that this executes pretty much instantly. Run ‘top’ in another terminal window, and look for the new ruby process. This might be around 30 MB. This tells you that using spawn as a runner will result low latency, but will take at least 30MB for each request you make.
You cannot run your workers on a remote machine or cluster them with spawn. You also have no persistence: if you’ve fired of a lot of work and everything dies, there’s no way of picking up where you left off.
Using the Starling runner
If you want cross machine jobs with low latency and a low memory overhead, you might want to look into using the Starling Runner.
Installing Starling
As of 27. September 2008, the recommended Starling setup is as follows:
gem sources -a http://gems.github.com/
sudo gem install starling-starling
mkdir /var/spool/starling
The robot Co-Op Memcached Gem version 1.5.0 has several bugs, which have been fixed in the fiveruns-memcache-client gem. The starling-starling gem will install this as a dependency. Refer to the fiveruns README to see what the exact fixes are.
The Rubyforge Starling gem is also out of date. Currently, the most authorative Project is starling-starling on github (27. September 2008).
Workling will now automatically detect and use Starling, unless you have also installed Spawn. If you have Spawn installed, you need to tell Workling to use Starling by putting this in your environment.rb:
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
Starting up the required processes
Here’s what you need to get up and started in development mode. Look in config/workling.yml to see what the default ports are for other environments.
sudo starling -d -p 22122
script/workling_client start
Configuring workling.yml
Workling copies a file called workling.yml into your applications config directory. You can delete this file if you’re not planning to use Starling. The config file tells Workling on which port Starling is listening.
Notice that the default production port is 15151. This means you’ll need to start Starling with -p 15151 on production.
You can also use this config file to pass configuration options to the memcache client which workling uses to connect to starling. use the key ‘memcache_options’ for this.
You can also set sleep time for each Worker. See the key ‘listeners’ for this. Put in the modularized Class name as a key.
development:
listens_on: localhost:22122
sleep_time: 2
reset_time: 30
listeners:
Util:
sleep_time: 20
memcache_options:
namespace: myapp_development
production:
listens_on: localhost:22122, localhost:221223, localhost:221224
sleep_time: 2
reset_time: 30
Note that you can cluster Starling instances by passing a comma separated list of values to
Sleep time determines the wait time between polls against polls. A single poll will do one .get on every queue (there is a corresponding queue for each worker method).
If there is a memcache error, the Poller will hang for a bit to give it a chance to fire up again and reset the connection. The wait time can be set with the key reset_time.
Seeing what Starling is doing
Starling comes with it’s own script, starling_top. If you want statistics specific to workling, run:
script/starling_status.rb
A Quick Starling Primer
You might wonder what exactly starling does. Here’s a little snippet you can play with to illustrate how it works:
4 # Put messages onto a queue:
5 require 'memcache'
6 starling = MemCache.new('localhost:22122')
7 starling.set('my_queue', 1)
8
9 # Get messages from the queue:
10 require 'memcache'
11 starling = MemCache.new('localhost:22122')
12 loop { puts starling.get('my_queue') }
13
Using RudeQueue
RudeQueue is a Starling-like Queue that runs on top of your database and requires no extra processes. Use this if you don’t need very fast job processing and want to avoid managing the extra process starling requires.
Install the RudeQ plugin like this:
1 ./script/plugin install git://github.com/matthewrudy/rudeq.git
2 rake queue:setup
3 rake db:migrate
Configure Workling to use RudeQ. Add this to your environment:
Workling::Clients::MemcacheQueue.memcache_client_class = RudeQ::Client
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
Now start the Workling Client:
1 ./script/workling_client start
You’re good.
Using BackgroundJob
If you don’t want to bother with seperate processes, are not worried about latence or memory footprint, then you might want to use Bj to power workling.
Install the Bj plugin like this:
1 ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
2 ./script/bj setup
Workling will now automatically detect and use Bj, unless you have also installed Starling. If you have Starling installed, you need to tell Workling to use Bj by putting this in your environment.rb:
Workling::Remote.dispatcher = Workling::Remote::Runners::BackgroundjobRunner.new
Progress indicators and return stores
Your worklings can write back to a return store. This allows you to write progress indicators, or access results from your workling. As above, this is fairly slim. Again, you can swap in any return store implementation you like without changing your code. They all behave like memcached. For tests, there is a memory return store, for production use there is currently a starling return store. You can easily add a new return store (over the database for instance) by subclassing Workling::Return::Store::Base. Configure it like this in your test environment:
Workling::Return::Store.instance = Workling::Return::Store::MemoryReturnStore.new
Setting and getting values works as follows. Read the next paragraph to see where the job-id comes from.
Workling.return.set("job-id-1", "moo")
Workling.return.get("job-id-1") => "moo"
Here is an example worker that crawls an addressbook and puts results into a return store. Workling makes sure you have a :uid in your argument hash - set the value into the return store using this uid as a key:
require 'blackbook'
class NetworkWorker < Workling::Base
def search(options)
results = Blackbook.get(options[:key], options[:username], options[:password])
Workling.return.set(options[:uid], results)
end
end
call your workling as above:
@uid = NetworkWorker.asynch_search(:key => :gmail, :username => "foo@gmail.com", :password => "bar")
you can now use the @uid to query the return store:
results = Workling.return.get(@uid)
of course, you can use this for progress indicators. just put the progress into the return store.
enjoy!
Comments
There are 51 Comments for this post. Write comment →
Thankyou very much for workling.
Is it possible to have multiple worklings popping jobs from a single starling queue?
Multiple runners seems as simple as changeing multiples => flase flag in /script/workling_starling_client and running ‘ruby script/workling_starling_client start’ multiple times.
Hi,
thank you for your plugin workling, it’s really simple and very very useful. And it works fine for me.
Here’s my question :
I have several applications that run the same rails engine containing the same workling workers.
To allow web apps to communicate with their associated workling daemon, I was wondering myself if it’s better to run one starling daemon centralized all the messages for all the apps. but it means I have to modify queue name with a prefix to avoid identical names.
Or starting a starling daemon for each web apps ?
Any advice ?
many thanks. (sorry for my poor english)
thanks for the plugin, seems lke I could use this in multiple places in my app. However, running into some probs here :)
And I started the starling on this port and that went fine. However, when I do script/workling_starling_client start, it fails in creating the working clients. This is what the working.output shows:So I edited the starling.yml file and added a block for environment like this:
about:
listens_on: localhost:22122
=> Loading Rails…
/opt/local//lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:263:in `load_missing_constant’: uninitialized constant MysqlCompat::MysqlRes (NameError)
from /Users/nandayadav/Documents/aboutsearchclu/vendor/plugins/workling/script/../lib/workling/starling/poller.rb:39:in `join’
from /Users/nandayadav/Documents/aboutsearchclu/vendor/plugins/workling/script/../lib/workling/starling/poller.rb:39:in `listen’
from /Users/nandayadav/Documents/aboutsearchclu/vendor/plugins/workling/script/../lib/workling/starling/poller.rb:39:in `each’
from /Users/nandayadav/Documents/aboutsearchclu/vendor/plugins/workling/script/../lib/workling/starling/poller.rb:39:in `listen’
from /Users/nandayadav/Documents/aboutsearchclu/vendor/plugins/workling/script/listen.rb:19
from /opt/local//lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `load’
from /opt/local//lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `start_load’
from /opt/local//lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/application.rb:257:in `start’
from /opt/local//lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run’
from /opt/local//lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:139:in `run’
from /opt/local//lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call’
from /opt/local//lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions’
from /opt/local//lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/daemons.rb:138:in `run’
from script/workling_starling_client:17
Any ideas why I get that `load_missing_constant’: uninitialized constant MysqlCompat::MysqlRes (NameError). It works when I try in development environment though.
Is it possible at all to run workling without starling? For previous version it was enough to drop requires, new one (probably because of auto-detection) loads processor and fills log with message WORKLING: couldn’t find a memcache client – you need one for the starling runner.
But I do not want a starling runner, I just want spawn!
For those of you who had problems using spawn, go and grab the latest version off github or svn.
There was a problem with the autodection code, it’s been fixed!
/r
Thanks a lot for a quick fix! Going to try it right away.
For such fixes, is it possible to change a sub-minor version with reference to what was changed?
@EK oops, forgot to bump the version. i’ll pull in some other stuff and bump it tonight!
Just a heads up, due to how rails’ MySQL adapter handles this call ‘ActiveRecord::Base.connection.active?’, you’ll need to wrap the code that checks for a connection in ‘lib/workling/starling/poller.rb:clazz_listen’ in a mutex:
@@Mutex.synchronize do
unless ActiveRecord::Base.connection.active?
unless ActiveRecord::Base.connection.reconnect!
logger.fatal(“FAILED – Database not available”)
break
end
end
end
I noticed this while working with a multi-core machine that was spawning multiple workling threads. Some of my workling threads would hit serious issues at this block of code without the mutex.
Hi there
I’m using Starling and Workling for sending
notification updates to the data for
Free online Database
http://www.mytaskhelper.com
Starling/Workling works great, BUT
it may cause problem when you start it in production.
Please, use this in your environment.rb:
RAILS_ENV ||= “production”
It works!
Thanks,
Sincerely,
Igor
Hi,
Morning was nervous.
Online database MyTaskHelper
couldn’t start after restart.
I noticed the problem with environment in which starling starts.
It mean that > “stops” helping here.
And it is not pretty much to use this statement in developemnt – the same problems with starting Starling/Workling.
I added port for starling start command.
It may be helpful for someone:
where port 15151 from config/starling.yml:
production:
listens_on: 127.0.0.1:15151
development:
listens_on: 127.0.0.1:22122
test:
listens_on: 127.0.0.1:12345
Sincerely,
Igor
@Igor,
you need to set your environment the normal rails way before starting the workling starling client:
also, you need to take care with the starling ports in config/starling.yml. the default production port is 15151, so when you start up starling you have to do it like this:
that is, unless you want a different port. in that case, just change the config.
you can see if everything is as expected by typing `ruby script/starling_status.rb` in your RAILS_ROOT. remember to export the environment you’re dealing with.
if workling cannot connect to starling under the given port, it will terminate everything with a helpful exception explaining how to start starling with the expected port.
this is all documented in README.markdown, under the point “Using the Starling runner instead of Spawn”.
/r
When do you plan 1.0 version?
@Rechnung I wouldn’t say that there’s a need to wait for 1.0, since this is already seeing quite heavy production use on many sites.
I’d like to gemify this and see it also working with merb before going to 1.0. Anybody want to do this? ;)
Hi
Using workling + starling, how can I get the workling client to process several requests at a time instead of sequentially one after another ?
And more importantly, how can I cap that limit ? (Let’s say I want a max of 4 running requests at a time, and the others in the starling queue should wait till a “slot” becomes available) ?
@mina at the moment there’s only a single strategy for parallel worker execution for workers behind starling. i’m planning to abstract this out so that people can use different strategies.
so currently what happens is that a threadpool is created when you start up the client. a thread is added to the pool for each defined worker method. so if you call 2 different worker methods at the same time, these each run in their own thread. so working already parallelizes work, as long as you call more than one worker method in your code.
what you’re describing, if i understand, is that it should be possible to start a pool of 4 threads for a single worker method. this is currently not supported.
what exact scenario are you wanting to support?
@rany
Thanks for the reply.
My specific need is for reporting purposes. The client in the web UI asks to generate a report, and the request is to be processed asynchronously.
Another related need for a different app is upload video from the user + convert it to a different format.
The machine that actually generates the reports or converts the videos is pretty powerful, so there’s no need to sequentially process one after another when it can handle a few in parallel.
At the same time, I don’t want 100 requests that come in to start processing 100 in parallel, since it’ll choke the machine’s resources, so I need to cap the number of allowed parallel workers.
@rany Why is it not possible to just start multiple workling clients on the same machine? What prevents two separate processes from popping off the starling queue for the same worker method?
I’m currently working with starling/workling and have ran into an issue. I call an async method with an activerecord model as a parameter. In order for workling not to through an uncaught exception, I need to either require ‘medel_name’ in my worker or I need to reference the model in the constructor. Am I using workling correct? Is there some unwritten understanding that I should only place native ruby objects on the queue?
@Matt yeah you should treat worker arguments with care and avoid passing through large objects. the reason for this, is that the arguments need to be treated like they are going to be marshalled across to a remote server. this means that the objects have to be serialized or deserialized at either end. Keep in mind that things are going to get brittle if you pass large Objects around like this.
You’ll be safe if you treat Worker arguments like you would treat http session objects. Same rules apply – your data may need to be serialized/deserialized, so keep it slim. For AR, it’s best to save the object, then pass the id across, then find the object in the worker.
If you still need to pass the entire model into the workling for some reason, you need to insure that the model definition is available when ruby deserializes it. Again, same rules as normal rails sessions.
hope this helps,
/r
Hi,
I couldn’t run rake db:migrate, because workling/starling should be run.
I should be able to run rake db:migrate without workling/starling.
How to do it?
Also, I’m experiencing problems like:
after runing tests(with starling/workling) I’m running development mode and see errors in log, where workling tried to look for test objects.
It seams that both test and development mode used the same queue files in log/ directory.
How to fix it?
Hi,
any input on my problems?
I’m using workling/starling in production. And need to fix this issues asap.
Thanks,
Sincerely,
Igor
http://www.MyTaskHelper.com
I’m getting this error in the workling.output log. Any input would be appreciated.
Thanks!
=> Loading Rails…/Library/Ruby/Gems/1.8/gems/memcache-client-1.5.0/lib/memcache.rb:214:in `load’: undefined class/module Person (ArgumentError)
from /Users/lkenyan/revolution_sept/vendor/plugins/workling/script/../lib/workling/starling/poller.rb:42:in `join’
from /Users/lkenyan/revolution_sept/vendor/plugins/workling/script/../lib/workling/starling/poller.rb:42:in `listen’
from /Users/lkenyan/revolution_sept/vendor/plugins/workling/script/../lib/workling/starling/poller.rb:42:in `each’
from /Users/lkenyan/revolution_sept/vendor/plugins/workling/script/../lib/workling/starling/poller.rb:42:in `listen’
from /Users/lkenyan/revolution_sept/vendor/plugins/workling/script/listen.rb:19
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/daemons-1.0.9/lib/daemons/application.rb:159:in `load’
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/daemons-1.0.9/lib/daemons/application.rb:159:in `start_load’
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/daemons-1.0.9/lib/daemons/application.rb:236:in `start’
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/daemons-1.0.9/lib/daemons/controller.rb:69:in `run’
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/daemons-1.0.9/lib/daemons.rb:136:in `run’
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/daemons-1.0.9/lib/daemons/cmdline.rb:105:in `call’
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/daemons-1.0.9/lib/daemons/cmdline.rb:105:in `catch_exceptions’
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/daemons-1.0.9/lib/daemons.rb:135:in `run’
from ./script/workling_starling_client:17
@ricksrock the problem is, that you should avoid passing active record objects to worker methods. its the same as with rails sessions – it’s discouraged, and if you do so, you’ll get loading errors. rails allows you to specify models you want to store in your session. workling currently does not support this – just pass ids and then find the model in the worker.
also, you might like to install the latest workling version!
/r
Thank you. Rolling now.
I’m using svn to install workling, so I can’t get with git.
i try to store values in the worker:
Workling::Return::Store.set(“job-id-1”, “moo”)
and then reading it with an ajax poll from a controller action:
Workling::Return::Store.get(“job-id-1”)
it give nothing back.
my env.rb i have:
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
Workling::Return::Store.instance = Workling::Return::Store::MemoryReturnStore.new
(and no memchache configuration yet)
if i try to start ruby script/starling_status.rb i get this error:
=> Loading Rails…
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require’: no such file to load—script/../vendor/plugins/workling/lib/workling/remote/invokers/poller (MissingSourceFile)
…
you wrote “remember to export the environment you’re dealing with.”
what do you mean by that ??
thx for help and keep up the good work!
kalle
@kalle,
I fixed the script/starling_status.rb problem by making this small change to that script:
OLD:
require File.dirname(FILE) + ’/../vendor/plugins/workling/lib/workling/remote/invokers/poller’
NEW:
require File.dirname(FILE) + ’/../vendor/plugins/workling/lib/workling/remote/invokers/basic_poller’
I also added ”!/usr/bin/env ruby” to the top of the script so that it can be invoked as a standalone.
I’m having the exact same issues as Kalle.
I’m using Starling as the return store :
Workling::Return::Store.instance = Workling::Return::Store::StarlingReturnStore.new
And I’m able to set values in my worker trough wokling using:
Workling.return.set(options[:uid], ‘value’)
But getting values does not work like excpected. I can only get the values trough memcache directly:
queue = MemCache.new(‘localhost:22122’)
result = queue.get(uid)
And starling_status throws.rb me the same missing error.
Thanks for the help and the great plugin.
Should mongrel/thin be loading the workers?
I’m trying to use workling to replace several backgroundrb workers. One of these maintains a persistent connection to my jabber server using xmpp4r. The problem that I’m seeing is that, in addition to the create method on this worker being called when workling starts, it also seems to get called whenever I start my app server. I tried moving it from app/workers to lib/workling (it also seems to get called when backgroundrb starts) without success. Is there any workaround for this?
Thank you very much for workling. This is exactly what I was looking for.
I have problem with starting starling daemon.
It compains about undefined symbol _ZNSs4_Rep11_S_terminalE accessed by
rubyeventmachine.so. I’m using ruby 1.8.7.
$ sudo starling -d
/usr/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.2/lib/rubyeventmachine.so: /usr/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.2/lib/rubyeventmachine.so: undefined symbol: _ZNSs4_Rep11_S_terminalE – /usr/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.2/lib/rubyeventmachine.so (LoadError)
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require’
from /usr/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.2/lib/eventmachine.rb:70
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require’
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require’
from /usr/local/lib/ruby/gems/1.8/gems/starling-0.9.8/lib/starling/server.rb:4
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require’
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require’
from /usr/local/lib/ruby/gems/1.8/gems/starling-0.9.8/lib/starling/server_runner.rb:1
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require’
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require’
from /usr/local/lib/ruby/gems/1.8/gems/starling-0.9.8/bin/starling:3
from /usr/local/bin/starling:19:in `load’
from /usr/local/bin/starling:19
@kalle @carlos you can’t use the memory return store with a remote runner. that’s because the client you start with workling_client lives in a different interpreter instance to your application. you can’t access the memory of a different process in this way. use the starling memory store instead.
@todd hmmm. this currently isn’t supported, because workling doesn’t know if it’s running on the ‘server’ (a remote machine), or on the ‘client’. i might add this distinction in, then you could check if you were in server or client. i’ve added this to TODO.markdown.
@andy i haven’t seen a monit example. there’s some god examples around though. thinking i might add this to the repo for people to work off. would you mind sharing your monit script?
@aidin this looks pretty bizarre. i have no idea what’s going on, you’d be better off posting to a starling list!
@carlos I ran into the same issue as you did—not realizing at first the memory return store was just that (stuff stored in memory for a given ruby VM) and that there was no way to share that between a “client” and a “remote”. What I ended up doing: write a return store using RudeQ, which is pretty simple to use.
I’ve been running a workling/starling queue for media file conversions, and have run into a problem.
When deploying the app to production using capistrano, it seems the the workling gets “lost”.
It’s still running (or so the script/workling_client status tells me) but things aren’t don’t appear to be getting placed in the queue (or something).
Is there any advice/guidelines anyone can offer for making sure all is in order after a deploy?
Thanks.
I’m trying to get a client working with RabbitMQ, but am having a strange problem with the workers:
calling this:
class CowWorker “some value”)
always fails because, by the time options gets to the worker, it has been flattened out in to a String.
I’ve tried it using two different probjects (one set up solely for the purpose of making sure it wasn’t something in the original project).
Both are on OSX 10.5.6 with eventmachine 0.2.12 and tmm1-amqp 0.5.9. The test project is using rails 2.2.2 with no other plugins, the original project is using 2.1.0 with a raft of plugins. Both are being served with thin 1.0.0.
I posted to the google group a while back, but no one responded, so I thought I’d try here.
Further on options being passed as a string with a RabbitMQ worker:
I’ve just tried it with starling on my test environment, and I can confirm that the bug is NOT present: in short, CowWorker#moo receives a hash when
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
is set. If
Workling::Remote.invoker = Workling::Remote::Invokers::EventmachineSubscriber
Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new
Workling::Remote.dispatcher.client = Workling::Clients::AmqpClient.new
are set, then the options hash gets flattened in to a string.
I’ll have a look at the ClientRunner and see if I can see a reason.
When I’m logged into the console, the workling daemon starts successfully:
“script/workling_client start”
However, when I try to place the “script/workling_client start” in my crontab:
@reboot ruby RAILS_ROOT/script/workling_client start
The workling_client fails to start with the following errors:
#
#
#
#
#
#= 0)
Any help?
Hi,
I’m having some strange issues. I’m running ruby on windows xp. Seems like when I make changes to my classes in app/workers, Rails does not reload them. I stop & start the server (ruby script/server), close all open documents, etc, but the script never seems to load. It leads to:
undefined method `group_mailer’ for MailWorker:Class.
On linux, my VPS where the site is actually hosted, I call MailWorker.asynch_group_mailer , but on Windows, to test, I call MailWorker.group_mailer. group_mailer can’t be found. asynch_group mailer throws the exception “fork() function is unimplemented on this machine”. I guess a solution to either of these would be a good enough workaround.
Thanks!
Rajat
@Mina,
I got around this with:
e.g
(1..5).each{CharacterWorker.send(“async_moo_#{rand(4)+1}”, :id => 5)}
That’ll hack yourself to calling the same method at the same time.
@Mina,
Hmmm…. that solves multiple queue items per for fast jobs—but doesn’t help slow jobs.
I’m experimenting with Object.const_get() so I can have multiple worklings which actually do the same work.
Gotta be a better way…
the problem is I have 5 mongrels say and they’ve all submitted a job at the same time
- if I didn’t make a workling out of them, the jobs would still operate in parallel (but tying up the mongrels), with the workling, as I understand it they’ll happen in series -so if I’m sitting there checking progress the 5th guy is waiting for the first 4 to finish which doesn’t seem much better really… (plus how can I tell him that there are 4 people in front of him as well?).Anyone else having starling go bonkers after a day or so of running? It’s spool directory gets giant and the memory spikes to 1gig+ and just kills the server….
Thanks for this great information! I almost have everything working, but can’t get the starling_status.rb script to run:
/Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `gem_original_require’: no such file to load—script/../vendor/plugins/workling/lib/workling/remote/invokers/poller (MissingSourceFile)
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require’
from /Library/Ruby/Gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:153:in `require’
from /Library/Ruby/Gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:521:in `new_constants_in’
from /Library/Ruby/Gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:153:in `require’
from script/starling_status.rb:6
Someone else suggested changing the invokers/poller to invokers/basic_poller. I did this and got the following error. This runs and gives me some information, but does error out at the end with:
/Library/Ruby/Gems/1.8/gems/activesupport-2.2.2/lib/active_support/vendor/memcache-client-1.5.1/memcache.rb:697:in `raise_on_error_response!’: bad command line format (MemCache::MemCacheError)
from /Library/Ruby/Gems/1.8/gems/activesupport-2.2.2/lib/active_support/vendor/memcache-client-1.5.1/memcache.rb:414:in `flush_all’
from /Library/Ruby/Gems/1.8/gems/activesupport-2.2.2/lib/active_support/vendor/memcache-client-1.5.1/memcache.rb:408:in `each’
from /Library/Ruby/Gems/1.8/gems/activesupport-2.2.2/lib/active_support/vendor/memcache-client-1.5.1/memcache.rb:408:in `flush_all’
from /Users/tom/Sites/buzzsprout/vendor/plugins/workling/lib/workling/clients/memcache_queue_client.rb:44:in `close’
from script/starling_status.rb:36
@Todd,
I am running into the same issue with Workling + RabbitMQ. My options always get flattened into a string. Have you found a workaround for this yet? I might just try Starling otherwise…
I’m getting the same “memcache.rb:697:in `raise_on_error_response!’: bad command line format (MemCache::MemCacheError)” from starling_status.rb. Anyone else having this problem or come up with a solution?
Also on a different note, I’m wondering if anyone has come up with good solutions for monitoring the size of a starling queue. At some point traffic on my site caused the queue to start growing faster than the workers were processing them. Seems like things can get out of hand, especially if nobody is alerted. Has anyone come up with a good way of monitoring this? Monit? A collectd plugin? I’m thinking about using the former with my RightScale deployment.
Lastly, is there any need to clear the spool files once in a while… do they just grow infinitely big??
Thank you so much for any advice!
I’m having similar error with RudeQ::Client while executing starling_status.rb
vendor/plugins/workling/lib/workling/clients/memcache_queue_client.rb:44:in `close’: undefined method `flush_all’ for # (NoMethodError) from script/starling_status.rb:36
I cannot get the starling_status script to run?
script/starling_status.rb: line 1: require: command not found
script/starling_status.rb: line 3: puts: command not found
script/starling_status.rb: line 5: syntax error near unexpected token `(‘
script/starling_status.rb: line 5: `require File.dirname(FILE) + ’/../config/environment’‘
Is anyone else seeing this issue?
Hi!
First things first: congratulations on great plugin which makes working with Starling easy :)
But I have a problem with workling: after running for some time (development mode, frequent deploys/restarts with Capistrano) I get a lot and lot of “schedule”, “schedule_monitor” and “workling” processes, each taking some memory and rendering the development server unusable at some moment (full RAM usage, more than half swap usage).
Where does it come from and how can I stop these processes from spawning OR let them die gracefully?
The rainbow at the bottom of your blog is brutal. If I search for something on the page, it’s invariably beneath the rainbow and I have to scroll to see it.
@Tom,
I had the same 2 errors with starling_status.rb
For the this error:
bad command line format (MemCache::MemCacheError)
memcached starts on port 11211, but the default in workling.yml is 22122. Doing a memcached -d -p 22122 fix it for me.
The other error:
script/starling_status.rb: line 5: `require File.dirname(FILE) + ’/../config/environment’‘
To fix this, you need to add ruby to the call:
ruby script/starling_status.rb
From a comment above, you can instead add this to the top of your starling_status.rb:
!/usr/bin/env ruby
Okay scratch that part about memcached’s port. It is starling that defaults to 22122, so setting memcached to port 22122 is quite wrong (and gave me an infinite loop).
Maybe just killing and starting memcached was enough to get things working again.
How to test background jobs using rspec?
Hi Rany
I have just downloaded a clean copy as per above. My worker method however seems to be firing twice. Its very easy to replicate and i have achieved it in a clean rails project with just print statements to confirm the behavior. Is there a known bug? i have tried it with both starling and rudemq -> same error.
Write a comment
Required in bold.