boomloop is no more! Vielen Dank an alle, die bei uns mitgemacht haben: it was fun while it lasted!

last.fm uses old school irc to connect developers and business people inside the company. the last.fm founder wrote up an article about how they do this with the irccat bot.

…Everyone that works at Last.fm is typically connected to our IRC server. We have different channels per team, as well as a company-wide channel, and a few channels dedicated to automated monitoring. Sometimes it makes much more sense to discuss / ask questions on IRC instead of email, and it’s useful to be able to raise people who are not in the office. That said, the main reason I’m writing this post is to mention the dev-support bot we use: irccat.

i’ve written a ruby eventmachine clone of the irc bot they wrote. it’s called mini, and i tried to keep it as tiny as possible, which was fun :-). using mini…

…you can easily send events to irc from shell scripts:

echo "Ilovethistuffs yes .. i .. do. " | nc localhost 12345 
vmstat | nc localhost 12345 
echo "#musicteam,#legal,@alice New album uploaded: ..." | nc somemachine 12345 
tail -f /var/log/important.log | nc somemachine 12345

you can also send commands from inside the channels using `?command` syntax. this will invoke a script called minictl on your $PATH.

you can also post to the bot over a url. there’s simple mechanism for adding plugins, as well.

it’s a neat little toy, check out the source on github.

Installation

sudo gem install purzelrakete-mini --source=http://gems.github.com

create a directory for mini and run the following

minigen create 

this creates two files in your mini dir. edit both as you see fit, then add your mini dir to your $PATH inside of your shell’s rc file.

I discovered balsamiq mockups a few weeks ago. It’s a prototyping too with which you can sketch out user interfaces. it’s very pleasant to use, and the results are excellent input for product development.

Balsamiq is good for keeping the discussion at a sketch level. The result is clearly not a finished design, which promotes more meaningful discussions about changes.

check it out! it’s a one man effort, which i find very impressive.

It’s possible to write Linux desktop apps using ruby! All you need is a distribution running KDE 4.2. I have Kubuntu running on my Macbook.

First off, get the Ruby bindings installed:

apt-get install ruby-kde4

Next up, create a project directory for your Plasmoid. You can follow along with ‘Rakete’, which is my attempt at a Twitter client. The only thing Rakete does so far is to load and display an SVG.

you can call your root directory anything you like. In my case, it’s ‘application’ instread of ‘rakete’.

The `contents`directory holds the main files in a number of subdirectories. The most important of these is `code`, which, you guessed it, holds your ruby entry point, `main.rb`. `images` holds the SVG we want to render. You can create SVGs with a number of tools such as Adobe Illustrator or Inkscape (open source).

require 'plasma_applet'

module Rakete
  class Main < PlasmaScripting::Applet
end

def initialize(parent, args = nil)
    super
  end
end

def init
  @svg = Plasma::Svg.new(self)
  @svg.imagePath = 'widgets/rakete'
end

def paintInterface(painter, option, contentsRect)
  @svg.resize(size())
  @svg.paint(painter, 0, 0)
end

`main.rb`should contain a class called `Main`, inside of a module named after your application. There are a number of callbacks you have to implement.

First off, you have ìnit`, which is called when the applet is first loaded. In here, we construct Plasma::Svg, which loads up rakete.svg.

SVGs are read from the svg.imagePath. This is relative to the $KDE_ROOT/apps/desktoptheme/default. KDE_ROOT varies from system to system. In the case of Kubuntu, it is /usr/share/kde4. So specifying an svg image path of

@svg.imagePath = 'widgets/rakete'

will mean that the svg is searched for under

/usr/share/kde4/apps/desktoptheme/default/widgets/rakete.svg

This lies outside of your Plasmoid directory structure, so the SVGs you package have to be staged into the widgets directory. Currently I don’t see any way of automating this staging process, so you’ll have to copy the svg from images/ over to the widgets directory. I’ll post more info as soon as I get it.

UPDATE 25.12.08: you can reference images inside of your plasmoid directory structure in your code. it turns out that doing so is very easy. applets have a `package` method, which returns an object representing your plasmoid package. the package object can resolve the canonical filename of your svg like this:

@svg.imagePath = package.file_path("images", "rakete.svg")

now you don’t need to worry about staging anything.

Next, create the metadata.desktop file:

[Desktop Entry]
Name=Rakete
Comment=an attempt at a better twitter plasmoid using ruby. 
Icon=
Type=Service
ServiceTypes=Plasma/Applet
X-Plasma-API=ruby-script

X-KDE-PluginInfo-Author=Rany Keddo
X-KDE-PluginInfo-Email=purzelrakete@gmail.com    
X-KDE-PluginInfo-Name=rakete
X-KDE-PluginInfo-Version=pre0.1    
X-KDE-PluginInfo-Website=http://playtype.net
X-KDE-PluginInfo-Category=
X-KDE-PluginInfo-Depends=
X-KDE-PluginInfo-License=MIT
X-KDE-PluginInfo-EnabledByDefault=true

Replace the pertinent field values.

Finally, you’ll want to run this thing. Run the following above the plasma root directory:

plasmapkg --upgrade application

This will install the Plasmoid on your system, or update it is already installed. Note that if you plasmoid root directory has a different name to “application”, you’ll need to cahnge the packaging command accordingly.

Finally, open the cashew in the top righthand corner of your desktop and select “Add Widget”. You’ll find your widget in the list.

As it stands, Plasma will sometimes crash when running this Applet. I’ll pass this information on to Richard Dale, the maintainer of the Ruby bindings. Hopefully he will be able to locate problems in the runtime, which is still in Beta1.

Richard was also super helpful in getting this example to run. Thank you Richard :-)

I’ll write more posts as I make progress with the Twitter client. You can find Rakete on Github.

If you want to send mail asynchronously in your rails app, but don’t want to bother with using a queue server like Starling, then have a go at this…

First, run this:

./script/plugin install git://github.com/purzelrakete/workling.git
./script/plugin install git://github.com/langalex/workling_mailer.git
./script/plugin install git://github.com/matthewrudy/rudeq.git
rake queue:setup
rake db:migrate

now add this to your environment:

Workling::Clients::MemcacheQueueClient.memcache_client_class = RudeQ::Client
Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new

Finally, add this into the body of your mailer classes:

include AsynchMail

now start the workling client:

./script/workling_client start

done!

I did some refactoring this week, and the result is a much improved workling. Adding new work brokers is a snap now. I’ll show you how I added RabbitMQ, so that you can go about adding your own brokers. So without further ado…

RabbitMQ, a quick introduction

A lot of Ruby people have been talking about using RabbitMQ as their Queue of choice. Soundcloud.com are using it, as is new bamboo founder Johnathan Conway, who is using it at his video startup http://www.vzaar.com/. He says:

RabbitMQ – Now this is the matrons knockers when it comes to kick ass, ultra fast and scalable messaging. It simply rocks, with performance off the hook. It’s written in Erlang and supports the AMPQ protocol.

Follow the instructions here to get this beauty installed.

Adding Ampq to Workling

There are two new base classes you can extend to add new brokers. I’ll describe how this is done, but the code i show is already a part of workling. Skip to ‘activating amqp in your application’ to see how this is activated.

Clients

Clients help workling to connect to job brokers. To add an AmqpClient, we need to extend from Workling::Client::Base and implement a couple of methods.

require 'workling/clients/base'
require 'mq'

#
#  An Ampq client
#
module Workling
  module Clients
    class AmqpClient < Workling::Clients::Base

      # starts the client. 
      def connect
        @amq = MQ.new
      end

      # stops the client.
      def close
        @amq.close
      end

      # request work
      def request(queue, value)
        @amq.queue(queue).publish(value)
      end

      # retrieve work
      def retrieve(queue)
        @amq.queue(queue)
      end

      # subscribe to a queue
      def subscribe(queue)
        @amq.queue(queue).subscribe do |value|
          yield value
        end
      end

    end
  end
end

Were’s using the eventmachine amqp client for this, you can find it up on github. connect and close do exactly what it says on the tin: connecting to rabbitmq and closing the connection.

request and retrieve are responsible for placing work on rabbitmq. The methods are passed the correct queue, and a value that contains the worker method arguments. If you need control over the queue names, look at the RDoc for Workling::Routing::Base. In our case, there’s no special requirement here.

Finally, we implement a subscribe method. Use this if your broker supports callbacks, as is the case with amqp. This method expects to a block, which we pass into the amqp subscribe method here. The block will be called when a message is available on the queue, and the result is yielded into the block.

Having subscription callbacks is very nice, because this way, we don’t need to keep calling get on the queue to see if something new is waiting.

So now we’re done! That’s all you need to add RabbitMQ to workling. Configure it in your application as descibed below.

Invokers

There’s still potential to improve things though. Workling 0.4.0 introduces the idea of invokers. Invokers grab work off a job broker, using a client (see above). They subclass Workling::Remote::Invokers::Base. Read the RDoc for a description of the methods.

Workling comes with a couple of standard invokers, like the BasicPoller. This invoker simply keeps hitting the broker every n seconds, checking for new work and executing it immediately. The ThreadedInvoker does the same, but spawns a Thread for every Worker class the project defines.

So Amqp: it would be nice if we had an invoker that makes use of the subscription callbacks. Easily done, lets have a look:

require 'eventmachine'
require 'workling/remote/invokers/base'

#
#  Subscribes the workers to the correct queues. 
# 
module Workling
  module Remote
    module Invokers
      class EventmachineSubscriber < Workling::Remote::Invokers::Base

        def initialize(routing, client_class)
          super
        end

        #
        #  Starts EM loop and sets up subscription callbacks for workers. 
        #
        def listen
          EM.run do
            connect do
              routes.each do |queue|
                @client.subscribe(queue) do |args|
                  run(queue, args)
                end
              end
            end
          end
        end

        def stop
          EM.stop if EM.reactor_running?
        end
      end
    end
  end
end

Invokers have to implement two methods, listen and stop. Listen starts the main listener loop, which is responsible for starting work when it becomes available.

In our case, we need to start an EM loop around listen. This is because the Ruby AMQP library needs to run inside of an eventmachine reactor loop.

Next, inside of listen, we need to iterate through all defined routes. There is a route for each worker method you defined in your application. The routes double as queue names. For this, you can use the helper method routes. Now we attach a callback to each queue. We can use the helper method run, which executes the worker method associated with the queue, passing along any supplied arguments.

That’s it! We now have a more effective Invoker, which we can activate in our application like this…

Configuring your application to use AMQP (with RabbitMQ)

Again, follow the instructions here to get RabbitMQ running on your OSX development machine.

Workling::Remote.invoker = Workling::Remote::Invokers::EventmachineSubscriber
Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new
Workling::Remote.dispatcher.client = Workling::Clients::AmqpClient.new

enjoy!

here’s a little thing that made me happy today. i spent some time automating pdf printing. i have a directory structure like so: receipts/2008/10. when needed, everything is packed into a single pdf and sent to the printer down the road. to do this, every file has to be converted to pdf and combined to a single file. input formats are rtf, txt, pdf or multipage tiff.

so today wrote a dirty ruby script for this. mashing little unix tools into your ruby code is good. dirty, trash-eating shell opening code, i salute you! it’s taken me a while to come around to this, being the ex-java i am. 2 years away from jcp.org seem to have done me good ;)

Low effort pdf conversion using the shell

sam, suggested i should look into ruby-cocoa, since preview can print many formats to pdf. but a bit of googling lead me to CUPS-PDF, with which you can set up a printer which dumps pdfs into a specified directory. Follow along at http://www.codepoetry.net/projects/cups-pdf-for-mosx.

So lets see if we have everything we need to get started. First off, lets check if the CUPS printer has been installed:

    raise Exception.new("install cups.") unless `lpstat -a`.grep Regexp.new(CUPS_PRINTER)

lpstat -a gives you a list of all printing devices configured on your machine. We’re going to search this for CUPS_PRINTER using grep. We’ll actually use this printer to convert input files, like rtfs, into pdf files on disk.

So now we want to Grab the files and pass them to the printer. This should all be automatic. A quick search reveals lp:

    `lp -d #{ CUPS_PRINTER } #{ file }`

Nice, this now dumps the converted file into the CUPS root directory, which by default is cups-pdf on your desktop.

Multipage Tiffs

Unfortunately lp only prints the first page of multipage TIFFS to pdf. So what we need to to do is to extract the pages from the tiff. Lets see if there’s anything available:

    man -k tiff

This shows us tiffutil, which can be utilized to ‘manipulate tiff images’. Lets see how this works:

    man tiffutil

Bingo, there’s an -extract option which pulls out the specified page. So now we just need to know how many pages are in the tiff. There’s an -info option that gives us the information we need. Each image is described with an entry that starts with “Directory at”. Lets count the number of times this appears:

     `tiffutil -info #{ file } | grep "Directory at" | wc -l`.strip.to_i

Here we’re piping the output of tiffutil to grep, which results in a line per found image. We pipe this through wc -l to count the number of images.

Combining the pdf

Thanks again to sam for pointing me to pdfcombine. just specify all files you want to combine, the outfile with -out, and you’re done. So let’s see if it’s installed…

    raise Exception.new("install pdfcombine command line tool. ") unless `which pdfcombine`

which searches $PATH for the specified executable. The full path is returned. To run it:

     `pdfcombine #{ exports.join(" ") } -o #{ packfile }`

And you’re done!

The script

    require 'fileutils'
    require 'logger'

    #
    #  Combines all txt, rtf, tiff multipage, pdf files in a named subdirectory. 
    #  Results in a single pdf called <subdir>-pack.pdf 
    #
    class ReceiptsPrinter

      # requires you to be in this subdirectory for safety reasons (FileUtils.rm_r is used.)
      RECEIPTS_ROOT = "/Users/me/Documents/2008"

      # cups prints to this directory
      CUPS_ROOT = "/Users/me/Desktop/cups-pdf"

      # cups printer name
      CUPS_PRINTER = "CUPS_PDF"

      # 
      #  pass in the subdirectory name which includes the files
      #
      def initialize(subdir)
        @subdir = subdir
        @files = Dir.glob("#{ subdir }/*.*")
        @export_dir = "#{ subdir }/export/"
        @logger = Logger.new(STDOUT)

        # please meet the conditions.   
        raise Exception.new("specify the subdir to process.") unless @subdir
        raise Exception.new("cd to #{ RECEIPTS_ROOT } before starting.") if `pwd`.chomp != RECEIPTS_ROOT
        raise Exception.new("install pdfcombine command line tool. ") unless `which pdfcombine`
        raise Exception.new("install cups to enable conversions: http://www.codepoetry.net/projects/cups-pdf-for-mosx ") unless `lpstat -a`.grep Regexp.new(CUPS_PRINTER)

        # sets up the export directory
        FileUtils.rm_r @export_dir rescue nil
        FileUtils.mkdir_p @export_dir
      end

      #
      #  iterates through all files in subdirectory, converting them to pdf if necessary, then combines them
      #  into a single pdf. 
      #
      def print
        @files.each { |file| convert(file) }
        packfile = combine

        @logger.info "Done! Packed it up into #{ packfile }."
      end

      private
        def convert(file)
          case type(file)
          when "pdf" : stage(file)
          when "tiff" 
            extract_tiff_pages(file).each do |file| 
              stage cups_converter(file)
            end
          else 
            stage cups_converter(file)
          end
        end

        def stage(file)
          @logger.info "staging file #{ file }"
          dest = @export_dir + file.split('/').last
          `cp #{ file } #{ dest }`
        end

        def combine
          exports = Dir.glob("#{ @export_dir }*.pdf")
          packfile = "#{ @export_dir }/#{ @subdir }-packed.pdf"
          command = "pdfcombine #{ exports.join(" ") } -o #{ packfile }"

          # pack it up!
          `#{ command }`

          packfile
        end

        def cups_converter(file)
          command = "lp -d #{ CUPS_PRINTER } #{ file }"
          regexp = Regexp.new("Auftrags-ID ist #{ CUPS_PRINTER }-(\d*) .*")
          string, id =  *`#{ command }`.match(regexp)
          raise Exception.new("cups printing failed for #{ file }: #{ string }") unless id

          # give the thing a moment to generate
          sleep 1 while !(pdf = Dir.glob("#{ CUPS_ROOT }/job_#{ id }-*.pdf").first)

          pdf
        end

        def extract_tiff_pages(file)
          (1..tiff_page_number(file)).map do |page|
            extract_tiff_page(file, page - 1)
          end
        end

        def extract_tiff_page(file, page)
          filename = "#{ @export_dir }#{ page }-#{ file.split('/').last }"

          @logger.info "tiff: #{ filename }"
          `tiffutil -extract #{ page } #{ file } -out #{ filename }`

          filename
        end

        def tiff_page_number(file)
          `tiffutil -info #{ file } | grep "Directory at" | wc -l`.strip.to_i
        end

        def type(file); file.split(".").last; end
    end

    printer = ReceiptsPrinter.new(month = ARGV.first)
    puts ">>>>> Please back the files in #{ month } before doing this. Starting; press CTL-C to abort. \n\n"
    printer.print

First off, you’ll need to install RabbitMQ. Instructions for OSX here. Once you’ve done that, install the pure ruby amqp library:

    gem sources -a http://gems.github.com
    gem install tmm1-amqp

You’re good to go. Now open up two IRB sessions. Paste the following code into the first session:

    require 'mq'

    EM.run {
      amq = MQ.new
      EM.add_periodic_timer(1) { amq.queue("noises").publish("moo") }
    }

Your publishing code has to run inside of an Event Machine loop. You can start it using EM.run. If you’re running inside of an Evented Container such as Thin or Evented Mongrel, you can skip this. The meat of it is just amq.queue("noises").publish("moo")that.

Now open a second irb session in another terminal tab, and paste this in:

    require 'mq'

    EM.run {
      amq = MQ.new
      amq.queue("noises").subscribe { |noise|
        puts noise
      }
    }

You’ll get the moos off the Queue “noises”. No polling required, which is very nice, you simply register a callback with amqp, and it’ll be called with the message when one becomes available.

Niceness!

RabbitMQ is a reliable, high performance queue Server written in Erlang.

Lets fiddle.

Before you start

Before you can start, you have to have installed Macports. Just in case, here’s how it’s done:

• Register at the Apple Developers Connection. Go to downloads/developer tools, and download the latest version of the XCode Developer tools. Grab a Flask of coffee.
• Download and run the latest macports installer http://svn.macports.org/repository/macports/downloads/. Grab another Flask.

Installing Erlang

If you’re on the 10.5.3 update, you have to edit the erlang portsfile to avoid a bus error bug:

    sudo vi /opt/local/var/macports/sources/rsync.macports.org/release/ports/lang/erlang/Portfile

Delete this line from configure.args attribute:

    --enable-hipe \

Now you can use port to install erlang:

    port install erlang

this is going to take a while.

Installing RabbitMQ

Grab and unpack the lastest generic Unix version of RabbitMQ. The current version as of writing is 1.4.0.

    mkdir /tmp/rabbit-mq && cd /tmp/rabbit-mq
    wget http://www.rabbitmq.com/releases/rabbitmq-server/v1.4.0/rabbitmq-server-generic-unix-1.4.0.tar.gz
    tar xvfz rabbitmq-server-generic-unix-1.4.0.tar.gz

Now move this stuff into erlang’s magical mystery directory:

    sudo mv rabbitmq_server-1.4.0 /opt/local/lib/erlang/lib

You’re good to go! Start up like this:

    sudo /opt/local/lib/erlang/lib/rabbitmq_server-1.4.0/sbin/rabbitmq-server

Change permissions if you don’t want to run this as root.

UPDATE: I’ve been having troubes starting the server up, since the tables in the mnesia database backing rabbitmq are locked. I don’t know why this is the case. You can get this running again brute force styleee by deleting the database:

    sudo rm -rf /var/lib/rabbitmq/mnesia

Workling 0.3 is up on GitHub and svn.playtype.net. This release improves error logging and includes an updated README - finally ;). Here it is….

Workling

Workling gives your Rails App a simple API that you can use to make code run in the background, outside of the your request.

You can configure how the background code will be run. Currently, workling supports Starling, BackgroundJob and Spawn Runners. Workling is a bit like Actve* for background work: you can write your code once, then swap in any of the supported background Runners later. This keeps things flexible.

Installing Workling

The easiest way of getting started with workling is like this:

script/plugin install git://github.com/purzelrakete/workling.git
script/plugin install git://github.com/tra/spawn.git

If you’re on an older Rails version, there’s also a subversion mirror wor workling (I’ll do my best to keep it synched) at:

script/plugin install http://svn.playtype.net/plugins/workling/

Writing and calling Workers

This is pretty easy. Just put cow_worker.rb into into app/workers, and subclass Workling::Base:

# handle asynchronous mooing.
class CowWorker < Workling::Base 
  def moo(options)
    cow = Cow.find(options[:id])
    logger.info("about to moo.")
    cow.moo
  end
end

Make sure you have exactly one hash parameter in your methods, workling passes the job :uid into here. Btw, in case you want to follow along with the Mooing, grab ‘cows-not-kittens’ off github, it’s an example workling project. Look at the branches, there’s one for each Runner.

Next, you’ll want to call your workling in a controller. Your controller might looks like this:

class CowsController < ApplicationController

  # milking has the side effect of causing
  # the cow to moo. we don't want to
  # wait for this while milking, though,
  # it would be a terrible waste ouf our time.
  def milk
    @cow = Cow.find(params[:id])
    CowWorker.asynch_moo(:id => @cow.id)
  end
end

Notice the asynch_moo call to CowWorker. This will call the moo method on the CowWorker in the background, passing any parameters you like on. In fact, workling will call whatever comes after asynch_ as a method on the worker instance.

Worker Lifecycle

All worker classes must inherit from this class, and be saved in app/workers. The Worker is loaded once, at which point the instance method create is called.

Calling async_my_method on the worker class will trigger background work. This means that the loaded Worker instance will receive a call to the method my_method(:uid => "thisjobsuid2348732947923").

Exception handling in Workers

If an exception is raised in your Worker, it will not be propagated to the calling code by workling. This is because the code is called asynchronously, meaning that exceptions may be raised after the calling code has already returned. If you need your calling code to handle exceptional situations, you have to pass the error into the return store.

Workling does log all exceptions that propagate out of the worker methods.

Logging with Workling

RAILS_DEFAULT_LOGGER is available in all workers. Workers also have a logger method which returns the default logger, so you can log like this:

logger.info("about to moo.")

What should I know about the Spawn Runner?

Workling automatically detects and uses Spawn, if installed. Spawn basically forks Rails every time you invoke a workling. To see what sort of characteristics this has, go into script/console, and run this:

>> fork { sleep 100 } 
=> 1060 (the pid is returned)

You’ll see that this executes pretty much instantly. Run ‘top’ in another terminal window, and look for the new ruby process. This might be around 30 MB. This tells you that using spawn as a runner will result low latency, but will take at least 30MB for each request you make.

You cannot run your workers on a remote machine or cluster them with spawn. You also have no persistence: if you’ve fired of a lot of work and everything dies, there’s no way of picking up where you left off.

Using the Starling runner

If you want cross machine jobs with low latency and a low memory overhead, you might want to look into using the Starling Runner.

Installing Starling

As of 27. September 2008, the recommended Starling setup is as follows:

gem sources -a http://gems.github.com/ 
sudo gem install starling-starling 
mkdir /var/spool/starling

The robot Co-Op Memcached Gem version 1.5.0 has several bugs, which have been fixed in the fiveruns-memcache-client gem. The starling-starling gem will install this as a dependency. Refer to the fiveruns README to see what the exact fixes are.

The Rubyforge Starling gem is also out of date. Currently, the most authorative Project is starling-starling on github (27. September 2008).

Workling will now automatically detect and use Starling, unless you have also installed Spawn. If you have Spawn installed, you need to tell Workling to use Starling by putting this in your environment.rb:

Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new

Starting up the required processes

Here’s what you need to get up and started in development mode. Look in config/workling.yml to see what the default ports are for other environments.

sudo starling -d -p 22122
script/workling_client start

Configuring workling.yml

Workling copies a file called workling.yml into your applications config directory. You can delete this file if you’re not planning to use Starling. The config file tells Workling on which port Starling is listening.

Notice that the default production port is 15151. This means you’ll need to start Starling with -p 15151 on production.

You can also use this config file to pass configuration options to the memcache client which workling uses to connect to starling. use the key ‘memcache_options’ for this.

You can also set sleep time for each Worker. See the key ‘listeners’ for this. Put in the modularized Class name as a key.

development:
  listens_on: localhost:22122
  sleep_time: 2
  reset_time: 30
  listeners:
    Util:
      sleep_time: 20
  memcache_options:
    namespace: myapp_development

production:
  listens_on: localhost:22122, localhost:221223, localhost:221224
  sleep_time: 2
  reset_time: 30

Note that you can cluster Starling instances by passing a comma separated list of values to

Sleep time determines the wait time between polls against polls. A single poll will do one .get on every queue (there is a corresponding queue for each worker method).

If there is a memcache error, the Poller will hang for a bit to give it a chance to fire up again and reset the connection. The wait time can be set with the key reset_time.

Seeing what Starling is doing

Starling comes with it’s own script, starling_top. If you want statistics specific to workling, run:

script/starling_status.rb

A Quick Starling Primer

You might wonder what exactly starling does. Here’s a little snippet you can play with to illustrate how it works:

 4 # Put messages onto a queue:
 5 require 'memcache'
 6 starling = MemCache.new('localhost:22122')
 7 starling.set('my_queue', 1)
 8 
 9 # Get messages from the queue:
10 require 'memcache'
11 starling = MemCache.new('localhost:22122')
12 loop { puts starling.get('my_queue') }
13

Using RudeQueue

RudeQueue is a Starling-like Queue that runs on top of your database and requires no extra processes. Use this if you don’t need very fast job processing and want to avoid managing the extra process starling requires.

Install the RudeQ plugin like this:

1 ./script/plugin install git://github.com/matthewrudy/rudeq.git
2 rake queue:setup
3 rake db:migrate

Configure Workling to use RudeQ. Add this to your environment:

Workling::Clients::MemcacheQueue.memcache_client_class = RudeQ::Client
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new

Now start the Workling Client:

1 ./script/workling_client start

You’re good.

Using BackgroundJob

If you don’t want to bother with seperate processes, are not worried about latence or memory footprint, then you might want to use Bj to power workling.

Install the Bj plugin like this:

1 ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
2 ./script/bj setup

Workling will now automatically detect and use Bj, unless you have also installed Starling. If you have Starling installed, you need to tell Workling to use Bj by putting this in your environment.rb:

Workling::Remote.dispatcher = Workling::Remote::Runners::BackgroundjobRunner.new

Progress indicators and return stores

Your worklings can write back to a return store. This allows you to write progress indicators, or access results from your workling. As above, this is fairly slim. Again, you can swap in any return store implementation you like without changing your code. They all behave like memcached. For tests, there is a memory return store, for production use there is currently a starling return store. You can easily add a new return store (over the database for instance) by subclassing Workling::Return::Store::Base. Configure it like this in your test environment:

Workling::Return::Store.instance = Workling::Return::Store::MemoryReturnStore.new

Setting and getting values works as follows. Read the next paragraph to see where the job-id comes from.

Workling.return.set("job-id-1", "moo")
Workling.return.get("job-id-1")           => "moo"

Here is an example worker that crawls an addressbook and puts results into a return store. Workling makes sure you have a :uid in your argument hash - set the value into the return store using this uid as a key:

require 'blackbook'
class NetworkWorker < Workling::Base
  def search(options)
    results = Blackbook.get(options[:key], options[:username], options[:password])
    Workling.return.set(options[:uid], results)
  end
end

call your workling as above:

@uid = NetworkWorker.asynch_search(:key => :gmail, :username => "foo@gmail.com", :password => "bar")

you can now use the @uid to query the return store:

results = Workling.return.get(@uid)

of course, you can use this for progress indicators. just put the progress into the return store.

enjoy!

Just got back from 9 great days in Berlin: Railsconf 2008 is over.

I gave a talk about the Workling plugin, which solves the issue of deciding on a rails background work solution by providing a single interface over various solutions.

To illustrate the talk with a nice example, I created a project on github. Besides being a shameless attempt at winning over the audience by making their laptops moo (try running CowSubsystem.moo once you’ve cloned it - osx required), it shows how the same codebase can be made to send mooing to the background using 1. spawn, 2. bj, and 3. starling. Oh yeah, there’s a branch for each solution up on github.

If you were at Railsconf and liked my talk, then please do rate it at the conference site If you didn’t like my talk then… hey look, it’s the simpsons voiceover artists on actors studio! http://www.youtube.com/watch?v=lMOQnJ-SLHI

You can find the slides up on the Railsconf site, or on Slideshare:

first off, thanks to thomas for creating another fantastic conference this year. reboot is really special for me, and you pretty much kick every other conference’s ass, which is quite an achievement considering the sheer number of conferences that have popped up in the last few years.

in the spirit of sharing your shit, i’d like to throw up some thoughts i had on how we could make reboot even better next year.

better how? well, reboot is about sharing your thoughts in a public forum, thereby helping others to get a fresh perspective on their own work, right? so i think better means more sharing, at more levels.

tools

bring back the past! my first reboot 4 years ago completely blew my mind. this had a lot to do with the kinds of tools people were using at the conference. this year, a lot of those tools either just weren’t there, or i couldn’t discover them.

wiki

the reboot wiki used to kick ass. who took my wiki away? this year, i didn’t really feel any sense of ownership with the website. the social network was nice enough, but kind of off the mark. the low entry barriers of the wiki had fostered a feeling of common ownership of the website, and so people used it a lot more for ad-hoc communication. also, the wiki was simply a better hammer - i remember pages opening pre-conference, where people listed books they were offering to swap. this year, i didn’t feel like i could bend the website to do whatever i wanted.

irc

was there an irc channel this year? i couldn’t find it. back in the day, there was a huge stream of commentary available for each talk, realtime. links and supplementary information would pop up around what was being said, and if i could hear something properly, i could just ask in the channel for an instant answer. irc was the shit! you could sit outside and just ask into ‘reboot-space’: what’s room 2 like right now?

i realize that this was pre-twitter. but twitter was only valuable to me over summize.com, and i couldnt get this to aggregate into anything near the sort of speed of irc. to my mind, twitter is still to (micro) bloggy for something like “is room 2 any good” - i certainly did not see it used this way.

there are other backchannel tools around, like onlinr.com, maybe. but frankly, is there anything wrong with irc? open question. the use of twitter i say seemed like a step backwards in terms of the sharing going on, anyway.

subethaedit

was this happening? if so, i didn’t see it. collborative note-taking was one of the most kick-ass things i saw 4 years ago, it simply blew me away. i didn’t have a mac at the time, so i could’t play, but seeing others do this: unbelievable. sharing ideas, supercharged. if there was no large scale use of subethaedit, was there something else? and if not - why not? open question.

service discovery

okay, so maybe all of this stuff was happening under my nose and i couldn’t find it. the only backchannel i found was summize.com. well, i’m sure i wasn’t the only person with a discovery problem. so i propse a new tool for conferences: a service registry. this idea is stolen from the recent revival of bonjour in the ruby community. people are using it to share git repositories, share their clipboards and do all kinds of perverse and fantastic stuff. there is also a bonjour based discovery service which, yes, can be used to discover all bonjour services in the network.

we need something like this, but it should be highly accessible. it should register bonjour services as well as summize.com, as well as irc channels, subethaditing, whatever. it should be easy to register services, it should even support automatic self-registration of services, like bonjour.

the website

i don’t see the need for a social network for reboot. certainly, i didn’t see this tool increasing the level of sharing. to increase sharing activity, i think the website should go back to being wiki focussed. it should also serve as the central point for service discovery.

another thing i would like to see is aggregation of flickr / 23 / ipernity, as well as blogposts and tweets. this is simply a matter of convenience. i didn’t look at flickr at all, neither did i look for blogposts. i simply forgot. i’m not a big friendfeed user, so i don’t know if this was being used as a partial aggregator. certainly, i want to aggregate around the conference, not around my friends at the conference.

also, i have no idea where aggregation of ‘feeds’ ends, and where aggregation of ‘services’ like irc channels begins. maybe all of this is the same stuff, it’s something to chew over.

thinking of the digital classroom

one of the most thought-provoking talks i ever saw was at the lift conference in geneva several years ago. pierre dillenbourge of the epfl talked about his vision of the digital classroom. his classroom is still a physically shared space, there is no remote learning or any of the crap associated with e-learning during the .com 1.0 era. but it is an augmented and collaborative space, that is full of new kinds of tools, made to share your shit! i think his ruminations are extremely applicable to reboot.

so lets bring in some of this thinking. lets find out what new sorts of tools dillenbourge would advocate for reboot. let’s invite him, or others who are looking at modern learning (or whatever you want to calle it ;), to participate in the tools design of reboot. it’s a fantastic test-bed!

rolling snowballs

i’d like to see more context around the full timeline of reboot up on the website. there’s already a bunch of stuff happening: we have the blog and proposals before the conference, then the schedule and who’s interested around the conference, and after the conference the videos go up.

there’s a lot more we could do before/around/after. mainly i’d be interested in after. i’ve always come home from reboot, laden with new ideas, but the digestion process was always something i did alone. my favourite way of digesting is to try stuff out in small, exploratory projects. wouldn’t it be great if a bunch of small group projects were kicked off at reboot, and developed after the conference?

here’s an example. the guys from soundcloud had some interesting ideas about how a track on compact disc might be less powerful than a track under a url + a lot fo context built around that url. context could be information on how the artist evolved the track in the pre-release phase, or user participation in that release phase. well, how might this apply to something else - books, for instance? the pragmatic programmers series lead the way in pre-release pdf books, but could we take it even further? a micro design project in a group of 3-4, with some people i could really jam with, that would be great. no need to finish, or code / build anything, either. then throw up all the materials on the website

it might even be possible to share the projects in the same way that github.com does it. on github, anybody can fork a public project without asking for permission. but this happens in public. this leads to huge trees of collaboration. i know that this is a programmers thing, but i suspect that there’s a general pattern here: share / fork / pull request. something to think about, maybe.

so in summing up, thanks to thomas for creating this fantastic conference! let’s all think about how to make it even more kick ass.

at play/type, we’re building a social event calender called boomloop.com. the boomloop API is a REST based API. It can be used to publish and fetch events on boomlop. you can find the api description here: http://code.google.com/p/boomloopapi/

building the API showed some shortcomings in rails out of the box REST API. Here’s a vanilla implementation of a show method:

A standard Rails REST controller

def show
  @post = Post.find(params[:id])  respond_to do |format|
    format.html
    format.xml  { render :xml => @post }
    format.js
  end
end

calling the xml representation looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<posts type="array">
  <post>
    <id>1</id>
    <title>moo</title>
    <body>grazing. lovely. </body>
    <author_notes>note to self: really must improve my crappy posts.</author_notes>
    <author_id>43</author_id>
  </post>
</posts>

Where’s the problem?

1. we’re returning internal data – author_notes is private data. attribtutes should be whitelisted by default.
2. there’s a bad smell on our keys and foreign keys. id and auhtor_id expose our id internals, but worse, they don’t talk about resources using urls. the external representation of a restful resource should be a url.

Further:

1. authentication with http basic sucks. we have much better options these days – the api should support oauth.
2. we want this stuff out of the box: sensible defaults.

Getting events closest to your current location on boomloop.com

I’ll show you how we solved these problems at boomloop.com, using an example.

Say you want to get a listing of events closest to you current location, opernplatz in frankfurt am main.

curl 'http://api.boomloop.com/events?order=proximity&origin=opernplatz,frankfurt+am+main'

Oops. This will tell you you’re not authorized. Grab your oauth client token key by checking your application under http://boomloop.com/oauth/applications. You need to put this into your header like this:

curl -H 'Authorization: loopy consumer_key="klHgtzJ78jko7Jh7g"' 'http://api.boomloop.com/events?order=proximity&origin=opernplatz,frankfurt+am+main'

Authentication

API authentication is handled by the rails OAuth plugin. You need to register your application before you can go. We decided that you can pass your oauth client key in a special header to identify you for this request. Protexted resources (such as create event) require the client to have negotiated the permission using the normal signing process.

I don’t know if this is the best way of doing things, since this way, the consoumer key can be passed around. Also, the header is a bit invented.

If you want to try this out, go to boomloop.com/api to register an application, the replace the consumer key with the key you received.

XML Structure

Now you should get some results that look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<events type="array">
  <event>
    <all-day type="boolean">true</all-day>
    <attentions-count type="integer">1</attentions-count>
    <cached-tags>opernplatz fest frankfurt</cached-tags>
    <category-id type="integer" nil="true"></category-id>
    <created-at type="datetime">2008-05-01T20:16:18Z</created-at>
    <description>Diesem Leitsatz folgt auch das Opernplatzfest, das auf gehobene Weise kulinarische und kulturelle Gen&#252;sse auf das Sch&#246;nste vereint. Urspr&#252;nglich als Brunnenfest rund um den Lucae-Springbrunnen entstanden, hat sich das Opernplatzfest im Laufe der Zeit zu einem zahlreich besuchten Sommerfest entwickelt, dessen einmaliges Flair nicht nur Banker und Werber anzieht.Getreu dem Motto "Sehen und gesehen werden" trifft man sich hier gerne unter den gro&#223;en wei&#223;en Sonnenschirmen, um bei einem Gl&#228;schen Wein oder Sekt internationale Spezialit&#228;ten zu genie&#223;en. Die Gastronomiest&#228;nde bieten den Besuchern ein gro&#223;es Spektrum an Spezialit&#228;ten, von spanischen Tapas &#252;ber frische Meeresfr&#252;chte bis hin zu japanischen Sushi. F&#252;r musikalische Unterhaltung sorgen bekannte internationale K&#252;nstler, die auf mehreren B&#252;hnen Live-Musik spielen. Besonders reizvoll wird die Stimmung des Opernplatzfestes am Abend, wenn die Fassade der Alten Oper und der Platz um den Brunnen mit vielen Lichtern feierlich beleuchtet werden.</description>
    <end-time type="datetime">2008-06-13T23:59:00Z</end-time>
    <ends type="boolean">true</ends>
    <geo type="geometry">50.115848, 8.672066</geo>
    <language>de</language>
    <main-photo-resource-url type="link" mime-type="text/xml" rel="Photo" nil="true"></main-photo-resource-url>
    <mapped type="boolean">false</mapped>
    <picture nil="true"></picture>
    <place-resource-url type="link" mime-type="text/xml" rel="Place">http://api.boomloop.com/places/389-opernplatz</place-resource-url>
    <start-time type="datetime">2008-06-13T00:00:00Z</start-time>
    <title>Opernplatzfest</title>
    <user-resource-url type="link" mime-type="text/xml" rel="User">http://api.boomloop.com/users/dribbdebach</user-resource-url>
    <website nil="true"></website>
    <resource-url>http://api.boomloop.com/events/537-opernplatzfest</resource-url>
  </event>
</events>

Notice that all ids have been replaced by a new node type, “link”. links have the general form of -resource-url, as well as a rel=type telling you what this link is about. so

<user-resource-url type="link" mime-type="text/xml" rel="User">http://api.boomloop.com/users/dribbdebach</user-resource-url>

this tells us that the link points to a User, in this case the user that created the event. staying restful, we have the full url to the text/xml representation of that user: http://api.boomloop.com/users/dribbdebach

For assoications, this strikes a compromise between something like ATOM, and the RESTful XML used by rails as a basis for Active Resource.

Writing to the API

What goes out, should go in: assocations are handled the same way when saving over the api. so user-resource-url also works when saving an event.

Code

We wrote a plugin to handle whitelisting (xml and json) and association de-referencing automatically. This is a first step in the direction of sensible defaults. If anybody is interested, i can publish the plugin and explain the code in detail in a second post.

There – it had to be said. This is hardly news, but the latest changes to comments and the network graph visualizer are just incredible goodness.

Keep up the great work guys!

This plugin fixes the rails db: tasks to respect multiple databases, and allows you to place migrations for your other dbs under db/migrate/your-special-db, starting from 0. The plugin aims to be small and non-invasive.

Making Jim Weirich feel a little bit uncomfortable

You might wonder WTF this is supposed to mean, since nobody wants to make Jim Weirich feel even slightly uncomfortable. Let me explain!

When writing this plugin, I tried to use alias_method_chain to stay the hell out of monkey patching land. The other plugin I found that does DB stuff like this had screeds of stuff copy/pasted, and this just sucks in terms of maintainablity.

You don’t get very far with this approach when you’re trying to change existing rake tasks, such as db:migrate. Sure, you can inject tasks before db:migrate like this:

task :nefarious_activities do
  puts "Pinky, are you pondering what I'm pondering?"
end

task :migrate => :nefarious_activities

But what if you want to actually run db:migrate once for each extra db you defined?

Enter alias_task_chain. It can be used just like alias_method_chain, but for tasks. Here’s an example:

  desc "migrates all defined databases."
  task :migrate_with_other_databases => :environment do
    puts "Migrating #{ RAILS_ENV }...\n\n"
    puts %x{ rake db:migrate_without_other_databases }

    for database in Loopy::MultipleDatabases.find_databases do
      all, matching_db, *mop = *RAILS_ENV.match(Loopy::MultipleDatabases::ANY_DB)
      unless matching_db.blank?
        target_environment = "#{ database }_#{ matching_db }"
        puts "Migrating #{ database }...\n\n"
        puts %x{ rake db:migrate_without_other_databases RAILS_ENV=#{ target_environment } }
      end
    end
  end

  alias_task_chain :migrate, :other_databases

I mailed Jim Weirich with this idea, and I think he found it quite unsavoury. I think he said something along the lines of ‘it makes me feel uncomfortable’. I can’t say I can blame him - it kind of creeps me out too.

I can’t think of a more natural way to achieve this though. Any ideas out there?

Tell us what the plugin is about already!

Lets say you have an Analytics DB. Start off by creating a base class like this:

class AnalyticsModel < ActiveRecord::Base
  self.abstract_class = true
  establish_connection("analytics_#{ RAILS_ENV }")
end

now you need to add the appropriate entries in your database.yml:

analytics_development:
  username: analytics
  database: analytics_development
  password: super_secret

analytics_test:
  username: analytics
  database: analytics_test
  password: super_secret

analytics_production:
  username: analytics_prod
  database: analytics_production
  password: super_secret

finally, add environment files to /environments. ie: analytics_test.rb, analytics_production.rb, analytics_development.rb.

db:migrate and db:test:clone have been enhanced to take care of all databases.

Example

With the above setup in place, you can now create a foder under db/migrate for your separate db. In this example, you’d create db/migrate/analytics.

Now create migrations in there, starting with 001-your-first-migration.rb.

You can run all the db tasks for your special dbs by setting RAILS_ENV. For example:

export RAILS_ENV=analytics_development
rake db:schema:dump

this will create db/analytics_development_schema.rb.

rake db:migrate works as normal. if you set RAILS_ENV=production, it will migrate ‘analytics_production’ as well as ‘production’.

Installation

Install the plugin:

./script/plugin install http://svn.playtype.net/plugins/loopy_multiple_databases/

Then add the following lines to your application’s Rakefile.rb, just above require ‘tasks/rails’:

# enhances rake with alias_task_chain method.
module Loopy
  module RakeExtensions
    module AliasTaskChain
      def alias_task_chain(enhancable_task, feature)
        original_name = Rake.application.current_scope.empty? ? 
          target_task : ( Rake.application.current_scope << enhancable_task ).join(":")
        chained_orignal_name = "#{ original_name }_without_#{ feature }"
        chained_enhanced_name = "#{ enhancable_task }_with_#{ feature }"
        target_task = Rake::Task[original_name]
        tasks = Rake.application.send(:eval, "@tasks")
        target_task.instance_variable_set("@name", chained_orignal_name)
        tasks.delete(original_name)
        tasks[chained_orignal_name] = target_task
        task enhancable_task => chained_enhanced_name
      end
    end
  end
end

send :include, Loopy::RakeExtensions::AliasTaskChain

Here’s a little thing I wrote for this blog. You can try it out straight up: click on the contacts link on the sidebar, then on ‘send mail to rany’. This will open in your mail application with my email address, as it should.

Now turn off javascript and reload the main page. You’ll see ‘send mail to rany’ again, but this time the link will take you to http://playtype.net/contactto/new/enal+cynlglcr+arg. This is what the spam bots are going to see. Clicking here takes you to a form that forwards your message without divulging the email in question.

This technique is called ‘Graceful email obfuscation’, and you can read up about this on a list apart in this article.

This behaviour is wrapped into the graceful mailto obfuscator rails plugin. once you’ve installed this, your mail_to helper will produce obfuscated links which behave exacty like on this page. Niceness!

Requirements

Lowpro and Prototype. Lowpro enhances Prototype with some nifty unobtrusive shit. Download from dan’s svn repository. Currently that’s 0.5: http://svn.danwebb.net/external/lowpro/tags/rel_0.5/dist/lowpro.js.

Put that in /public/javascripts then include it in your application layout:

<%= javascript_include_tag "lowpro" %>

Example

Calling mail_to(“me@mail.com”, “mail me”) will generate the following link:

<a class="obfuscated" href="/contactto/new/zr+znvy+pbz">mail me</a>

For users who do not have javascript turned on, /contactto/new collects a message which can be submitted to an action which sends the email, without it ever being divulged.

When javascript is turned on, a behaviour converts the obfuscated anchor into a normal mailto: link with the orignal email address. spam bots will now bounce off your site like frustrated popcorn.

Installation

Install the plugin:

./script/plugin install http://svn.playtype.net/plugins/graceful_mailto_obfuscator/

Then add the following lines to your application.js:

Event.addBehavior({
  "a.obfuscated": EmailDecoder
});

Include the EmailDecoder behaviour in page head:

<%= javascript_include_tag "email_decoder" %>

Graceful Degradation

Now you’ll need something to handle /contactto/new for those people who are not using javascript. the easiest thing is to have a textarea here for the message. when submitted, the server sends the email ob behalf of the user, without ever divulging the email address. here’s how to do this:

1. create a form that takes a message and passes param[:email] along to the create action after a submit.
2. inside create, decode the email like this: email = Loopy::EmailObfuscator.decode_email(CGI.escape(params[:email]))
3. send an email to this address from the server.

Done!

we’re all gearing up for an ultra-portable internet where your data is not locked up in silos, but as free and careless as ringo starr. until that day comes along, coders will continue to scrape addressbooks, unsexy 2007 stylee.

until two weeks ago, the best way of doing this in ruby was the contacts gem. the new kid on the block is the blackbook gem. it’s based on mechanize and hpricot, and comes with yahoo, gmail, hotmail, and aol scrapers ready to go. it’s much nicer than contacts, and pretty much exacly what we had in mind when we developed thief. we won’t continue developing thief, as blackbook is perfectly good for the job. instead, we’ll port the gmx, web.de and freenet providers back to blackbook.

blackbook is easy to use. first install the gem.

gem install blackbook --include-dependencies

then extract addresses like this:

contacts = Blackbook.get(:username => "bill@hotmail.com", :password => "secret")

Creating new scrapers for blackbook

lets see how to implement a new provider for gmx.de. the completed provider looks like this:

require 'blackbook/importer/page_scraper'

class Blackbook::Importer::GMX < Blackbook::Importer::PageScraper
  LOGIN_URL = "https://www.gmx.net/"

  def =~( options )
    options && options[:username] =~ /@gmx\.de$/i
  end

  def login
    username, password = options[:username], options[:password]

    begin
      page = agent.get LOGIN_URL

      form = page.forms.with.name("login").first
      form.id = username
      form.p = password

      page = form.submit

      if (continue_link = page.links.select { |link| link.text =~ /E-Mail/ }.first and 
          page.at("div.index").inner_html != "Ordnerwahl")
        page = continue_link.click
      end

      @next = page
    rescue
      raise Blackbook::BadCredentialsError.new
    end
  end

  def prepare
    login
  end

  def scrape_contacts
    page = @next

    contacts = [/Posteingang/, /Archiv/, /Gesendet/].map do |folder|
      page = page.links.select { |link| link.text =~ folder }.first.click
      find_contacts(page)
    end

    contacts.inject([]) do |memo, contact|
      memo << contact unless memo.include? contact
      memo
    end
  end

  protected

    def find_contacts(page)
      links = page.search("form#MI a").select { |link| link.attributes["title"] =~ /@/ }
      links.map do |link|
        recp = link.attributes["title"].gsub(/\n/, "").split(/\s/)
        email = recp.pop.gsub(/[<>]/, "")
        fullname = recp.join(" ")

        { :name => fullname, :email => email }
      end
    end
    Blackbook.register :gmx, self
end

here’s how it’s built:

1. create a class that extends Blackbook::Importer::PageScraper.
2. provide a =~ method which tests if the email address can be handled by this provider.
3. write a login method. in here you have access to a mechanize agent. use it to navigate your target page. you’ll find detailed documentation here.
4. create a scrape_contacts method. return a contacts hash.
5. call Blackbook.register(:gmx, self) so that blackbooks can find your provider.

Adding gmx, freenet and web.de scrapers

basically, requiring your provider and making sure it calls register is enough to get blackbook to notice you.

if you can’t be bothered and just want the goods here and now, then you can install a blackbook_extensions plugin with gmx, web.de and freenet like this:

./script/plugin install http://svn.playtype.net/plugins/schwarzesbuch/

go crazy!

how do you offload asynchrous tasks out of your rails request cycle? we needed this ability in boomloop.com, where we write data into a statistics database.

frankly, i find backgroundrb a bit scary. rumours about instability persist, and it seems like a lot of weight for a little problem. what else is out there? topfunky has a few ideas here. after checking out a few of those options, i asked evan weaver of chow.com what he thought. he said:

…If you really need a queue, use Starling, which Blaine Cook of Twitter released, like, yesterday. Or SQS if you need really huge storage. If you just want to fire and forget a local process as you say, I think Spawn is pretty good ( http://rubyforge.org/projects/spawn ). I haven’t actually used it but seems like the best of the forking bunch. That should eliminate the startup overhead. On the other hand, you don’t get any message reliability or cross-machine scheduling. ..I agree that BDrb is shady (actually all of Drb is shady). ap4r is too
bloated. Thruqueue is promising if you make it past the crazy dependencies list. BackgroundFu is like a worse Spawn.”

result: i wrote a rails plugin called workling that integrates starling into your rails app. it also lets you swap starling for any other system you might want to offload work to, without needing to change your client code. along with starling, i’ve implemented a spawn runner and a local runner.

Client code

First, create analytics_worker.rb in app/workers:

class AnalyticsWorker < Workling::Base
  def potential_invited(options)
    Hit.create :potential_user_id => options[:potential_user_id], :action => "invited" 
  end  def potential_converted(options)
    Hit.create :potential_user_id => options[:potential_user_id], :action => "converted" 
  end
end

then, call it like this anywhere in your code:

AnalyticsWorker.asynch_potential_invited(:potential_user_id => 1234)

Starling Runner

This uses Twitter’s Starling to enable your asynch code to run on different VMs. Activate it like this in your environment:

Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new

The starling runner takes care of several things:

1. mapping of queue names to worker code. this is done with Workling::ClassAndMethodRouting, but you can use your own by sublassing Workling::Routing. Some examples of Worker to queue routing: AnalyticsWorker class above routes the queues ‘analytics_worker:potential_invited’ and ‘analytics_worker:potential_converted’ to those methods on AnalyticsWorker. If you put your worker in a module, the queue will start with the_module_name:<worker_class>. don’t worry about any of this if you’re not dealing directly with the queues.
2. there’s a client daemon that waits for messages and dispatches these to the responsible workers. if you intend to run this on a remote machine, then just check out your rails project there and start up the starling client.

Other runners

workling comes with a spawn based runner. this currently seems to be one of the better spawning/forking options. use this if you don’t want to run a separate starling server in your setup.

Getting started with starling and workling

start by installing starling and the fiveruns memcached client. then install workling into your rails application. install the spawn plugin if you want the option of swapping this in instead of starling.

gem sources -a http://gems.github.com/ 
sudo gem install starling-starling 
script/plugin install git@github.com:purzelrakete/workling.git

the fiveruns-memcache-client is pulled in as a dependency to starling-starling. now you can start the little twitter birdie, followed by workling. and off you go!

mkdir /var/spool/starling
sudo starling -d
script/workling_client start

the leading workling repository is here: it://github.com/purzelrakete/workling.git. go forth and fork!

Hey! We are play/type GmbH, a tiny startup that creates lecker social web applications. In here, we will write about Ruby on Rais, Web2.0 and social software. As well as anything else that comes up while we are working.

I hope you'll like it here :).