play/type blog

last.fm uses old school irc to connect developers and business people inside the company. the last.fm founder wrote up an article about how they do this with the irccat bot.

…Everyone that works at Last.fm is typically connected to our IRC server. We have different channels per team, as well as a company-wide channel, and a few channels dedicated to automated monitoring. Sometimes it makes much more sense to discuss / ask questions on IRC instead of email, and it’s useful to be able to raise people who are not in the office. That said, the main reason I’m writing this post is to mention the dev-support bot we use: irccat.

i’ve written a ruby eventmachine clone of the irc bot they wrote. it’s called mini, and i tried to keep it as tiny as possible, which was fun :-). using mini…

…you can easily send events to irc from shell scripts:

echo "Ilovethistuffs yes .. i .. do. " | nc localhost 12345 
vmstat | nc localhost 12345 
echo "#musicteam,#legal,@alice New album uploaded: ..." | nc somemachine 12345 
tail -f /var/log/important.log | nc somemachine 12345

you can also send commands from inside the channels using `?command` syntax. this will invoke a script called minictl on your $PATH.

you can also post to the bot over a url. there’s simple mechanism for adding plugins, as well.

it’s a neat little toy, check out the source on github.

Installation

sudo gem install purzelrakete-mini --source=http://gems.github.com

create a directory for mini and run the following

minigen create 

this creates two files in your mini dir. edit both as you see fit, then add your mini dir to your $PATH inside of your shell’s rc file.

It’s possible to write Linux desktop apps using ruby! All you need is a distribution running KDE 4.2. I have Kubuntu running on my Macbook.

First off, get the Ruby bindings installed:

apt-get install ruby-kde4

Next up, create a project directory for your Plasmoid. You can follow along with ‘Rakete’, which is my attempt at a Twitter client. The only thing Rakete does so far is to load and display an SVG.

you can call your root directory anything you like. In my case, it’s ‘application’ instread of ‘rakete’.

The `contents`directory holds the main files in a number of subdirectories. The most important of these is `code`, which, you guessed it, holds your ruby entry point, `main.rb`. `images` holds the SVG we want to render. You can create SVGs with a number of tools such as Adobe Illustrator or Inkscape (open source).

require 'plasma_applet'

module Rakete
  class Main < PlasmaScripting::Applet
end

def initialize(parent, args = nil)
    super
  end
end

def init
  @svg = Plasma::Svg.new(self)
  @svg.imagePath = 'widgets/rakete'
end

def paintInterface(painter, option, contentsRect)
  @svg.resize(size())
  @svg.paint(painter, 0, 0)
end

`main.rb`should contain a class called `Main`, inside of a module named after your application. There are a number of callbacks you have to implement.

First off, you have ìnit`, which is called when the applet is first loaded. In here, we construct Plasma::Svg, which loads up rakete.svg.

SVGs are read from the svg.imagePath. This is relative to the $KDE_ROOT/apps/desktoptheme/default. KDE_ROOT varies from system to system. In the case of Kubuntu, it is /usr/share/kde4. So specifying an svg image path of

@svg.imagePath = 'widgets/rakete'

will mean that the svg is searched for under

/usr/share/kde4/apps/desktoptheme/default/widgets/rakete.svg

This lies outside of your Plasmoid directory structure, so the SVGs you package have to be staged into the widgets directory. Currently I don’t see any way of automating this staging process, so you’ll have to copy the svg from images/ over to the widgets directory. I’ll post more info as soon as I get it.

UPDATE 25.12.08: you can reference images inside of your plasmoid directory structure in your code. it turns out that doing so is very easy. applets have a `package` method, which returns an object representing your plasmoid package. the package object can resolve the canonical filename of your svg like this:

@svg.imagePath = package.file_path("images", "rakete.svg")

now you don’t need to worry about staging anything.

Next, create the metadata.desktop file:

[Desktop Entry]
Name=Rakete
Comment=an attempt at a better twitter plasmoid using ruby. 
Icon=
Type=Service
ServiceTypes=Plasma/Applet
X-Plasma-API=ruby-script

X-KDE-PluginInfo-Author=Rany Keddo
X-KDE-PluginInfo-Email=purzelrakete@gmail.com    
X-KDE-PluginInfo-Name=rakete
X-KDE-PluginInfo-Version=pre0.1    
X-KDE-PluginInfo-Website=http://playtype.net
X-KDE-PluginInfo-Category=
X-KDE-PluginInfo-Depends=
X-KDE-PluginInfo-License=MIT
X-KDE-PluginInfo-EnabledByDefault=true

Replace the pertinent field values.

Finally, you’ll want to run this thing. Run the following above the plasma root directory:

plasmapkg --upgrade application

This will install the Plasmoid on your system, or update it is already installed. Note that if you plasmoid root directory has a different name to “application”, you’ll need to cahnge the packaging command accordingly.

Finally, open the cashew in the top righthand corner of your desktop and select “Add Widget”. You’ll find your widget in the list.

As it stands, Plasma will sometimes crash when running this Applet. I’ll pass this information on to Richard Dale, the maintainer of the Ruby bindings. Hopefully he will be able to locate problems in the runtime, which is still in Beta1.

Richard was also super helpful in getting this example to run. Thank you Richard :-)

I’ll write more posts as I make progress with the Twitter client. You can find Rakete on Github.

here’s a little thing that made me happy today. i spent some time automating pdf printing. i have a directory structure like so: receipts/2008/10. when needed, everything is packed into a single pdf and sent to the printer down the road. to do this, every file has to be converted to pdf and combined to a single file. input formats are rtf, txt, pdf or multipage tiff.

so today wrote a dirty ruby script for this. mashing little unix tools into your ruby code is good. dirty, trash-eating shell opening code, i salute you! it’s taken me a while to come around to this, being the ex-java i am. 2 years away from jcp.org seem to have done me good ;)

Low effort pdf conversion using the shell

sam, suggested i should look into ruby-cocoa, since preview can print many formats to pdf. but a bit of googling lead me to CUPS-PDF, with which you can set up a printer which dumps pdfs into a specified directory. Follow along at http://www.codepoetry.net/projects/cups-pdf-for-mosx.

So lets see if we have everything we need to get started. First off, lets check if the CUPS printer has been installed:

    raise Exception.new("install cups.") unless `lpstat -a`.grep Regexp.new(CUPS_PRINTER)

lpstat -a gives you a list of all printing devices configured on your machine. We’re going to search this for CUPS_PRINTER using grep. We’ll actually use this printer to convert input files, like rtfs, into pdf files on disk.

So now we want to Grab the files and pass them to the printer. This should all be automatic. A quick search reveals lp:

    `lp -d #{ CUPS_PRINTER } #{ file }`

Nice, this now dumps the converted file into the CUPS root directory, which by default is cups-pdf on your desktop.

Multipage Tiffs

Unfortunately lp only prints the first page of multipage TIFFS to pdf. So what we need to to do is to extract the pages from the tiff. Lets see if there’s anything available:

    man -k tiff

This shows us tiffutil, which can be utilized to ‘manipulate tiff images’. Lets see how this works:

    man tiffutil

Bingo, there’s an -extract option which pulls out the specified page. So now we just need to know how many pages are in the tiff. There’s an -info option that gives us the information we need. Each image is described with an entry that starts with “Directory at”. Lets count the number of times this appears:

     `tiffutil -info #{ file } | grep "Directory at" | wc -l`.strip.to_i

Here we’re piping the output of tiffutil to grep, which results in a line per found image. We pipe this through wc -l to count the number of images.

Combining the pdf

Thanks again to sam for pointing me to pdfcombine. just specify all files you want to combine, the outfile with -out, and you’re done. So let’s see if it’s installed…

    raise Exception.new("install pdfcombine command line tool. ") unless `which pdfcombine`

which searches $PATH for the specified executable. The full path is returned. To run it:

     `pdfcombine #{ exports.join(" ") } -o #{ packfile }`

And you’re done!

The script

    require 'fileutils'
    require 'logger'

    #
    #  Combines all txt, rtf, tiff multipage, pdf files in a named subdirectory. 
    #  Results in a single pdf called <subdir>-pack.pdf 
    #
    class ReceiptsPrinter

      # requires you to be in this subdirectory for safety reasons (FileUtils.rm_r is used.)
      RECEIPTS_ROOT = "/Users/me/Documents/2008"

      # cups prints to this directory
      CUPS_ROOT = "/Users/me/Desktop/cups-pdf"

      # cups printer name
      CUPS_PRINTER = "CUPS_PDF"

      # 
      #  pass in the subdirectory name which includes the files
      #
      def initialize(subdir)
        @subdir = subdir
        @files = Dir.glob("#{ subdir }/*.*")
        @export_dir = "#{ subdir }/export/"
        @logger = Logger.new(STDOUT)

        # please meet the conditions.   
        raise Exception.new("specify the subdir to process.") unless @subdir
        raise Exception.new("cd to #{ RECEIPTS_ROOT } before starting.") if `pwd`.chomp != RECEIPTS_ROOT
        raise Exception.new("install pdfcombine command line tool. ") unless `which pdfcombine`
        raise Exception.new("install cups to enable conversions: http://www.codepoetry.net/projects/cups-pdf-for-mosx ") unless `lpstat -a`.grep Regexp.new(CUPS_PRINTER)

        # sets up the export directory
        FileUtils.rm_r @export_dir rescue nil
        FileUtils.mkdir_p @export_dir
      end

      #
      #  iterates through all files in subdirectory, converting them to pdf if necessary, then combines them
      #  into a single pdf. 
      #
      def print
        @files.each { |file| convert(file) }
        packfile = combine

        @logger.info "Done! Packed it up into #{ packfile }."
      end

      private
        def convert(file)
          case type(file)
          when "pdf" : stage(file)
          when "tiff" 
            extract_tiff_pages(file).each do |file| 
              stage cups_converter(file)
            end
          else 
            stage cups_converter(file)
          end
        end

        def stage(file)
          @logger.info "staging file #{ file }"
          dest = @export_dir + file.split('/').last
          `cp #{ file } #{ dest }`
        end

        def combine
          exports = Dir.glob("#{ @export_dir }*.pdf")
          packfile = "#{ @export_dir }/#{ @subdir }-packed.pdf"
          command = "pdfcombine #{ exports.join(" ") } -o #{ packfile }"

          # pack it up!
          `#{ command }`

          packfile
        end

        def cups_converter(file)
          command = "lp -d #{ CUPS_PRINTER } #{ file }"
          regexp = Regexp.new("Auftrags-ID ist #{ CUPS_PRINTER }-(\d*) .*")
          string, id =  *`#{ command }`.match(regexp)
          raise Exception.new("cups printing failed for #{ file }: #{ string }") unless id

          # give the thing a moment to generate
          sleep 1 while !(pdf = Dir.glob("#{ CUPS_ROOT }/job_#{ id }-*.pdf").first)

          pdf
        end

        def extract_tiff_pages(file)
          (1..tiff_page_number(file)).map do |page|
            extract_tiff_page(file, page - 1)
          end
        end

        def extract_tiff_page(file, page)
          filename = "#{ @export_dir }#{ page }-#{ file.split('/').last }"

          @logger.info "tiff: #{ filename }"
          `tiffutil -extract #{ page } #{ file } -out #{ filename }`

          filename
        end

        def tiff_page_number(file)
          `tiffutil -info #{ file } | grep "Directory at" | wc -l`.strip.to_i
        end

        def type(file); file.split(".").last; end
    end

    printer = ReceiptsPrinter.new(month = ARGV.first)
    puts ">>>>> Please back the files in #{ month } before doing this. Starting; press CTL-C to abort. \n\n"
    printer.print

First off, you’ll need to install RabbitMQ. Instructions for OSX here. Once you’ve done that, install the pure ruby amqp library:

    gem sources -a http://gems.github.com
    gem install tmm1-amqp

You’re good to go. Now open up two IRB sessions. Paste the following code into the first session:

    require 'mq'

    EM.run {
      amq = MQ.new
      EM.add_periodic_timer(1) { amq.queue("noises").publish("moo") }
    }

Your publishing code has to run inside of an Event Machine loop. You can start it using EM.run. If you’re running inside of an Evented Container such as Thin or Evented Mongrel, you can skip this. The meat of it is just amq.queue("noises").publish("moo")that.

Now open a second irb session in another terminal tab, and paste this in:

    require 'mq'

    EM.run {
      amq = MQ.new
      amq.queue("noises").subscribe { |noise|
        puts noise
      }
    }

You’ll get the moos off the Queue “noises”. No polling required, which is very nice, you simply register a callback with amqp, and it’ll be called with the message when one becomes available.

Niceness!

how do you offload asynchrous tasks out of your rails request cycle? we needed this ability in boomloop.com, where we write data into a statistics database.

frankly, i find backgroundrb a bit scary. rumours about instability persist, and it seems like a lot of weight for a little problem. what else is out there? topfunky has a few ideas here. after checking out a few of those options, i asked evan weaver of chow.com what he thought. he said:

…If you really need a queue, use Starling, which Blaine Cook of Twitter released, like, yesterday. Or SQS if you need really huge storage. If you just want to fire and forget a local process as you say, I think Spawn is pretty good ( http://rubyforge.org/projects/spawn ). I haven’t actually used it but seems like the best of the forking bunch. That should eliminate the startup overhead. On the other hand, you don’t get any message reliability or cross-machine scheduling. ..I agree that BDrb is shady (actually all of Drb is shady). ap4r is too
bloated. Thruqueue is promising if you make it past the crazy dependencies list. BackgroundFu is like a worse Spawn.”

result: i wrote a rails plugin called workling that integrates starling into your rails app. it also lets you swap starling for any other system you might want to offload work to, without needing to change your client code. along with starling, i’ve implemented a spawn runner and a local runner.

Client code

First, create analytics_worker.rb in app/workers:

class AnalyticsWorker < Workling::Base
  def potential_invited(options)
    Hit.create :potential_user_id => options[:potential_user_id], :action => "invited" 
  end  def potential_converted(options)
    Hit.create :potential_user_id => options[:potential_user_id], :action => "converted" 
  end
end

then, call it like this anywhere in your code:

AnalyticsWorker.asynch_potential_invited(:potential_user_id => 1234)

Starling Runner

This uses Twitter’s Starling to enable your asynch code to run on different VMs. Activate it like this in your environment:

Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new

The starling runner takes care of several things:

1. mapping of queue names to worker code. this is done with Workling::ClassAndMethodRouting, but you can use your own by sublassing Workling::Routing. Some examples of Worker to queue routing: AnalyticsWorker class above routes the queues ‘analytics_worker:potential_invited’ and ‘analytics_worker:potential_converted’ to those methods on AnalyticsWorker. If you put your worker in a module, the queue will start with the_module_name:<worker_class>. don’t worry about any of this if you’re not dealing directly with the queues.
2. there’s a client daemon that waits for messages and dispatches these to the responsible workers. if you intend to run this on a remote machine, then just check out your rails project there and start up the starling client.

Other runners

workling comes with a spawn based runner. this currently seems to be one of the better spawning/forking options. use this if you don’t want to run a separate starling server in your setup.

Getting started with starling and workling

start by installing starling and the fiveruns memcached client. then install workling into your rails application. install the spawn plugin if you want the option of swapping this in instead of starling.

gem sources -a http://gems.github.com/ 
sudo gem install starling-starling 
script/plugin install git@github.com:purzelrakete/workling.git

the fiveruns-memcache-client is pulled in as a dependency to starling-starling. now you can start the little twitter birdie, followed by workling. and off you go!

mkdir /var/spool/starling
sudo starling -d
script/workling_client start

the leading workling repository is here: it://github.com/purzelrakete/workling.git. go forth and fork!