at play/type, we’re building a social event calender called boomloop.com. the boomloop API is a REST based API. It can be used to publish and fetch events on boomlop. you can find the api description here: http://code.google.com/p/boomloopapi/
building the API showed some shortcomings in rails out of the box REST API. Here’s a vanilla implementation of a show method:
A standard Rails REST controller
def show
@post = Post.find(params[:id]) respond_to do |format|
format.html
format.xml { render :xml => @post }
format.js
end
end
calling the xml representation looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<posts type="array">
<post>
<id>1</id>
<title>moo</title>
<body>grazing. lovely. </body>
<author_notes>note to self: really must improve my crappy posts.</author_notes>
<author_id>43</author_id>
</post>
</posts>
Where’s the problem?
- we’re returning internal data – author_notes is private data. attribtutes should be whitelisted by default.
- there’s a bad smell on our keys and foreign keys. id and auhtor_id expose our id internals, but worse, they don’t talk about resources using urls. the external representation of a restful resource should be a url.
Further:
- authentication with http basic sucks. we have much better options these days – the api should support oauth.
- we want this stuff out of the box: sensible defaults.
Getting events closest to your current location on boomloop.com
I’ll show you how we solved these problems at boomloop.com, using an example.
Say you want to get a listing of events closest to you current location, opernplatz in frankfurt am main.
curl 'http://api.boomloop.com/events?order=proximity&origin=opernplatz,frankfurt+am+main'Oops. This will tell you you’re not authorized. Grab your oauth client token key by checking your application under http://boomloop.com/oauth/applications. You need to put this into your header like this:
curl -H 'Authorization: loopy consumer_key="klHgtzJ78jko7Jh7g"' 'http://api.boomloop.com/events?order=proximity&origin=opernplatz,frankfurt+am+main'Authentication
API authentication is handled by the rails OAuth plugin. You need to register your application before you can go. We decided that you can pass your oauth client key in a special header to identify you for this request. Protexted resources (such as create event) require the client to have negotiated the permission using the normal signing process.
I don’t know if this is the best way of doing things, since this way, the consoumer key can be passed around. Also, the header is a bit invented.
If you want to try this out, go to boomloop.com/api to register an application, the replace the consumer key with the key you received.
XML Structure
Now you should get some results that look something like this:
<?xml version="1.0" encoding="UTF-8"?>
<events type="array">
<event>
<all-day type="boolean">true</all-day>
<attentions-count type="integer">1</attentions-count>
<cached-tags>opernplatz fest frankfurt</cached-tags>
<category-id type="integer" nil="true"></category-id>
<created-at type="datetime">2008-05-01T20:16:18Z</created-at>
<description>Diesem Leitsatz folgt auch das Opernplatzfest, das auf gehobene Weise kulinarische und kulturelle Genüsse auf das Schönste vereint. Ursprünglich als Brunnenfest rund um den Lucae-Springbrunnen entstanden, hat sich das Opernplatzfest im Laufe der Zeit zu einem zahlreich besuchten Sommerfest entwickelt, dessen einmaliges Flair nicht nur Banker und Werber anzieht.Getreu dem Motto "Sehen und gesehen werden" trifft man sich hier gerne unter den großen weißen Sonnenschirmen, um bei einem Gläschen Wein oder Sekt internationale Spezialitäten zu genießen. Die Gastronomiestände bieten den Besuchern ein großes Spektrum an Spezialitäten, von spanischen Tapas über frische Meeresfrüchte bis hin zu japanischen Sushi. Für musikalische Unterhaltung sorgen bekannte internationale Künstler, die auf mehreren Bühnen Live-Musik spielen. Besonders reizvoll wird die Stimmung des Opernplatzfestes am Abend, wenn die Fassade der Alten Oper und der Platz um den Brunnen mit vielen Lichtern feierlich beleuchtet werden.</description>
<end-time type="datetime">2008-06-13T23:59:00Z</end-time>
<ends type="boolean">true</ends>
<geo type="geometry">50.115848, 8.672066</geo>
<language>de</language>
<main-photo-resource-url type="link" mime-type="text/xml" rel="Photo" nil="true"></main-photo-resource-url>
<mapped type="boolean">false</mapped>
<picture nil="true"></picture>
<place-resource-url type="link" mime-type="text/xml" rel="Place">http://api.boomloop.com/places/389-opernplatz</place-resource-url>
<start-time type="datetime">2008-06-13T00:00:00Z</start-time>
<title>Opernplatzfest</title>
<user-resource-url type="link" mime-type="text/xml" rel="User">http://api.boomloop.com/users/dribbdebach</user-resource-url>
<website nil="true"></website>
<resource-url>http://api.boomloop.com/events/537-opernplatzfest</resource-url>
</event>
</events>
Notice that all ids have been replaced by a new node type, “link”. links have the general form of
<user-resource-url type="link" mime-type="text/xml" rel="User">http://api.boomloop.com/users/dribbdebach</user-resource-url>
this tells us that the link points to a User, in this case the user that created the event. staying restful, we have the full url to the text/xml representation of that user: http://api.boomloop.com/users/dribbdebach
For assoications, this strikes a compromise between something like ATOM, and the RESTful XML used by rails as a basis for Active Resource.
Writing to the API
What goes out, should go in: assocations are handled the same way when saving over the api. so user-resource-url also works when saving an event.
Code
We wrote a plugin to handle whitelisting (xml and json) and association de-referencing automatically. This is a first step in the direction of sensible defaults. If anybody is interested, i can publish the plugin and explain the code in detail in a second post.
This plugin fixes the rails db: tasks to respect multiple databases, and allows you to place migrations for your other dbs under db/migrate/your-special-db, starting from 0. The plugin aims to be small and non-invasive.
Making Jim Weirich feel a little bit uncomfortable
You might wonder WTF this is supposed to mean, since nobody wants to make Jim Weirich feel even slightly uncomfortable. Let me explain!
When writing this plugin, I tried to use alias_method_chain to stay the hell out of monkey patching land. The other plugin I found that does DB stuff like this had screeds of stuff copy/pasted, and this just sucks in terms of maintainablity.
You don’t get very far with this approach when you’re trying to change existing rake tasks, such as db:migrate. Sure, you can inject tasks before db:migrate like this:
task :nefarious_activities do
puts "Pinky, are you pondering what I'm pondering?"
end
task :migrate => :nefarious_activities
But what if you want to actually run db:migrate once for each extra db you defined?
Enter alias_task_chain. It can be used just like alias_method_chain, but for tasks. Here’s an example:
desc "migrates all defined databases."
task :migrate_with_other_databases => :environment do
puts "Migrating #{ RAILS_ENV }...\n\n"
puts %x{ rake db:migrate_without_other_databases }
for database in Loopy::MultipleDatabases.find_databases do
all, matching_db, *mop = *RAILS_ENV.match(Loopy::MultipleDatabases::ANY_DB)
unless matching_db.blank?
target_environment = "#{ database }_#{ matching_db }"
puts "Migrating #{ database }...\n\n"
puts %x{ rake db:migrate_without_other_databases RAILS_ENV=#{ target_environment } }
end
end
end
alias_task_chain :migrate, :other_databases
I mailed Jim Weirich with this idea, and I think he found it quite unsavoury. I think he said something along the lines of ‘it makes me feel uncomfortable’. I can’t say I can blame him - it kind of creeps me out too.
I can’t think of a more natural way to achieve this though. Any ideas out there?
Tell us what the plugin is about already!
Lets say you have an Analytics DB. Start off by creating a base class like this:
class AnalyticsModel < ActiveRecord::Base
self.abstract_class = true
establish_connection("analytics_#{ RAILS_ENV }")
end
now you need to add the appropriate entries in your database.yml:
analytics_development:
username: analytics
database: analytics_development
password: super_secret
analytics_test:
username: analytics
database: analytics_test
password: super_secret
analytics_production:
username: analytics_prod
database: analytics_production
password: super_secret
finally, add environment files to /environments. ie: analytics_test.rb, analytics_production.rb, analytics_development.rb.
db:migrate and db:test:clone have been enhanced to take care of all databases.
Example
With the above setup in place, you can now create a foder under db/migrate for your separate db. In this example, you’d create db/migrate/analytics.
Now create migrations in there, starting with 001-your-first-migration.rb.
You can run all the db tasks for your special dbs by setting RAILS_ENV. For example:
export RAILS_ENV=analytics_development
rake db:schema:dump
this will create db/analytics_development_schema.rb.
rake db:migrate works as normal. if you set RAILS_ENV=production, it will migrate ‘analytics_production’ as well as ‘production’.
Installation
Install the plugin:
./script/plugin install http://svn.playtype.net/plugins/loopy_multiple_databases/
Then add the following lines to your application’s Rakefile.rb, just above require ‘tasks/rails’:
# enhances rake with alias_task_chain method.
module Loopy
module RakeExtensions
module AliasTaskChain
def alias_task_chain(enhancable_task, feature)
original_name = Rake.application.current_scope.empty? ?
target_task : ( Rake.application.current_scope << enhancable_task ).join(":")
chained_orignal_name = "#{ original_name }_without_#{ feature }"
chained_enhanced_name = "#{ enhancable_task }_with_#{ feature }"
target_task = Rake::Task[original_name]
tasks = Rake.application.send(:eval, "@tasks")
target_task.instance_variable_set("@name", chained_orignal_name)
tasks.delete(original_name)
tasks[chained_orignal_name] = target_task
task enhancable_task => chained_enhanced_name
end
end
end
end
send :include, Loopy::RakeExtensions::AliasTaskChain
Here’s a little thing I wrote for this blog. You can try it out straight up: click on the contacts link on the sidebar, then on ‘send mail to rany’. This will open in your mail application with my email address, as it should.
Now turn off javascript and reload the main page. You’ll see ‘send mail to rany’ again, but this time the link will take you to http://playtype.net/contactto/new/enal+cynlglcr+arg. This is what the spam bots are going to see. Clicking here takes you to a form that forwards your message without divulging the email in question.
This technique is called ‘Graceful email obfuscation’, and you can read up about this on a list apart in this article.
This behaviour is wrapped into the graceful mailto obfuscator rails plugin. once you’ve installed this, your mail_to helper will produce obfuscated links which behave exacty like on this page. Niceness!
Requirements
Lowpro and Prototype. Lowpro enhances Prototype with some nifty unobtrusive shit. Download from dan’s svn repository. Currently that’s 0.5: http://svn.danwebb.net/external/lowpro/tags/rel_0.5/dist/lowpro.js.
Put that in /public/javascripts then include it in your application layout:
<%= javascript_include_tag "lowpro" %>
Example
Calling mail_to(“me@mail.com”, “mail me”) will generate the following link:
<a class="obfuscated" href="/contactto/new/zr+znvy+pbz">mail me</a>
For users who do not have javascript turned on, /contactto/new collects a message which can be submitted to an action which sends the email, without it ever being divulged.
When javascript is turned on, a behaviour converts the obfuscated anchor into a normal mailto: link with the orignal email address. spam bots will now bounce off your site like frustrated popcorn.
Installation
Install the plugin:
./script/plugin install http://svn.playtype.net/plugins/graceful_mailto_obfuscator/
Then add the following lines to your application.js:
Event.addBehavior({
"a.obfuscated": EmailDecoder
});
Include the EmailDecoder behaviour in page head:
<%= javascript_include_tag "email_decoder" %>
Graceful Degradation
Now you’ll need something to handle /contactto/new for those people who are not using javascript. the easiest thing is to have a textarea here for the message. when submitted, the server sends the email ob behalf of the user, without ever divulging the email address. here’s how to do this:
- create a form that takes a message and passes param[:email] along to the create action after a submit.
- inside create, decode the email like this: email = Loopy::EmailObfuscator.decode_email(CGI.escape(params[:email]))
- send an email to this address from the server.
Done!
how do you offload asynchrous tasks out of your rails request cycle? we needed this ability in boomloop.com, where we write data into a statistics database.
frankly, i find backgroundrb a bit scary. rumours about instability persist, and it seems like a lot of weight for a little problem. what else is out there? topfunky has a few ideas here. after checking out a few of those options, i asked evan weaver of chow.com what he thought. he said:
…If you really need a queue, use Starling, which Blaine Cook of Twitter released, like, yesterday. Or SQS if you need really huge storage. If you just want to fire and forget a local process as you say, I think Spawn is pretty good ( http://rubyforge.org/projects/spawn ). I haven’t actually used it but seems like the best of the forking bunch. That should eliminate the startup overhead. On the other hand, you don’t get any message reliability or cross-machine scheduling. ..I agree that BDrb is shady (actually all of Drb is shady). ap4r is too
bloated. Thruqueue is promising if you make it past the crazy dependencies list. BackgroundFu is like a worse Spawn.”
result: i wrote a rails plugin called workling that integrates starling into your rails app. it also lets you swap starling for any other system you might want to offload work to, without needing to change your client code. along with starling, i’ve implemented a spawn runner and a local runner.
Client code
First, create analytics_worker.rb in app/workers:
class AnalyticsWorker < Workling::Base
def potential_invited(options)
Hit.create :potential_user_id => options[:potential_user_id], :action => "invited"
end def potential_converted(options)
Hit.create :potential_user_id => options[:potential_user_id], :action => "converted"
end
end
then, call it like this anywhere in your code:
AnalyticsWorker.asynch_potential_invited(:potential_user_id => 1234)
Starling Runner
This uses Twitter’s Starling to enable your asynch code to run on different VMs. Activate it like this in your environment:
Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
The starling runner takes care of several things:
- mapping of queue names to worker code. this is done with Workling::ClassAndMethodRouting, but you can use your own by sublassing Workling::Routing. Some examples of Worker to queue routing: AnalyticsWorker class above routes the queues ‘analytics_worker:potential_invited’ and ‘analytics_worker:potential_converted’ to those methods on AnalyticsWorker. If you put your worker in a module, the queue will start with the_module_name:<worker_class>. don’t worry about any of this if you’re not dealing directly with the queues.
- there’s a client daemon that waits for messages and dispatches these to the responsible workers. if you intend to run this on a remote machine, then just check out your rails project there and start up the starling client.
Other runners
workling comes with a spawn based runner. this currently seems to be one of the better spawning/forking options. use this if you don’t want to run a separate starling server in your setup.
Getting started with starling and workling
start by installing starling and a memcached client. then install workling into your rails application. install the spawn plugin if you want the option of swapping this in instead of starling.
sudo gem install starling --include-dependencies
sudo gem install memcache-client --include-dependencies
./script/plugin install http://svn.playtype.net/plugins/workling/
./script/plugin install http://spawn.rubyforge.org/svn/spawn/
now you can start the little twitter birdie, followed by workling. and off you go!
mkdir /var/spool/starling
sudo starling -d
script/workling_starling_client start
the leading workling repository is here: it://github.com/purzelrakete/workling.git. go forth and fork!
