So here's a fairly typical problem that happens in the Rails world: you have a long-running process that you need to offload/queue up, and you don't have time to fuck around with some kind of elegant solution. Before, either I'd just make the user eat it and wait for the long request (at the risk of timing out the request and/or pissing off the user), or figure out some way to procrastinate on implementing the feature in the first place.
But alas, I came to a situation that absolutely had to be offloaded: a 50MB file import that takes about 3-5 minutes to process. The request is guaranteed to time out, and it is something that will be done fairly regularly. I'd love to tell my users "Hey, this shit is going to break, but it will be the best and most awesome breaking you've ever seen," but somehow I don't think that would fly. Damn. I don't want to screw around for two days trying to cook up some kind of "scalable" solution involving worker processes and messaging queues. I don't need scalability (this will be run 2 or 3 times a week, max), I just need it to work.
When faced with a problem like this, I think web developers in general underestimate the massive amounts of thought and effort that have gone into the systems that we use every day and generally take for granted. We tend to live in our own little bubbles and think that somehow, our problems are brand new, and that they've never been solved before. Well, guess what: almost all of them have been solved before, have been solved better, and are included in almost all Unix systems out there.
Enter at
If you have a command, and want to have it wait a tick before it starts working, and want the work of several jobs to be handled in some kind of sane fashion, at is the perfect solution. Like most things Unix, at is pretty damned simple. You have a set of commands in some kind of a file. You want to execute them at some date in the future. You send those to at:
at -t 07021824 -f /path/to/my/commands. That's pretty damned beautiful.
Now, putting it in Rails is actually damned easy. I wrote a 3-line method in my ApplicationController so that I can offload any arbitrary command. Take a look at this awesome sauce:
def offload(command) job_id = MD5.hexdigest("#{command}+#{Time.now.to_i}+#{$$}") `echo "#{RAILS_ROOT}/script/runner -e #{RAILS_ENV} \\"#{command}\\"" > /tmp/#{job_id} && at -t #{1.minute.from_now.strftime("%m%d%H%M")} -f /tmp/#{job_id}` end
Every time I need to offload some arbitrary command, all I do is pass it to offload:
offload("Part.import_bom('#{massive_file}')"). Simple as pie.
Scaling? You say you want scaling? You have 3 application servers that you want to dish these offloaded processes to as equitably as possible? You are also as lazy as I am? Awesome:
APP_SERVERS = ["172.16.200.50", "172.16.200.51", "172.16.200.52"] def offload(command) job_id = MD5.hexdigest("#{command}+#{Time.now.to_i}+#{$$}") `ssh user@#{APP_SERVERS[rand(APP_SERVERS.size)]} "echo \\"#{RAILS_ROOT}/script/runner -e #{RAILS_ENV} \\\\"#{command}\\\\"\\" > /tmp/#{job_id} && at -t #{1.minute.from_now.strftime("%m%d%H%M")} -f /tmp/#{job_id}"` end
Now you've got your distributed job queue implemented. Three lines of code. Win.
Sidenote: If your dev machine is running OSX like mine is, and you want this to work, you're going to have to run the command sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.atrun.plist and restart before this stuff will work locally.