Sunday, December 1, 2013

Upgrade to Storm 0.9.0-wip16 with JRuby 1.7.4

At Korrelate, we've been running Storm with JRuby 1.6 and Storm 0.8.3-wip3 for many months and we recently decided to upgrade to JRuby 1.7 and Storm 0.9.0. We made a few brief attempts in the past and ran into problems that exceeded the amount of time we had to do the upgrade, but this time we had more downtime to get through the problems. Here is a rundown of what we did to get everything working.

We are using version 0.6.6 of Redstorm to handle deployment and testing.

The first hurdle to overcome was getting it to run locally using redstorm local.

In your project's ivy directory, create a file called storm_dependencies.xml. Notice the snakeyaml version is set to 1.11. This is a fix for https://github.com/nathanmarz/storm/pull/567

Make sure you are using JRuby 1.7.4, we ran into additional problems with 1.7.8 and backed out for now. Now create a file called topology_dependencies.xml in your ivy directory

This will deploy your topology with JRuby 1.7.4 and it's dependencies.

You should now be able to run your topology in local mode, the next step is to deploy it to your cluster. First, we'll look at the changes required in your topology code. There are several bugs in 1.7.4 related to loading files from a jar. Make sure you only require RUBY-6970.rb in your topology. Thanks to Jordan Sissel for doing most of the work for this fix.

Since we had to upgrade snakeyaml to run it locally, we also have to update it in the storm cluster. The following change is required for project.clj in the root of your storm distribution

The last thing you may have to do is make a small change to the storm code for a bug that occurs if you're deploying on Ubuntu (possibly others??). If you're using storm-deploy you can fork the storm github repo to make the changes and point storm-deploy to your repo in the storm.clj file

A few other related notes:

For java.lang.OutOfMemoryError: PermGen space errors you can put this in your .ruby-env file if you're using a compatible ruby version manager like RVM:

Use t.options = '--use-color' if you are missing color output in your tests

I didn't track this as I was doing it so let me know if I missed anything. Good luck!

No comments:

Post a Comment