Errors while running the specs with jruby 1.5

It looks like the before_define blocks are not passed correctly, and tasks are not created correctly. For example, the compile task doesn't become an instance of CompileTask. The from method is thus not defined on it. That gives a weird error message:
2)
NoMethodError in 'javac compiler should identify from source directories using custom layout'
undefined method `from' for <Rake::Task foo:compile => []>:Rake::Task
./spec/java/../spec_helpers.rb:144:in `execute'
./spec/java/../spec_helpers.rb:338:in `define'
./spec/java/../spec_helpers.rb:338:in `tap'
./spec/java/../spec_helpers.rb:338:in `define'
./spec/java/compiler_spec.rb:32:

The very same code, without changes, passes on JRuby 1.4.

I lean towards something fishy in the JRuby 1.5 codebase. Since they worked towards getting better integration with Rake, I wonder if they did some monkey-patching that causes us issues. I will discuss this with the JRuby maintainers over IRC (I was hinted to discuss with enebo).