Performance degradation in SpecificRecordBuilder introduced in 1.9.0

The change introduced in spark 1.9.0 which changed:

 SpecificData.get()

into:

SpecificData.getForSchema(schema)

introduced a significant performance degradation in environments where the class of schema is provided by a different classloader then the classloader containing SpecificData.

A possible solution is to use the classCache of the default SpecificData so the sometimes expensive classloader codepath is cached. (PR coming up)

We noticed this in after trying out a spark upstep from spark 3.1.0 (avro 1.8.2) to 3.2.0 (spark 1.10.2) where 74% of the time was spend in millions of times resolving the same class.

With this patch this resolving time was brought back from 74% to 0.70%.

JMC flamegraph showing this issue: