Performance regression of numeric field uninversion time

Somehow related to LUCENE-7407, after the transition, the uninvert time of numeric fields has gone up substantially. I haven't tested all field types yet, just integer fields, which show a 55% performance regression for the initial uninvert time.

This was tested with a numeric field of cardinality 1M on a 10M doc index.

q=id:1&sort=my_numeric_field desc