Solr is very slow with term vectors. Takes 55 seconds for 1000000 records.

Hi
 
I made my custom qparser plugin in Solr - 6.6.0 for scoring. The plugin only does cosine similarity of vectors for each record. I use term vectors here. Results are fine!
 
BUT, Solr response is very slow with term vectors. It takes around 55 seconds for each request for 1000000 records.
How do I make it faster to get my results in ms ?
Please respond soon as its lil urgent.
 
Note: All my values are stored and indexed. I am not using Solr Cloud.