Details
-
Task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
New
Description
I thought it was just the testUpdatesOnDiskFull, but looks like this one needs to be nightly too.
Should look more into the test, but I know something causes it to make such an insane amount of files, that sorting them becomes a bottleneck.
I guess also related is that it would be great if MockDirectoryWrapper's disk full check didn't trigger a sort of the files (via listAll): it does this check on like every i/o, would be nice for it to be less absurd. Maybe instead the test could check for disk full on not every i/o but some random sample of them?
Temporarily lets make it nightly...
PROFILE SUMMARY from 182501 samples tests.profile.count=10 tests.profile.stacksize=1 tests.profile.linenumbers=false PERCENT SAMPLES STACK 15.89% 28995 java.lang.StringLatin1#compareTo() 6.61% 12069 java.util.TimSort#mergeHi() 5.96% 10878 java.util.TimSort#binarySort() 3.41% 6231 java.util.concurrent.ConcurrentHashMap#tabAt() 2.98% 5433 java.util.Comparators$NaturalOrderComparator#compare() 2.12% 3876 org.apache.lucene.store.DataOutput#copyBytes() 2.03% 3712 java.lang.String#compareTo() 1.84% 3350 java.util.concurrent.ConcurrentHashMap#get() 1.83% 3337 java.util.TimSort#mergeLo() 1.67% 3047 java.util.ArrayList#add()
All the file sorting is called from stacks like this, so its literally happening every writeByte() and so on
0.73% 1329 java.util.TimSort#binarySort() at java.util.TimSort#sort() at java.util.Arrays#sort() at java.util.ArrayList#sort() at java.util.stream.SortedOps$RefSortingSink#end() at java.util.stream.AbstractPipeline#copyInto() at java.util.stream.AbstractPipeline#wrapAndCopyInto() at java.util.stream.AbstractPipeline#evaluate() at java.util.stream.AbstractPipeline#evaluateToArrayNode() at java.util.stream.ReferencePipeline#toArray() at org.apache.lucene.store.ByteBuffersDirectory#listAll() at org.apache.lucene.store.MockDirectoryWrapper#sizeInBytes() at org.apache.lucene.store.MockIndexOutputWrapper#checkDiskFull() at org.apache.lucene.store.MockIndexOutputWrapper#writeBytes() at org.apache.lucene.store.MockIndexOutputWrapper#writeByte() at org.apache.lucene.store.DataOutput#writeInt() at org.apache.lucene.codecs.CodecUtil#writeFooter() at org.apache.lucene.codecs.lucene50.Lucene50LiveDocsFormat#writeLiveDocs() at org.apache.lucene.codecs.asserting.AssertingLiveDocsFormat#writeLiveDocs() at org.apache.lucene.index.PendingDeletes#writeLiveDocs()