Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.4.0
-
None
-
None
Description
Typical result are as follows:
2.12.16
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1031-azure Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Coalesced RDD: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ---------------------------------------------------------------------------------------------------------------------------- Coalesce Num Partitions: 500 Num Hosts: 1 659 671 19 0.2 6588.2 0.4X Coalesce Num Partitions: 1000 Num Hosts: 1 1111 1133 25 0.1 11114.3 0.3X Coalesce Num Partitions: 5000 Num Hosts: 1 4573 4580 12 0.0 45727.1 0.1X Coalesce Num Partitions: 10000 Num Hosts: 1 8949 9107 240 0.0 89494.7 0.0X
2.13.8
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1031-azure Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Coalesced RDD: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ---------------------------------------------------------------------------------------------------------------------------- Coalesce Num Partitions: 500 Num Hosts: 1 964 1131 284 0.1 9643.3 0.3X Coalesce Num Partitions: 1000 Num Hosts: 1 1732 1742 10 0.1 17318.2 0.2X Coalesce Num Partitions: 5000 Num Hosts: 1 7534 7539 4 0.0 75339.8 0.0X Coalesce Num Partitions: 10000 Num Hosts: 1 14524 14565 37 0.0 145245.0 0.0X
From the analysis of jfr results, the hot path is `ArrayBuffer.min`
def getLeastGroupHash(key: String): Option[PartitionGroup] =
groupHash.get(key).filter(_.nonEmpty).map(_.min)
then I write a new simple bench to test `ArrayBuffer.min` and `ArrayBuffer.max`:
private def min(numIters: Int, bufferSize: Int, loops: Int): Unit = { val benchmark = new Benchmark("Array Buffer", loops, output = output) val buffer = new mutable.ArrayBuffer[Int]() (0 until bufferSize).foreach(i => buffer += i) benchmark.addCase(s"buffer min with $bufferSize items", numIters) { _ => (0 until loops).foreach(_ => buffer.min) } benchmark.addCase(s"buffer sorted head with $bufferSize items", numIters) { _ => (0 until loops).foreach(_ => buffer.sorted.head) } benchmark.addCase(s"buffer max with $bufferSize items", numIters) { _ => (0 until loops).foreach(_ => buffer.max) } benchmark.addCase(s"buffer sorted last with $bufferSize items", numIters) { _ => (0 until loops).foreach(_ => buffer.sorted.last) } benchmark.run() } override def runBenchmarkSuite(mainArgs: Array[String]): Unit = { val numIters = 3 val loops = 20000 runBenchmark("ArrayBuffer min and max") { Seq(1000, 10000, 100000).foreach { bufferSize => min(numIters, bufferSize, loops) } } }
The result as follows:
Scala 2.12
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1031-azure Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Array Buffer: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ buffer min with 1000 items 121 121 0 0.2 6043.9 1.0X buffer max with 1000 items 134 134 0 0.1 6705.5 0.9XOpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1031-azure Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Array Buffer: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ buffer min with 10000 items 1272 1272 0 0.0 63601.9 1.0X buffer max with 10000 items 1339 1340 0 0.0 66972.2 0.9XOpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1031-azure Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Array Buffer: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ buffer min with 100000 items 12785 12788 3 0.0 639232.4 1.0X buffer max with 100000 items 13433 13434 1 0.0 671654.2 1.0X
Scala 2.13
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1031-azure Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Array Buffer: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ buffer min with 1000 items 273 273 0 0.1 13646.5 1.0X buffer max with 1000 items 162 162 0 0.1 8076.9 1.7XOpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1031-azure Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Array Buffer: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ buffer min with 10000 items 1676 1676 0 0.0 83782.1 1.0X buffer max with 10000 items 1610 1610 0 0.0 80485.5 1.0XOpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1031-azure Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Array Buffer: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ buffer min with 100000 items 16849 16856 10 0.0 842469.3 1.0X buffer max with 100000 items 15513 15515 4 0.0 775641.0 1.1X
Already submit a issue to scala: https://github.com/scala/bug/issues/12614