Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.15.0
-
None
-
None
Description
A benchmark failure due to the recently-introduced PartitionedPutStringBenchmark was observed:
This is ITERATION 1 of benchmarking against baseline.
P2pPartitionedGetBenchmark avg ops/sec Baseline: 853001.60 Test: 867151.67 Difference: +1.7%
avg latency Baseline: 842007.55 Test: 828545.06 Difference: -1.6%
P2pPartitionedPutBenchmark avg ops/sec Baseline: 128283.47 Test: 126510.92 Difference: -1.4%
avg latency Baseline: 5785619.62 Test: 5915913.49 Difference: +2.3%
P2pPartitionedPutBytesBenchmark avg ops/sec Baseline: 175658.08 Test: 174865.97 Difference: -0.5%
avg latency Baseline: 4130071.43 Test: 4130753.09 Difference: +0.0%
PartitionedFunctionExecutionBenchmark avg ops/sec Baseline: 254788.26 Test: 268132.99 Difference: +5.2%
avg latency Baseline: 846158.41 Test: 804199.42 Difference: -5.0%
PartitionedFunctionExecutionWithArgumentsBenchmark avg ops/sec Baseline: 278669.87 Test: 281504.58 Difference: +1.0%
avg latency Baseline: 1031826.82 Test: 1021314.54 Difference: -1.0%
PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec Baseline: 372204.82 Test: 348815.81 Difference: -6.3%
avg latency Baseline: 1545217.38 Test: 1649706.37 Difference: +6.8%
PartitionedGetBenchmark avg ops/sec Baseline: 823740.09 Test: 819044.99 Difference: -0.6%
avg latency Baseline: 872172.75 Test: 877580.02 Difference: +0.6%
PartitionedGetLongBenchmark avg ops/sec Baseline: 1047221.43 Test: 1045565.89 Difference: -0.2%
avg latency Baseline: 685757.55 Test: 687005.43 Difference: +0.2%
PartitionedGetStringBenchmark avg ops/sec Baseline: 1055904.14 Test: 1045420.73 Difference: -1.0%
avg latency Baseline: 680031.44 Test: 687045.15 Difference: +1.0%
PartitionedIndexedQueryBenchmark avg ops/sec Baseline: 31596.35 Test: 31653.48 Difference: +0.2%
avg latency Baseline: 18221302.10 Test: 18216097.86 Difference: -0.0%
PartitionedNonIndexedQueryBenchmark avg ops/sec Baseline: 95.78 Test: 100.35 Difference: +4.8%
avg latency Baseline: 750871203.78 Test: 716853923.95 Difference: -4.5%
PartitionedPutAllBenchmark avg ops/sec Baseline: 8675.75 Test: 8628.10 Difference: -0.5%
avg latency Baseline: 16595044.73 Test: 16685258.91 Difference: +0.5%
PartitionedPutAllLongBenchmark avg ops/sec Baseline: 1382.38 Test: 1380.50 Difference: -0.1%
avg latency Baseline: 104866853.92 Test: 104775538.34 Difference: -0.1%
PartitionedPutBenchmark avg ops/sec Baseline: 491790.40 Test: 479926.75 Difference: -2.4%
avg latency Baseline: 1461947.23 Test: 1497519.77 Difference: +2.4%
PartitionedPutBytesBenchmark avg ops/sec Baseline: 472520.77 Test: 475046.43 Difference: +0.5%
avg latency Baseline: 1523521.43 Test: 1515515.20 Difference: -0.5%
PartitionedPutLongBenchmark avg ops/sec Baseline: 412720.03 Test: 389975.92 Difference: -5.5%
avg latency Baseline: 1740407.45 Test: 1842985.87 Difference: +5.9%
PartitionedPutStringBenchmark avg ops/sec Baseline: 430083.15 Test: 402523.17 Difference: -6.4%
avg latency Baseline: 1671189.90 Test: 1786949.71 Difference: +6.9%
ReplicatedFunctionExecutionBenchmark avg ops/sec Baseline: 513699.75 Test: 514378.00 Difference: +0.1%
avg latency Baseline: 418897.75 Test: 418374.49 Difference: -0.1%
ReplicatedFunctionExecutionWithArgumentsBenchmark avg ops/sec Baseline: 525004.44 Test: 522304.20 Difference: -0.5%
avg latency Baseline: 2192090.72 Test: 2203176.97 Difference: +0.5%
ReplicatedFunctionExecutionWithFiltersBenchmark avg ops/sec Baseline: 540464.25 Test: 532795.86 Difference: -1.4%
avg latency Baseline: 1330260.37 Test: 1349440.25 Difference: +1.4%
ReplicatedGetBenchmark avg ops/sec Baseline: 827036.49 Test: 817813.56 Difference: -1.1%
avg latency Baseline: 868818.39 Test: 878720.34 Difference: +1.1%
ReplicatedGetLongBenchmark avg ops/sec Baseline: 1085165.62 Test: 1071977.64 Difference: -1.2%
avg latency Baseline: 661809.29 Test: 670064.70 Difference: +1.2%
ReplicatedIndexedQueryBenchmark avg ops/sec Baseline: 32216.21 Test: 31293.98 Difference: -2.9%
avg latency Baseline: 17873159.19 Test: 18437420.16 Difference: +3.2%
ReplicatedNonIndexedQueryBenchmark avg ops/sec Baseline: 51.62 Test: 47.74 Difference: -7.5%
avg latency Baseline: 1390417763.11 Test: 1505134279.26 Difference: +8.3%
ReplicatedPutAllBenchmark avg ops/sec Baseline: 788.90 Test: 814.00 Difference: +3.2%
avg latency Baseline: 182735327.71 Test: 177046841.08 Difference: -3.1%
ReplicatedPutAllLongBenchmark avg ops/sec Baseline: 832.81 Test: 826.93 Difference: -0.7%
avg latency Baseline: 173063206.58 Test: 174233069.86 Difference: +0.7%
ReplicatedPutBenchmark avg ops/sec Baseline: 490054.06 Test: 494297.13 Difference: +0.9%
avg latency Baseline: 1466451.86 Test: 1454784.46 Difference: -0.8%
ReplicatedPutLongBenchmark avg ops/sec Baseline: 457689.80 Test: 457973.53 Difference: +0.1%
avg latency Baseline: 1571708.57 Test: 1569573.85 Difference: -0.1%
This is ITERATION 2 of benchmarking against baseline.
PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec Baseline: 395993.05 Test: 358748.93 Difference: -9.4%
avg latency Baseline: 1452604.27 Test: 1603625.58 Difference: +10.4%
PartitionedPutLongBenchmark avg ops/sec Baseline: 422650.17 Test: 415015.59 Difference: -1.8%
avg latency Baseline: 1702486.44 Test: 1731996.79 Difference: +1.7%
PartitionedPutStringBenchmark avg ops/sec Baseline: 429296.09 Test: 391046.61 Difference: -8.9%
avg latency Baseline: 1675896.07 Test: 1837288.98 Difference: +9.6%
ReplicatedNonIndexedQueryBenchmark avg ops/sec Baseline: 49.02 Test: 50.55 Difference: +3.1%
avg latency Baseline: 1464289182.72 Test: 1420848686.90 Difference: -3.0%
This is ITERATION 3 of benchmarking against baseline.
PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec Baseline: 407931.35 Test: 434871.25 Difference: +6.6%
avg latency Baseline: 1410246.93 Test: 1322795.29 Difference: -6.2%
PartitionedPutStringBenchmark avg ops/sec Baseline: 427102.32 Test: 402755.46 Difference: -5.7%
avg latency Baseline: 1682556.85 Test: 1786603.33 Difference: +6.2%
This is ITERATION 4 of benchmarking against baseline.
PartitionedPutStringBenchmark avg ops/sec Baseline: 434609.00 Test: 402475.52 Difference: -7.4%
avg latency Baseline: 1656172.72 Test: 1785229.42 Difference: +7.8%
This is ITERATION 5 of benchmarking against baseline.
PartitionedPutStringBenchmark avg ops/sec Baseline: 423047.66 Test: 402475.29 Difference: -4.9%
avg latency Baseline: 1700791.11 Test: 1787257.59 Difference: +5.1%
Looking at some previous runs of the benchmarks since this particular benchmark was introduced, it appears that there is some instability in this benchmark:
PartitionedPutStringBenchmark average ops/second Baseline: 384659.34 Test: 375511.88 Difference: -2.4% average latency Baseline: 1868722.73 Test: 1918780.45 Difference: +2.7% average ops/second Baseline: 434900.93 Test: 391933.67 Difference: -9.9% average latency Baseline: 1656702.07 Test: 1830827.53 Difference: +10.5% average ops/second Baseline: 419108.79 Test: 401341.06 Difference: -4.2% average latency Baseline: 1712897.05 Test: 1791919.35 Difference: +4.6% average ops/second Baseline: 421598.58 Test: 399128.86 Difference: -5.3% average latency Baseline: 1707314.91 Test: 1801691.41 Difference: +5.5% average ops/second Baseline: 424445.58 Test: 403798.53 Difference: -4.9% average latency Baseline: 1694539.23 Test: 1781948.40 Difference: +5.2% average ops/second Baseline: 417622.25 Test: 388290.32 Difference: -7.0% average latency Baseline: 1721608.57 Test: 1850134.14 Difference: +7.5% average ops/second Baseline: 413538.43 Test: 396040.63 Difference: -4.2% average latency Baseline: 1738858.16 Test: 1820714.96 Difference: +4.7% average ops/second Baseline: 363423.19 Test: 394963.41 Difference: +8.7% average latency Baseline: 1981399.62 Test: 1822056.94 Difference: -8.0% average ops/second Baseline: 369520.84 Test: 394239.61 Difference: +6.7% average latency Baseline: 1948697.59 Test: 1824966.98 Difference: -6.3%