Flaky Test

Hello, we are a group of researchers interested in understanding methods to reduce costs associated with Test Flakiness [1]. We developed an approach to detect flaky tests, dubbed Shaker [2]. Shaker spawns stressor tasks in the running environment (e.g., tasks that consume lots of cpu or memory) to identify if perturbations can make tests fail.

We ran Shaker on hbase project and found the following likely flaky tests:

• org.apache.hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile.testRetrieveFromFile - failed 5 times in 12 reruns.

org.apache.hadoop.hbase.io.hfile.TestLruBlockCache.testBackgroundEvictionThread - failed 4 times in 12 reruns.

org.apache.hadoop.hbase.master.normalizer.TestRegionNormalizerWorkQueue.testTake - failed 1 time in 12 reruns.

Denini, Leopoldo and Marcelo.

[1] https://hackernoon.com/flaky-tests-a-war-that-never-ends-9aa32fdef359

[2] D. Silva, L. Teixeira and M. d’Amorim, "Shake It! Detecting Flaky Tests Caused by Concurrency with Shaker," 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), Adelaide, Australia, 2020, pp. 301-311, https://ieeexplore.ieee.org/document/9240694/.