Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-9472

True concurrent test suite duplication and execution (beasting)

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Workaround
    • None
    • None
    • None
    • None
    • New

    Description

      I toyed with this a bit because it's hacker's delight... Forked the default Gradle test task and made it emit the same test/ suite name multiple times. This results in the same test being executed multiple times:

      # ./gradlew -p lucene\analysis\kuromoji testy --tests TestToStringUtil.testHepburn -Ptests.dups=10
      
      > Task :randomizationInfo
      Running tests with randomization seed: tests.seed=88C21B0E8ABB2D46
      
      > Task :lucene:analysis:kuromoji:testy
      :lucene:analysis:kuromoji:testy (SUCCESS): 10 test(s)
      
      BUILD SUCCESSFUL in 13s
      21 actionable tasks: 6 executed, 15 up-to-date
      

      Upsides:

      • Everything configures itself because the custom task is a subtype of Gradle's Test task,
      • Tests are run in parallel, on multiple VMs,
      • Opens up the possibility of reordering test suites (for load-balancing).

      Downsides:

      • Requires a custom(ized) clone of Gradle's internal classes. Will require maintenance when upgrading gradle. There is no certainty it'll work.
      • Some things visibly don't like duplicated suite names - the progress bar shows the suite once, even though it is executed concurrently in multiple VMs,
      • XML reports will drive jenkins and other tooling crazy:
        <?xml version="1.0" encoding="UTF-8"?>
        <testsuite name="org.apache.lucene.analysis.ja.util.TestToStringUtil" tests="10" skipped="0" failures="0" errors="0" timestamp="2020-08-20T12:54:27" hostname="DWEISS-DESKTOP" time="3.177">
          <properties/>
          <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.04"/>
          <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.045"/>
          <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.043"/>
          <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.038"/>
          <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.008"/>
          <testcase name="testHepburn" classname="org.apache.lucene.analysis.ja.util.TestToStringUtil" time="1.005"/>
        ....
        
      • not sure what happens with sysouts from such concurrent runs...

      Attachments

        Issue Links

          Activity

            People

              dweiss Dawid Weiss
              dweiss Dawid Weiss
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m