Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1751

Surefire report shows flaky tests as failures

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 3.0.0-M4
    • None
    • None
    • None
    • Important

    Description

      We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our testing. Now we are trying to upgrade junit to junit5 and saw that the rerunFailingTestsCount parameter isn't working with surefire plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing tests are now being retried. But the issue is flaky tests show up in the report as Failures.

       

      ie. the log shows
      [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
       
      But the actual surefire report shows like below

       
      This is blocking our junit5 upgrade task. Can you please provide a fix for this at the earliest? Please let me know if you need anything else from me to help with the investigation.

      Attachments

        1. image-2020-02-13-11-41-03-906.png
          8 kB
          Anu
        2. Test.zip
          28 kB
          Anu
        3. TEST-com.test.Flakes.xml
          7 kB
          Anu

        Issue Links

          Activity

            People

              Unassigned Unassigned
              anujose Anu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: