Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18335

Improve Pre-Commit Time

    XMLWordPrintableJSON

Details

    • Task
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      As of now the complete build time has reached ~24 hours, which makes tracking Jira with root level changes very tough.

      Even at module level, it is on the higher side, kind of 6 hours or more, which is typically equivalent to 1 working day hours:

      Some areas to explore:

      • Enable Parallel-Test profile for the modules which don't have it.
      • Explore improvements in the area of our Test setup, increase memory, number of threads, or some parallel executions?
      • Remove the modules or atleast disable the test suites of modules which are no longer being used or maintained (Separate Mail Chain is there for this)
      • Spin Two Pre-Commit jobs rather than 1 and split some task between them, kind of one for Java-11 build & Javadoc, Checkstyle and related stuff.
      • Figure out & remove irrelevant or similar tests.
      • Improve existing tests, Like reusing MiniDfsClusters or so, rather than a bunch of test spinning one for themselves.
      • If possible, if there are multiple modules to be tested, run those modules test in parallel (Exploratory: may lead to OOM)

      Attachments

        Activity

          People

            Unassigned Unassigned
            ayushtkn Ayush Saxena
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated: